Crawl a site for only the pages you describe
Start from a page, say what kind of page you want, and get each one back as Markdown or as JSON in your shape. Listing pages read along the way are free, and a report shows why anything was skipped.
Describe the page, not the path to it
The same blog, crawled three ways for 10 pages each. What came back depended on how the pages were described.
"All pages"
10 credits{"baseUrl": "https://spidra.io/blog","crawlInstruction": "All pages","maxPages": 10}
The index, six category pages and three posts
All pages means all pages.
Describe the page
28 credits · 10 pages + AI{"baseUrl": "https://spidra.io/blog","crawlInstruction": "Individual blog posts","schema": { … title, author, published, category },"maxPages": 10}
Ten posts, each read into the schema
The index was read only to find posts, and wasn't charged.
Fence it with a path
10 credits · no AI{"baseUrl": "https://spidra.io/blog","crawlInstruction": "All pages","excludePaths": ["/blog/category/*"],"maxPages": 10}
The index and nine posts, as Markdown
Cheapest when a path tells the pages apart.
Every crawl explains itself
When a page you expected is missing, the report says how your instruction was read and why each link was skipped.
The report for the "Individual blog posts" run, trimmed
{"understoodAs": "Individual blog post articles.","navigationPages": 1,"skippedTotal": 55,"skipped": [{ "reason": "outside_domain", "count": 32 },{ "reason": "not_relevant", "count": 23 }]}
The full report also lists each skipped URL with the page it was found on, up to 2,000 per crawl. In the dashboard it's the How this crawl worked panel.
Why a link gets skipped
- not_relevantDidn't match your instruction
- outside_domainAnother site, or a subdomain you didn't allow
- outside_start_pathSame site, outside the section you started in
- include_pattern / exclude_patternFenced out by your path filters
- other_version_or_languageAnother version or translation of a page
- duplicate_pageThe same content under another URL
- file_typeImages, video, archives and scripts
- action_linkAdd to cart, log out and similar, never followed
Markdown with no AI, or your fields
Leave out the schema and instruction and every page comes back as Markdown for 1 credit. Add them and every page is read into the same shape.
Python's asyncio docs as Markdown: 5 pages, 5 credits
https://docs.python.org/3/library/asyncio-runner.html 10407 charactershttps://docs.python.org/3/library/asyncio-task.html 59841 charactershttps://docs.python.org/3/library/asyncio-stream.html 22038 charactershttps://docs.python.org/3/library/asyncio-subprocess.html 17414 charactershttps://docs.python.org/3/library/asyncio-queue.html 11371 characters
Code blocks stay fenced. Crawled Markdown is the whole page, menus included, so strip repeated navigation before chunking.
A blog post read into the schema
{"title": "How to Scrape PDFs and Extract Structured JSON with an API","author": "Joel Olawanle","published": "October 2, 2026","category": null}
category is null because a post page doesn't show its own category. A schema can only ask for what's on the page.
Re-extract without recrawling
Run a new instruction over the pages a crawl saved, without visiting the site again. You pay AI tokens only.
Big crawls, page by page
The first 100 pages come with the job. Read the rest through the paged results, up to 1,000 per call.
Everything as a ZIP
Download every page of a finished crawl: its data, HTML and Markdown in one folder each.
A start page and a sentence. The rest when you need it.
- baseUrlstringWhere to start: a home, category or docs index that links to what you want.
- crawlInstructionstringThe pages you want, in plain words. Name the kind of page.
- schemaobjectA JSON Schema for each page's data. Or transformInstruction in plain words.
- maxPagesnumberPages to return. Pages read only for links don't count.
- includePathsarrayOnly follow matching paths, like /blog/*. excludePaths leaves paths out.
- crawlEntireDomainfalseFollow links anywhere on the domain, not only under the start path.
- maxDepthunlimitedClicks from the start page. Usually best left unset.
- scrapeModedefaultfast reads pages without a browser when they don't need one.
- useProxyfalseStealth Mode, with proxyCountry and cookies as on a scrape.
A crawl stays under the path it starts in unless you set crawlEntireDomain. The instruction steers it; path filters fence it in.
What it costs
1 credit per page you get
Plus AI tokens when you ask for fields, and 5 per CAPTCHA solved. Pages read only to find the ones you asked for are free.
See plansCosts nothing
- Listing and contents pages read for links
- Pages that fail, and crawls that fail
- Re-reading saved pages costs tokens only
Pages per crawl
- 100 on Free, 1,000 on Starter
- 5,000 on Builder, 10,000 on Pro
- 60 crawls a minute on every plan
Questions, answered
Crawl
It follows the links it judges likely to lead to the pages you described, and skips the rest. For a hard boundary, add includePaths or excludePaths. The crawl report lists every skipped link with the reason.
You build the product.
Spidra handles the data work.
Start with 300 free credits, about 300 pages. No credit card required.