How to crawl a website and extract data with the Spidra API

Updated October 2026: rewritten for the new crawl engine. Crawls now return up to 10,000 pages on Pro, the pages read only to find your results are free, and every crawl comes with a report that explains why a page wasn't returned.

You know where the data lives, but not every URL. The books are somewhere under a category page, the job posts sit behind a listing, the docs are linked from an index. Writing a link-follower for each site is slow, and it breaks the first time the site changes its navigation.

A crawl takes one starting URL and a plain-language description of the pages you want. It follows the site's links to find them, returns only those pages, and extracts the same fields from each one. Below, one crawl of a sandbox bookstore returns three books as typed JSON for 5 credits, and its report explains why the other 51 links were skipped. The same crawl then runs in Python and Node.js, and the last sections cover what changes when a crawl gets large.

You need a Spidra API key (the free plan includes 300 credits, no card) exported as SPIDRA_API_KEY, and curl, Python 3.9+ or Node.js 18+.

When should you crawl instead of scrape?

Crawl when you can name the kind of page you want but can't list the URLs. If you already have the URLs, a batch scrape is simpler and more predictable: up to 50 URLs per request, one result each. If you don't know how a site is laid out yet, Map lists every URL it can find in seconds for 1 credit, and you can batch scrape the ones you need.

Your first crawl

The example site is books.toscrape.com, a sandbox bookstore built for scraping practice. The goal: the title, price and stock status of books in the Mystery category, as typed JSON.

Submit the crawl:

curl -X POST https://api.spidra.io/api/crawl \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "baseUrl": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    "crawlInstruction": "Follow links to individual book pages only",
    "schema": {
      "type": "object",
      "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "in_stock": {"type": "boolean"}
      }
    },
    "maxPages": 3
  }'

Only baseUrl and crawlInstruction are required. The schema sets the exact fields and types for every page, and maxPages caps how many pages come back. The API answers right away with a job ID:

{"status": "queued", "jobId": "JOB_ID", "message": "Crawl job queued. Poll /api/crawl/{jobId} for results."}

Crawls take seconds to minutes, so they run as jobs. Poll the job every few seconds until status is completed, failed or cancelled. Polling is free.

curl -H "Authorization: Bearer $SPIDRA_API_KEY" https://api.spidra.io/api/crawl/JOB_ID

While it runs, progress.message says what the crawl is doing ("Reading the instruction and the start page..."). Here's the finished result from our run on October 13, 2026, trimmed to the first of the three books:

{
  "status": "completed",
  "progress": {"status": "completed", "message": "Crawl completed", "progress": 1},
  "result": [
    {
      "url": "https://books.toscrape.com/catalogue/in-a-dark-dark-wood_963/index.html",
      "status": "success",
      "title": "In a Dark, Dark Wood | Books to Scrape - Sandbox",
      "depth": 1,
      "parentUrl": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
      "data": {"title": "In a Dark, Dark Wood", "price": 19.63, "in_stock": true},
      "html": "https://…signed link…",
      "markdown": "https://…signed link…"
    }
  ],
  "error": null,
  "totalPages": 3
}

The other two results were Sharp Objects (price: 47.82) and The Past Never Ends (price: 56.5), with the same three fields.

What happened, and what it cost

The category page you started from isn't in the results. The crawl read it to find the book links, then returned only the book pages, because that's what the instruction asked for. Pages read only to reach your results are called navigation pages, and they're free.

Each result tells you how it was found: depth is the number of clicks from the start page, and parentUrl is the page the link was on. html and markdown are signed links to the stored page. They expire, so download what you need soon after the crawl, or use the ZIP download endpoint covered later.

The crawl cost 5 credits: 1 per book page returned (3) plus 2 for the AI extraction, which read 3,450 input tokens and wrote 45 output tokens. The category page cost nothing.

That gives you a way to estimate bigger jobs. With a schema, budget about 2 credits per page: 1 for the page and about 1 for the extraction, depending on how long the pages are. A 1,000-page crawl with a schema comes to roughly 2,000 credits, which is 40% of the Starter plan's 5,000 monthly credits. Without a schema or transformInstruction, no AI runs, each page's data is its Markdown, and the cost is 1 credit per page (plus 5 for any CAPTCHA solved). Before a big crawl, run 10 or 20 pages and check the real cost in the dashboard's Logs, or in credits_used from GET /crawl/job/{jobId}.

Why use a schema instead of a transform instruction?

You can describe the fields in plain language with transformInstruction instead of a schema. We ran the same crawl with "Return the book title, price and star rating" and got this for one book:

…/sharp-objects_997/index.html  { book_title: 'Sharp Objects', price: '£47.82', star_rating: null }

The AI chose the field name book_title, kept the price as a string with the currency sign, and returned null for the rating because it isn't in the page text (the site shows it as a CSS class). Field names and types can also change from page to page. That's fine for reading by eye. When code reads the output, use a schema.

How to write a crawl instruction that finds the right pages

The crawl starts by working out what kind of page you described. Name the page type, not the topic: "Individual job postings" works better than "jobs", because a jobs site also has search pages, category pages and company pages. "All pages" returns any page on the site. The instruction works in any language.

The instruction steers the crawl, but it isn't a fence. For a hard boundary, add path filters:

{
  "includePaths": ["/blog/*"],
  "excludePaths": ["/tag/*", "/author/*"]
}

A pattern matches anywhere in the path, so /books/* also matches /catalogue/category/books/…. Leave maxDepth (how many clicks from the start page the crawl may go) unset in most cases: the pages you want often sit two clicks away, behind a category and then a listing page.

Why wasn't a page returned? Read the crawl report

Every finished crawl has a report. Ask for it when a page you expected is missing:

curl -H "Authorization: Bearer $SPIDRA_API_KEY" \
  "https://api.spidra.io/api/crawl/JOB_ID/report?limit=3"

For the Mystery crawl, it returned this (trimmed to the first skipped URL):

{
  "status": "success",
  "report": {
    "understoodAs": "Individual book detail pages.",
    "navigationPages": 1,
    "skippedTotal": 51,
    "skipped": [
      {"reason": "not_relevant", "count": 51, "description": "Didn't match the crawl instruction"}
    ],
    "skippedUrls": [
      {
        "url": "https://books.toscrape.com/index.html",
        "reason": "not_relevant",
        "foundOn": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
      }
    ],
    "skippedUrlsTotal": 51
  }
}

Read understoodAs first. It's the kind of page the crawl looked for, in its own words. If it isn't what you meant, reword the instruction. Then read the skip reasons. The common ones are not_relevant (didn't match the instruction), outside_domain (another site, or a subdomain without allowSubdomains), include_pattern and exclude_pattern (your path filters), duplicate_page, and robots_txt (when you set respectRobots). The full list is in the crawl report reference.

The usual fixes are a more specific instruction, an include path that matches the missing page, or a higher maxPages. In the dashboard, the same report appears as the How this crawl worked panel above the results:

It shows what the crawl looked for first, then each skip reason with its count and the links behind it.

The same crawl in Python

Install the SDK with pip install spidra (version 0.5.1 at the time of writing). This script starts the crawl, waits for it with a time limit, prints each book, and then reads the report using the job ID:

# crawl_books.py
import os
import time

from spidra import CrawlParams, Spidra

BOOK_SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "in_stock": {"type": "boolean"},
    },
}
POLL_SECONDS = 3
TIMEOUT_SECONDS = 600


def main() -> None:
    client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

    queued = client.start_crawl(CrawlParams(
        base_url="https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
        crawl_instruction="Follow links to individual book pages only",
        schema=BOOK_SCHEMA,
        max_pages=3,
    ))

    deadline = time.monotonic() + TIMEOUT_SECONDS
    job = client.get_crawl(queued.job_id)
    while job.status not in ("completed", "failed", "cancelled"):
        if time.monotonic() > deadline:
            raise TimeoutError(f"Crawl {queued.job_id} is still running")
        time.sleep(POLL_SECONDS)
        job = client.get_crawl(queued.job_id)

    if job.status != "completed":
        reason = getattr(job, "error", None) or "no reason given"
        raise RuntimeError(f"Crawl {queued.job_id} ended with status {job.status}: {reason}")

    for page in job.result:
        print(page.url, page.data)

    report = client.crawl_report(queued.job_id, limit=3).report
    print(report.understood_as, report.navigation_pages, report.skipped_total)


if __name__ == "__main__":
    main()

The output has the same three books as the curl run, then the report line:

https://books.toscrape.com/catalogue/in-a-dark-dark-wood_963/index.html {'title': 'In a Dark, Dark Wood', 'price': 19.63, 'in_stock': True}
…
Individual book detail pages. 1 51

Python parameters are snake_case (base_url, max_pages). If you don't need the job ID afterwards, client.crawl(CrawlParams(...)) does the waiting for you and returns the finished result in one call.

The same crawl in Node.js

Install the SDK with npm install spidra and run this as an ES module (a .mjs file, or "type": "module" in package.json):

// crawl-books.mjs
import { SpidraClient } from "spidra";

const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY });

const job = await spidra.crawl(
  {
    baseUrl: "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    crawlInstruction: "Follow links to individual book pages only",
    schema: {
      type: "object",
      properties: {
        title: { type: "string" },
        price: { type: "number" },
        in_stock: { type: "boolean" },
      },
    },
    maxPages: 3,
  },
  { timeout: 600_000 },
);

for (const page of job.result) {
  console.log(page.url, page.data);
}

crawl() polls until the job finishes and throws if it fails or passes the timeout (in milliseconds).

The npm package (0.7.0) doesn't have a crawl report method yet, and crawl() doesn't return the job ID. To read the report from Node, submit with spidra.startCrawl(params), which returns { jobId }, wait with spidra.getCrawl(jobId), and then call the report endpoint:

// crawl-report.mjs: run with the job ID, e.g. node crawl-report.mjs JOB_ID
const jobId = process.argv[2];
const res = await fetch(
  `https://api.spidra.io/api/crawl/${encodeURIComponent(jobId)}/report?limit=3`,
  {
    headers: { Authorization: `Bearer ${process.env.SPIDRA_API_KEY}` },
    signal: AbortSignal.timeout(30_000),
  },
);
if (!res.ok) {
  throw new Error(`Report request failed with HTTP ${res.status}: ${await res.text()}`);
}
const { report } = await res.json();
console.log(report.understoodAs, report.navigationPages, report.skippedTotal);

The Python SDK has a crawl_report() method.

Change what you extract without crawling again

Picked the wrong fields? You don't need to visit the site again. Re-extract runs a new instruction over the pages the crawl already saved:

curl -X POST https://api.spidra.io/api/crawl/JOB_ID/extract \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"transformInstruction": "Return the book title only, as {\"title\": string}"}'

It returns a new job ID to poll like any crawl. On our three-page crawl it finished in seconds and cost 2 credits, for the AI tokens only. That makes it a cheap way to try different prompts against the same pages. The instruction can be up to 5,000 characters, and the source crawl must have finished first.

Crawling thousands of pages

maxPages is the number of pages returned, and its ceiling depends on your plan: 100 on Free, 1,000 on Starter, 5,000 on Builder and 10,000 on Pro. Navigation pages don't count toward it, and if you don't set maxPages the default is 5. If your credits can't cover the maxPages you asked for, the crawl still runs with a lower maxPages, and the submit response tells you. Credits are held at submit, and anything unused is returned when the crawl ends.

The status response holds the first 100 pages. For bigger crawls it adds "resultTruncated": true, and you read the rest from the pages endpoint, up to 1,000 at a time:

curl -H "Authorization: Bearer $SPIDRA_API_KEY" \
  "https://api.spidra.io/api/crawl/JOB_ID/pages?offset=0&limit=1000"

The response has pages, total, offset, limit and hasMore; keep raising offset until hasMore is false. One thing to watch: on this endpoint each page's data is a JSON string, not an object, so parse it before use. In Python, client.crawl() reads every page for you. A manual get_crawl() loop like the one above gets only the first 100 pages, so check job.result_truncated and call client.crawl_all_pages(job_id) for the rest (its data is a JSON string too, so json.loads it). In Node, the npm package's crawl() (0.7.0) also stops at the first 100 pages: when job.resultTruncated is true, read the rest from the pages endpoint with the job ID from startCrawl().

To keep everything, GET /crawl/{jobId}/download returns a ZIP with a folder per page holding its extracted data, HTML and Markdown.

Other options worth knowing

You're responsible for the sites you crawl and the accounts you use: read each site's terms, set respectRobots where it applies, and don't collect personal data you have no right to. Our post on whether web scraping is legal covers the basics.

What can go wrong

• A page you expected is missing. Read the report, then tighten the instruction or add an include path.

• Field names change between pages or runs. That happens without a schema. Add one.

• A field comes back null. The data isn't in the page text, for example a rating shown only as a CSS class.

• A page extracts as {}. It's rare. Retry that page with POST /crawl/{jobId}/retry/{pageId} (the page id comes from the pages endpoint), or re-extract the crawl.

• The start page can't be read at all. The crawl fails with the reason and a hint (such as turning on Stealth Mode), and costs nothing. CAPTCHAs on other pages are solved automatically at 5 credits per solve.

• Some sites block automated visits even with a browser, proxies and CAPTCHA solving. Those pages fail and aren't charged.