Spidra API tutorial: complete guide to web scraping with the Spidra API

Getting data from websites programmatically has always involved more work than it should. You write selectors, they break when the site updates. You try a headless browser, anti-bot protection blocks you. You get the data, but it is raw HTML and you still have to parse it into something useful.

The Spidra API is designed to solve all three of those problems in one place. You send a URL, describe what you want, and get back structured data. The browser rendering, CAPTCHA solving, proxy rotation, and AI extraction all happen on Spidra's side.

This guide walks through the entire API from authentication to crawling. By the end you will know how every endpoint works, what the response structure looks like, and how to build a real scraping pipeline around it.

Before you start

You need a Spidra account and an API key.

Sign up at spidra.io. The free plan includes 300 credits with no credit card required. Once you are in, go to app.spidra.io → Settings → API Keys and create a key.

Keep it somewhere safe. Every request you make to the API includes this key in the header.

How the API works

The Spidra API is a REST API with one base URL:

https://api.spidra.io/api

Every request is authenticated by including your API key in the x-api-key header. There are no bearer tokens, no OAuth flows, just a header on every request.

curl -X POST https://api.spidra.io/api/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": [{"url": "https://example.com"}]}'

One important thing to understand before you make your first request: Spidra jobs are asynchronous. When you submit a scrape, you do not get the data back immediately. You get a job ID. You then poll a status endpoint every few seconds until the job is complete and the data is ready.

This is by design. Browser rendering, CAPTCHA solving, and AI extraction take a few seconds. The async pattern means you are not holding a connection open the whole time.

The flow for every job type looks like this:

1. Submit the job. Receive a job ID in the response.
2. Poll the status endpoint every 2 to 5 seconds.
3. When status is completed, read your results.

Now let us go through each part of the API.

Authentication

Every request needs the x-api-key header. That is it.

-H "x-api-key: YOUR_API_KEY"

If the key is missing or invalid, the API returns a 401. If your credits are exhausted, you get a 403.

Here is the full set of response codes you will encounter:

All errors come back in the same format:

{
  "status": "error",
  "message": "Detailed explanation of what went wrong"
}

Scraping a single page

The scrape endpoint is where most people start. You give it one to three URLs and it returns structured data from each one.

Endpoint: POST /api/scrape

The minimal request

The only required field is urls, which takes an array of URL objects. Each URL object requires a url field and optionally takes an actions array for browser interactions.

curl -X POST https://api.spidra.io/api/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [{"url": "https://news.ycombinator.com"}]
  }'

Response:

{
  "status": "queued",
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Scrape job has been queued. Poll /api/scrape/550e8400... to get the result."
}

Save that jobId. You need it to check on the job.

Polling for results

Call GET /api/scrape/{jobId} every few seconds until the status changes.

curl https://api.spidra.io/api/scrape/550e8400-e29b-41d4-a716-446655440000 \
  -H "x-api-key: YOUR_API_KEY"

While the job is running, you will see something like this:

{
  "status": "active",
  "progress": {
    "message": "Processing content with AI...",
    "progress": 0.6
  },
  "result": null,
  "error": null
}

The progress field goes from 0 to 1 as the job moves through its stages: loading the browser, executing actions, solving CAPTCHAs, running AI extraction.

When it finishes:

{
  "status": "completed",
  "progress": {
    "message": "Scrape completed successfully",
    "progress": 1
  },
  "result": {
    "content": "...",
    "data": [
      {
        "url": "https://news.ycombinator.com",
        "title": "Hacker News",
        "markdownContent": "...",
        "success": true,
        "screenshotUrl": null
      }
    ],
    "screenshots": [],
    "ai_extraction_failed": false,
    "stats": {
      "durationMs": 4200,
      "captchaSolvedCount": 0,
      "inputTokens": 312,
      "outputTokens": 84,
      "totalTokens": 396
    }
  },
  "error": null
}

The result.content field is the main output. What it contains depends on what you asked for:

• If you passed a prompt, content is the AI-extracted result
• If you did not pass a prompt, content is the raw page content as Markdown

result.data is an array with one entry per URL. Each entry has the page title, the full Markdown content for that URL, whether it succeeded, and a screenshot URL if you requested one.

result.stats tells you how long the job took, how many CAPTCHAs were solved, and how many tokens the AI extraction used.

A polling loop in Python

import requests
import time

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.spidra.io/api"
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}

def scrape(url):
    # Submit the job
    response = requests.post(
        f"{BASE_URL}/scrape",
        headers=HEADERS,
        json={"urls": [{"url": url}]}
    )
    response.raise_for_status()
    job_id = response.json()["jobId"]

    # Poll until complete
    while True:
        status_response = requests.get(
            f"{BASE_URL}/scrape/{job_id}",
            headers=HEADERS
        )
        data = status_response.json()

        if data["status"] == "completed":
            return data["result"]
        elif data["status"] == "failed":
            raise Exception(f"Scrape failed: {data['error']}")

        time.sleep(3)

result = scrape("https://news.ycombinator.com")
print(result["content"])

The same in Node.js:

const API_KEY = "YOUR_API_KEY";
const BASE_URL = "https://api.spidra.io/api";
const HEADERS = {
  "x-api-key": API_KEY,
  "Content-Type": "application/json"
};

async function scrape(url) {
  const submitRes = await fetch(`${BASE_URL}/scrape`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({ urls: [{ url }] })
  });
  const { jobId } = await submitRes.json();

  while (true) {
    const statusRes = await fetch(`${BASE_URL}/scrape/${jobId}`, {
      headers: HEADERS
    });
    const data = await statusRes.json();

    if (data.status === "completed") return data.result;
    if (data.status === "failed") throw new Error(data.error);

    await new Promise(r => setTimeout(r, 3000));
  }
}

const result = await scrape("https://news.ycombinator.com");
console.log(result.content);

AI extraction with prompts

The plain scrape above gives you raw Markdown. Most of the time you want something more specific. That is where the prompt field comes in.

Add a prompt and Spidra reads the rendered page and extracts exactly what you described. The AI understands context. It knows a number next to a currency symbol is a price, that a short bold line near the top of a product page is probably the title, and that a block of longer text is likely a description. You describe the output you want and it figures out where to find it.

curl -X POST https://api.spidra.io/api/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [{"url": "https://news.ycombinator.com"}],
    "prompt": "Extract the top 10 post titles and their point scores",
    "output": "json"
  }'

When the job completes, result.content contains the AI-extracted data as JSON:

[
  {"title": "Show HN: I built a thing", "points": 342},
  {"title": "Ask HN: What are you working on?", "points": 289}
]

The output field controls the format. It defaults to "json" but you can set it to "markdown" if you want the extracted content as formatted text instead of structured data.

One thing to know: if you set output: "json" without a prompt, Spidra still runs a default AI extraction pass. If you want the raw page content with no AI processing at all, omit both output and prompt.

If AI extraction fails for any reason (a near-empty page, a heavily obfuscated site), Spidra falls back to returning the raw page Markdown and sets ai_extraction_failed: true in the response so your code can detect and handle it.

Structured output with JSON schema

Prompts are flexible but they are not predictable. The AI decides what fields to return and what to call them. For production pipelines where downstream systems expect a specific shape, that is a problem.

The schema field solves this. Pass a JSON Schema object and the AI must return data that matches it exactly. Required fields always appear in the output, as null if the page does not have that value. Field names match exactly what you defined. The structure never varies between runs.

curl -X POST https://api.spidra.io/api/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [{"url": "https://jobs.example.com/senior-engineer"}],
    "prompt": "Extract the job details. Normalize salary to a number in USD.",
    "schema": {
      "type": "object",
      "required": ["title", "company", "remote", "employment_type"],
      "properties": {
        "title":           {"type": "string"},
        "company":         {"type": "string"},
        "remote":          {"type": ["boolean", "null"]},
        "salary_min":      {"type": ["number", "null"]},
        "salary_max":      {"type": ["number", "null"]},
        "employment_type": {
          "type": ["string", "null"],
          "enum": ["full_time", "part_time", "contract", null]
        }
      }
    }
  }'

The response will always have title, company, remote, and employment_type because they are in required. If the page does not mention a salary, salary_min and salary_max come back as null rather than being omitted.

When you provide a schema, output is automatically set to "json". You do not need to set it yourself.

The schema is validated before the job is queued. If it is malformed, the API returns a 422 with descriptive errors. Non-fatal issues like unsupported keywords come back as schema_warnings in the response.

Schema limits to be aware of: maximum nesting depth is 5 levels, maximum schema size is 10 KB.

Browser actions

Some pages do not show you the data you want until you interact with them first. Cookie banners blocking content. A "Load More" button that reveals the next batch of results. A search form you need to fill before anything appears. Tabs that hide content by default.

The actions array on each URL object lets you interact with the page before extraction runs. Actions execute in order, inside a real browser, before Spidra runs your prompt.

Here is an example that dismisses a cookie banner, fills a search form, and waits for results to load:

{
  "urls": [{
    "url": "https://example.com/search",
    "actions": [
      {"type": "click", "value": "Accept cookies button"},
      {"type": "type", "selector": "input[name='q']", "value": "wireless headphones"},
      {"type": "click", "selector": "button[type='submit']"},
      {"type": "wait", "duration": 1500},
      {"type": "scroll", "to": "80%"}
    ]
  }],
  "prompt": "Extract all product names and prices from the search results",
  "output": "json"
}

Notice that for the first click, the value field is a plain English description of the element. For the second click, the selector field is a CSS selector. Both approaches work and you can mix them in the same actions array.

For any click, check, or uncheck action:

• Use selector for a CSS selector or XPath expression like "#accept-cookies" or ".submit-btn"
• Use value for a plain English description like "Accept cookies button" and Spidra's AI will find the element for you

Both are equally valid. Use whichever makes more sense for the page you are working with.

Available actions

The forEach action

forEach is the most powerful action in the API. It finds a set of repeating elements on the page (product cards, search result links, accordion rows, directory listings) and processes each one individually, then combines all the results into a single output.

It supports three modes:

inline reads the content of each matched element directly. Use this for product cards, table rows, or any content that lives inside the element itself.

navigate follows each element as a link, loads the destination page, and scrapes it. Use this when the data you want is on detail pages that you need to navigate into.

click clicks each element to expand or reveal content, then scrapes what appears. Use this for accordions, modals, or expandable sections.

{
  "urls": [{
    "url": "https://directory.example.com/companies",
    "actions": [
      {"type": "click", "value": "Accept cookies"},
      {
        "type": "forEach",
        "value": "Find all company listing cards",
        "mode": "navigate",
        "maxItems": 20,
        "itemPrompt": "Extract company name, website, and industry",
        "pagination": {
          "nextSelector": "a.next-page",
          "maxPages": 3
        }
      }
    ]
  }],
  "output": "json"
}

This dismisses the cookie banner, finds every company card on the page, navigates into each one, extracts the company details, and repeats across 3 pages of pagination. All in a single API call.

Proxy and geo-targeting

Some sites block traffic from cloud IP ranges. Others serve different content based on location. The useProxy and proxyCountry fields route your requests through residential proxies to handle both situations.

{
  "urls": [{"url": "https://amazon.de/dp/B123456"}],
  "prompt": "Extract the product price",
  "output": "json",
  "useProxy": true,
  "proxyCountry": "de"
}

Setting useProxy: true routes the request through the residential proxy network. proxyCountry accepts:

• A two-letter ISO country code like "us", "de", "gb", "fr"
• "eu" to rotate randomly across all 27 EU member states
• "global" or omit it entirely for no country preference

Proxy usage is billed from your bandwidth quota, not your credits. There is no credit multiplier for using proxies.

Additional options

Extract content only

Strip navigation, headers, footers, and sidebars before extraction. Useful when you only want the main content of a page and want to reduce noise.

{
  "urls": [{"url": "https://blog.example.com/article"}],
  "prompt": "Summarize this article",
  "extractContentOnly": true
}

Screenshots

Capture screenshots of scraped pages for debugging, archival, or visual monitoring.

{
  "urls": [{"url": "https://example.com"}],
  "screenshot": true,
  "fullPageScreenshot": true
}

screenshot: true captures the visible viewport. fullPageScreenshot: true captures the entire scrollable page. The screenshot URLs are returned in result.screenshots and in each item's screenshotUrl field.

Authenticated scraping

Pass session cookies to access pages behind a login. Get the cookies from your browser's DevTools after logging in manually, then include them in your request.

{
  "urls": [{"url": "https://app.example.com/dashboard"}],
  "prompt": "Extract the account summary",
  "cookies": "session_id=abc123; auth_token=xyz789"
}

Standard cookie format (name=value; name2=value2) and Chrome DevTools paste format both work. Cookies are passed ephemerally to the browser worker and never stored.

Batch scraping

When you have a list of URLs to process, the batch endpoint handles up to 50 at a time in parallel. Each URL runs in its own independent worker.

Endpoint: POST /api/batch/scrape

import requests
import time

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.spidra.io/api"
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}

urls = [
    "https://example.com/product/1",
    "https://example.com/product/2",
    "https://example.com/product/3",
]

# Submit the batch
response = requests.post(
    f"{BASE_URL}/batch/scrape",
    headers=HEADERS,
    json={
        "urls": urls,
        "prompt": "Extract the product name, price, and availability",
        "output": "json",
    }
)
batch_id = response.json()["batchId"]

# Poll until complete
while True:
    status = requests.get(
        f"{BASE_URL}/batch/scrape/{batch_id}",
        headers=HEADERS
    ).json()

    if status["status"] in ("completed", "failed", "partial"):
        break

    time.sleep(3)

# Process results
for item in status["items"]:
    if item["status"] == "completed":
        print(f"{item['url']}: {item['result']['content']}")
    else:
        print(f"Failed: {item['url']} — {item['error']}")

The batch response includes a status for the overall batch and an items array with one entry per URL. Each item has its own status, result, and error so you can see exactly which URLs succeeded and which failed.

Credits are reserved upfront when you submit and reconciled per item when processing completes. If a URL fails, credits for that item are returned.

Batch with structured output

Everything that works in single scrape works in batch. Pass a schema and every item in the batch returns data matching that shape:

requests.post(
    f"{BASE_URL}/batch/scrape",
    headers=HEADERS,
    json={
        "urls": urls,
        "prompt": "Extract the product details",
        "schema": {
            "type": "object",
            "required": ["name", "price"],
            "properties": {
                "name":      {"type": "string"},
                "price":     {"type": ["number", "null"]},
                "currency":  {"type": ["string", "null"]},
                "available": {"type": ["boolean", "null"]}
            }
        }
    }
)

Managing batches

Beyond submitting and polling, the batch API has a few more endpoints worth knowing:

The retry endpoint is particularly useful for large batches where a handful of items fail due to transient issues. You do not need to resubmit the full batch, just the failures.

Crawling

Batch scraping works when you already know the URLs. Crawling is for when you want Spidra to discover pages for you.

You give it a starting URL, describe which pages to follow, and describe what to extract from each one. Spidra loads the base URL, finds links matching your crawl instruction, visits each one up to your maxPages limit, and applies your transform instruction to every page it visits.

Endpoint: POST /api/crawl

response = requests.post(
    f"{BASE_URL}/crawl",
    headers=HEADERS,
    json={
        "baseUrl": "https://docs.example.com",
        "crawlInstruction": "Follow all documentation pages. Skip changelog and login pages.",
        "transformInstruction": "Extract the page title and full body text as clean Markdown. Preserve all headings and code examples.",
        "maxPages": 20,
        "useProxy": False
    }
)
job_id = response.json()["jobId"]

Three fields are required: baseUrl, crawlInstruction, and transformInstruction. Everything else is optional.

maxPages defaults to 5 and goes up to 20. The crawl discovers links from the base URL first, then works through them in order of discovery.

Poll GET /api/crawl/{jobId} for status. When complete, results are available through several endpoints:

The extract endpoint is worth highlighting. If you crawl a site and later decide you want to extract different fields, you can run a new extraction on the cached pages without making a single new browser request. That saves time and credits.

A complete crawl example

import requests
import time
import json

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.spidra.io/api"
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}

# Submit the crawl
job = requests.post(
    f"{BASE_URL}/crawl",
    headers=HEADERS,
    json={
        "baseUrl": "https://blog.example.com",
        "crawlInstruction": "Follow all blog post pages. Skip tag pages, author pages, and the homepage.",
        "transformInstruction": "Extract the article title, author, publish date, and full body text.",
        "maxPages": 15
    }
).json()

job_id = job["jobId"]
print(f"Crawl job started: {job_id}")

# Poll until complete
while True:
    status = requests.get(
        f"{BASE_URL}/crawl/{job_id}",
        headers=HEADERS
    ).json()

    print(f"Status: {status['status']}")

    if status["status"] == "completed":
        break
    elif status["status"] == "failed":
        raise Exception("Crawl failed")

    time.sleep(5)

# Fetch all crawled pages
pages = requests.get(
    f"{BASE_URL}/crawl/{job_id}/pages",
    headers=HEADERS
).json()

# Save as JSONL
with open("crawl_results.jsonl", "w") as f:
    for page in pages["pages"]:
        f.write(json.dumps({
            "url": page["url"],
            "data": page["data"]
        }) + "\n")

print(f"Saved {len(pages['pages'])} pages")

Monitoring and logs

The Spidra API keeps a log of every scrape job you run. This is useful for debugging, auditing, and understanding your credit consumption.

# List recent scrape logs
logs = requests.get(
    f"{BASE_URL}/scrape-logs",
    headers=HEADERS
).json()

for log in logs["data"]:
    print(f"{log['started_at']} — {log['status']} — {log['latency_ms']}ms — {log['tokens_used']} tokens")

# Get full details of a specific log
log_detail = requests.get(
    f"{BASE_URL}/scrape-logs/{log['uuid']}",
    headers=HEADERS
).json()

Usage statistics

Track your credit consumption over time:

usage = requests.get(
    f"{BASE_URL}/account/usage",
    headers=HEADERS
).json()

print(usage)

This returns time-series data covering requests, tokens, crawls, and credit consumption over a configurable period.

Putting it all together: a real pipeline

Here is a complete example that combines scraping, batch processing, and structured output into a pipeline that collects job listings from multiple pages and saves them to a JSONL file:

import requests
import time
import json

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.spidra.io/api"
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}

JOB_SCHEMA = {
    "type": "object",
    "required": ["title", "company", "location"],
    "properties": {
        "title":           {"type": "string"},
        "company":         {"type": "string"},
        "location":        {"type": ["string", "null"]},
        "remote":          {"type": ["boolean", "null"]},
        "salary_min":      {"type": ["number", "null"]},
        "salary_max":      {"type": ["number", "null"]},
        "employment_type": {
            "type": ["string", "null"],
            "enum": ["full_time", "part_time", "contract", None]
        }
    }
}

def collect_job_urls(board_url):
    """Use forEach to collect job listing URLs from a board page."""
    response = requests.post(
        f"{BASE_URL}/scrape",
        headers=HEADERS,
        json={
            "urls": [{
                "url": board_url,
                "actions": [
                    {"type": "click", "value": "Accept cookies"},
                    {
                        "type": "forEach",
                        "value": "Find all job listing links",
                        "mode": "navigate",
                        "maxItems": 50,
                        "itemPrompt": "Extract job title, company, location, remote status, salary range, and employment type",
                        "pagination": {
                            "nextSelector": "a.next-page",
                            "maxPages": 3
                        }
                    }
                ]
            }],
            "output": "json",
            "schema": JOB_SCHEMA
        }
    )
    job_id = response.json()["jobId"]

    while True:
        status = requests.get(
            f"{BASE_URL}/scrape/{job_id}",
            headers=HEADERS
        ).json()

        if status["status"] == "completed":
            return status["result"]["content"]
        elif status["status"] == "failed":
            raise Exception(status["error"])

        time.sleep(3)

# Collect from multiple job boards
boards = [
    "https://jobs.example.com/engineering",
    "https://careers.anothersite.com/remote",
]

all_jobs = []
for board in boards:
    print(f"Collecting from {board}...")
    jobs = collect_job_urls(board)
    if isinstance(jobs, list):
        all_jobs.extend(jobs)
    print(f"  Got {len(jobs) if isinstance(jobs, list) else 0} jobs")

# Save results
with open("jobs.jsonl", "w") as f:
    for job in all_jobs:
        f.write(json.dumps(job) + "\n")

print(f"\nTotal: {len(all_jobs)} jobs saved to jobs.jsonl")

Error handling

Wrap your API calls properly and handle the cases that actually come up in production.

import requests

def safe_scrape(url, prompt):
    try:
        response = requests.post(
            f"{BASE_URL}/scrape",
            headers=HEADERS,
            json={
                "urls": [{"url": url}],
                "prompt": prompt,
                "output": "json"
            }
        )

        if response.status_code == 401:
            raise Exception("Invalid API key. Check your x-api-key header.")

        if response.status_code == 403:
            raise Exception("Credits exhausted or plan limit reached.")

        if response.status_code == 429:
            raise Exception("Rate limit hit. Wait before retrying.")

        response.raise_for_status()
        return response.json()["jobId"]

    except requests.exceptions.ConnectionError:
        raise Exception("Could not connect to the Spidra API.")

For polling loops, always handle the failed status and check ai_extraction_failed in the result:

if status["status"] == "completed":
    result = status["result"]

    if result.get("ai_extraction_failed"):
        # AI extraction failed, content is raw Markdown fallback
        print("AI extraction failed, using raw content")
        content = result["data"][0]["markdownContent"]
    else:
        content = result["content"]

API reference summary

What next

You now have a working understanding of every part of the Spidra API. Here are the natural next steps depending on what you are building:

If you want to go deeper on browser actions and forEach, read the Browser Actions Guide in the docs. It covers every option for each action type with real examples.

If you are building something that needs guaranteed output shapes, read the Structured Output Guide for full details on schemas, nullable fields, Zod and Pydantic integration, and schema limits.

If you are using an SDK in a specific language, each one has its own guide: Node.js, Python, Go, PHP, Ruby, Rust, .NET, Elixir, Java, and Swift.

Get your API key at app.spidra.io. The free plan has 300 credits and no card required.