Spidra batch scraping API: how to scrape 50 URLs in parallel

When you need data from more than a handful of pages, scraping one URL at a time stops making sense quickly. You are waiting for each request to complete before starting the next one, and the time adds up fast. If each page takes five seconds to scrape and you have two hundred URLs, that is over sixteen minutes of waiting.

The Spidra batch scraping API processes up to 50 URLs in parallel in a single request. Each URL runs in its own independent worker. You get all the results back together when the batch finishes, with a per-item status so you can see exactly which URLs succeeded and which did not.

This guide walks through everything about the batch API: how it works, how to use it in Python and Node.js, how to handle failures, and how to process URL lists larger than 50.

How batch scraping works

The batch endpoint follows the same async job pattern as single scraping. You submit a list of URLs and get a batchId back immediately. The URLs start processing in parallel right away. You then poll the status endpoint every few seconds until the batch is complete and the results are ready.

POST /api/batch/scrape  →  { batchId: "abc123" }
GET  /api/batch/scrape/abc123  →  { status: "running", ... }
GET  /api/batch/scrape/abc123  →  { status: "completed", items: [...] }

The status moves through these stages:

• queued — returned immediately on submit; workers are being allocated
• pending — batch accepted, not yet running
• running — URLs are being processed in parallel
• completed — all URLs finished (check failed_count for per-item failures)
• failed — the batch itself failed
• cancelled — cancelled via the cancel endpoint

On credits: when you submit a batch, 2 credits per URL are reserved upfront. If a URL fails for any reason, the credits for that item are returned. AI token costs, CAPTCHA solving, and proxy usage are calculated per item once it finishes, the same as a regular scrape.

Your first batch request

The only required field is urls, which takes an array of plain strings. Unlike the single scrape endpoint where URLs are objects, batch URLs are just strings.

cURL

# Submit the batch
curl -X POST https://api.spidra.io/api/batch/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/product/1",
      "https://example.com/product/2",
      "https://example.com/product/3"
    ]
  }'

Response:

{
  "status": "queued",
  "batchId": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Batch job queued. Poll /api/batch/scrape/550e8400... for results."
}

Poll until complete:

curl https://api.spidra.io/api/batch/scrape/550e8400-e29b-41d4-a716-446655440000 \
  -H "x-api-key: YOUR_API_KEY"

When the batch finishes:

{
  "status": "completed",
  "batchId": "550e8400-e29b-41d4-a716-446655440000",
  "totalUrls": 3,
  "completedCount": 3,
  "failedCount": 0,
  "items": [
    {
      "url": "https://example.com/product/1",
      "status": "completed",
      "result": {"product_name": "Example Product", "price": 49.99},
      "screenshotUrl": null,
      "error": null
    }
  ]
}

Each item in the items array has its own status, result, and error. A failure on one URL does not affect the others. They all run independently.

Python: raw REST API

import requests
import time

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.spidra.io/api"
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}

urls = [
    "https://www.toyota.com/camry",
    "https://www.ford.com/cars/mustang/",
    "https://www.kia.com/us/en/sportage",
]

# Submit the batch
response = requests.post(
    f"{BASE_URL}/batch/scrape",
    headers=HEADERS,
    json={
        "urls": urls,
        "prompt": "Extract the vehicle name and starting price",
        "output": "json",
    }
)
response.raise_for_status()
batch_id = response.json()["batchId"]
print(f"Batch submitted: {batch_id}")

# Poll until complete
while True:
    status = requests.get(
        f"{BASE_URL}/batch/scrape/{batch_id}",
        headers=HEADERS
    ).json()

    print(f"Status: {status['status']} ({status.get('completedCount', 0)}/{status.get('totalUrls', 0)})")

    if status["status"] in ("completed", "failed", "cancelled"):
        break

    time.sleep(3)

# Process results
for item in status["items"]:
    if item["status"] == "completed" and item["result"]:
        # item["result"] is the extracted JSON directly
        print(f"\n✓ {item['url']}")
        print(item["result"])
    else:
        print(f"\n✗ {item['url']}: {item.get('error')}")

Python SDK

The Python SDK's batch.run_sync() handles submission and polling for you. Note that batch URLs are plain strings, not ScrapeUrl objects. Per-URL browser actions are not supported in batch mode — for pages that require interaction before extraction, use the single scrape endpoint.

from spidra import SpidraClient, BatchScrapeParams
import os

spidra = SpidraClient(api_key=os.environ["SPIDRA_API_KEY"])

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=[
        "https://www.toyota.com/camry",
        "https://www.ford.com/cars/mustang/",
        "https://www.kia.com/us/en/sportage",
    ],
    prompt="Extract the vehicle name and starting price",
    output="json",
))

print(f"{batch.completed_count}/{batch.total_urls} completed")

for item in batch.items:
    if item.status == "completed":
        print(f"✓ {item.url} — {item.result}")
    else:
        print(f"✗ {item.url} — {item.error}")

Node.js SDK

The Node.js SDK works identically to Python but with camelCase properties.

import { SpidraClient } from 'spidra'

const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY! })

const batch = await spidra.batch.run({
  urls: [
    'https://www.toyota.com/camry',
    'https://www.ford.com/cars/mustang/',
    'https://www.kia.com/us/en/sportage',
  ],
  prompt: 'Extract the vehicle name and starting price',
  output: 'json',
})

console.log(`${batch.completedCount}/${batch.totalUrls} completed`)

for (const item of batch.items) {
  if (item.status === 'completed') {
    console.log(`✓ ${item.url}`)
  } else {
    console.error(`✗ ${item.url} — ${item.error}`)
  }
}

AI extraction across a batch

Without a prompt, Spidra still applies AI extraction but with no specific instructions — the output shape varies and you cannot rely on consistent field names. Add a prompt to control what gets extracted and use output="json" to enforce structured output.

Note: the Python SDK requires prompt as a positional argument. The no-prompt path is only available via the REST API directly.

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=[
        "https://www.toyota.com/camry",
        "https://www.ford.com/cars/mustang/",
        "https://www.kia.com/us/en/sportage",
    ],
    prompt="Extract the vehicle name and starting price",
    output="json",
))

for item in batch.items:
    if item.status == "completed":
        print(item.url, item.result)

Each item returns its own extracted JSON. The AI runs independently on each page.

Enforcing output shape with JSON schema

When you need every item in the batch to come back in exactly the same shape, use a schema. If you are new to writing schemas, Spidra's structured output docs cover every option in detail. Required fields always appear in the output, as null if the page does not have that value. The structure never varies between items regardless of how different the source pages are.

PRODUCT_SCHEMA = {
    "type": "object",
    "required": ["name", "price", "in_stock"],
    "properties": {
        "name":       {"type": "string"},
        "price":      {"type": ["number", "null"]},
        "currency":   {"type": ["string", "null"]},
        "in_stock":   {"type": ["boolean", "null"]},
        "sku":        {"type": ["string", "null"]},
        "rating":     {"type": ["number", "null"]},
        "review_count": {"type": ["number", "null"]},
    }
}

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=product_urls,
    prompt="Extract the product details",
    output="json",
    schema=PRODUCT_SCHEMA,
))

for item in batch.items:
    if item.status == "completed":
        product = item.result  # always matches PRODUCT_SCHEMA
        print(product["name"], product["price"], product["in_stock"])

This is the pattern to use when batch results feed directly into a database or downstream service. No validation step required. No surprises on field names.

One note on item.result types: when you use output="json" without a schema, the AI may return a list. When you use a schema with "type": "object", item.result is always a dict matching your schema. Use a schema whenever you need to access fields directly with item.result["name"].

Proxy routing and geo-targeting

Pass use_proxy and proxy_country to route every URL in the batch through a residential proxy. Useful when scraping e-commerce sites that show different prices by region, or when the target sites block cloud IP ranges.

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=amazon_de_urls,
    prompt="Extract the product name and price in EUR",
    output="json",
    use_proxy=True,
    proxy_country="de",
))

All proxy options that work in single scraping work in batch: two-letter country codes ("us", "de", "gb"), "eu" for rotating across EU countries, or omit for global rotation.

Authenticated pages

Batch scraping supports session cookies for pages behind a login. Pass the cookies as a raw cookie header string, the same format as single scraping.

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=[
        "https://app.example.com/invoices/2026-01",
        "https://app.example.com/invoices/2026-02",
        "https://app.example.com/invoices/2026-03",
    ],
    prompt="Extract invoice number, amount, and status",
    output="json",
    cookies="session=abc123; auth_token=xyz789",
))

The cookies are applied to every URL in the batch.

Screenshots

Enable screenshot or full_page_screenshot to capture every page in the batch. The screenshot URL for each item is available at item.screenshot_url (Python SDK) or item.screenshotUrl (REST API).

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=competitor_pages,
    prompt="Extract the page title",
    screenshot=True,
    full_page_screenshot=True,
))

for item in batch.items:
    if item.status == "completed":
        print(item.url, item.screenshot_url)

Handling failures and retrying

Real-world batches have failures. A site times out. A page throws a 403. A CAPTCHA does not resolve in time. The batch API handles this gracefully: a failed item does not affect the others, and you can retry just the failures without resubmitting the whole batch.

Checking what failed

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=urls,
    prompt="Extract the product name and price",
    output="json",
))

succeeded = [i for i in batch.items if i.status == "completed"]
failed = [i for i in batch.items if i.status != "completed"]

print(f"{len(succeeded)} succeeded, {len(failed)} failed")
for item in failed:
    print(f"  ✗ {item.url}: {item.error}")

Retrying failed items

If some items fail, retry() re-queues only those items without touching the ones that already succeeded. You do not pay again for the successful ones.

There is an important limitation in the Python SDK: calling multiple *_sync() methods on the same SpidraClient instance crashes with RuntimeError: Event loop is closed. Each *_sync() call uses asyncio.run() internally, which creates and closes an event loop. The second call finds a closed loop and fails.

This means the submit_sync() + get_sync() polling pattern does not work in Python. You have two working alternatives:

Option A — Use run_sync() and check failed_count afterward. This is the simplest approach for most cases:

batch = spidra.batch.run_sync(BatchScrapeParams(
    urls=urls,
    prompt="Extract the product name and price",
    output="json",
))

if batch.failed_count > 0:
    print(f"{batch.failed_count} items failed:")
    for item in batch.items:
        if item.status != "completed":
            print(f"  ✗ {item.url}: {item.error}")

Option B — Use the REST API directly for submit, poll, and retry. This avoids the event loop issue entirely:

import requests, time

# Submit
resp = requests.post(
    f"{BASE_URL}/batch/scrape",
    headers=HEADERS,
    json={
        "urls": urls,
        "prompt": "Extract the product name and price",
        "output": "json",
    }
)
batch_id = resp.json()["batchId"]

# Poll
while True:
    status = requests.get(
        f"{BASE_URL}/batch/scrape/{batch_id}",
        headers=HEADERS
    ).json()
    if status["status"] in ("completed", "failed", "cancelled"):
        break
    time.sleep(3)

# Retry failed items
if status.get("failedCount", 0) > 0:
    requests.post(
        f"{BASE_URL}/batch/scrape/{batch_id}/retry",
        headers=HEADERS
    )

In Node.js, the submit() + get() + retry() pattern works correctly because Node.js does not have this event loop limitation:

import time from 'node:timers/promises'

const queued = await spidra.batch.submit({
  urls,
  prompt: 'Extract the product name and price',
  output: 'json',
})
const batchId = queued.batchId

let batch
while (true) {
  batch = await spidra.batch.get(batchId)
  if (['completed', 'failed', 'cancelled'].includes(batch.status)) break
  await time.setTimeout(3000)
}

if (batch.failedCount > 0) {
  await spidra.batch.retry(batchId)
}

Cancelling a batch

If you need to stop a running batch, cancelling it returns credits for any items that have not started processing yet.

result = spidra.batch.cancel_sync(batch_id)
print(f"Cancelled {result.cancelled_items} items, refunded {result.credits_refunded} credits")

Processing URL lists larger than 50

The batch endpoint caps at 50 URLs per request. For larger lists, chunk them and process sequentially. The lead generation scraping guide shows a real example of this pattern against Yellow Pages across multiple pages.

import os, json
from spidra import SpidraClient, BatchScrapeParams

spidra = SpidraClient(api_key=os.environ["SPIDRA_API_KEY"])

def batch_scrape_all(
    urls: list[str],
    prompt: str,
    schema: dict | None = None,
    chunk_size: int = 50,
) -> list[dict]:
    results = []
    total_batches = -(-len(urls) // chunk_size)  # ceiling division

    for i in range(0, len(urls), chunk_size):
        chunk = urls[i:i + chunk_size]
        batch_num = i // chunk_size + 1
        print(f"Batch {batch_num}/{total_batches}: {len(chunk)} URLs")

        params = BatchScrapeParams(
            urls=chunk,
            prompt=prompt,
            output="json",
        )
        if schema:
            params.schema = schema

        batch = spidra.batch.run_sync(params)

        for item in batch.items:
            if item.status == "completed":
                results.append({"url": item.url, "data": item.result})
            else:
                print(f"  ✗ Failed: {item.url}")

        print(f"  {batch.completed_count}/{batch.total_urls} succeeded")

    return results

# Use it
urls = [f"https://example.com/product/{i}" for i in range(1, 201)]

results = batch_scrape_all(
    urls=urls,
    prompt="Extract product name, price, and availability",
    schema={
        "type": "object",
        "required": ["name", "price"],
        "properties": {
            "name":     {"type": "string"},
            "price":    {"type": ["number", "null"]},
            "available": {"type": ["boolean", "null"]},
        }
    }
)

# Save as JSONL
with open("products.jsonl", "w") as f:
    for record in results:
        f.write(json.dumps(record) + "\n")

print(f"\nSaved {len(results)} products to products.jsonl")

Listing batch history

from spidra import BatchListParams

page = spidra.batch.list_sync(BatchListParams(page=1, limit=20))

for job in page.jobs:
    print(f"{job.uuid}  {job.status}  {job.completed_count}/{job.total_urls}")

Via the REST API directly:

curl https://api.spidra.io/api/batch/scrape \
  -H "x-api-key: YOUR_API_KEY"

A complete real-world example: e-commerce price monitoring

This pulls current prices from a list of competitor product pages, compares them to the previous run, and reports what changed. It follows the same pattern used in the market research use case.

import os, json, time
from datetime import datetime, timezone
from pathlib import Path
from spidra import SpidraClient, BatchScrapeParams

spidra = SpidraClient(api_key=os.environ["SPIDRA_API_KEY"])

PRICE_SCHEMA = {
    "type": "object",
    "required": ["name", "price"],
    "properties": {
        "name":     {"type": "string"},
        "price":    {"type": ["number", "null"]},
        "currency": {"type": ["string", "null"]},
        "in_stock": {"type": ["boolean", "null"]},
    }
}

def load_previous(path="data/prices.json") -> dict:
    p = Path(path)
    return json.loads(p.read_text()) if p.exists() else {}

def save_current(data: dict, path="data/prices.json"):
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    Path(path).write_text(json.dumps(data, indent=2))

def check_prices(urls: list[str]) -> dict:
    results = {}

    for i in range(0, len(urls), 50):
        chunk = urls[i:i + 50]
        batch = spidra.batch.run_sync(BatchScrapeParams(
            urls=chunk,
            prompt="Extract the product name, current price as a number, currency code, and whether it is in stock",
            output="json",
            schema=PRICE_SCHEMA,
            use_proxy=True,
        ))

        for item in batch.items:
            if item.status == "completed" and item.result:
                results[item.url] = item.result

    return results

def find_changes(previous: dict, current: dict, threshold_pct=3.0) -> list[dict]:
    changes = []

    for url, data in current.items():
        if not data or data.get("price") is None:
            continue

        prev = previous.get(url, {})
        prev_price = prev.get("price")
        curr_price = data["price"]

        if prev_price is None or prev_price == 0:
            continue

        change_pct = ((curr_price - prev_price) / prev_price) * 100

        if abs(change_pct) >= threshold_pct:
            changes.append({
                "url":        url,
                "name":       data.get("name", ""),
                "prev_price": prev_price,
                "curr_price": curr_price,
                "change_pct": round(change_pct, 2),
                "currency":   data.get("currency", ""),
                "direction":  "up" if change_pct > 0 else "down",
            })

    return sorted(changes, key=lambda x: abs(x["change_pct"]), reverse=True)

# Load URLs from config
urls = Path("config/competitor_urls.txt").read_text().splitlines()
urls = [u.strip() for u in urls if u.strip()]

print(f"Checking {len(urls)} URLs...")
previous = load_previous()
current = check_prices(urls)
save_current(current)

changes = find_changes(previous, current)

if changes:
    print(f"\n{len(changes)} significant price changes:")
    for c in changes:
        direction = "↑" if c["direction"] == "up" else "↓"
        print(
            f"  {direction} {c['name']}: "
            f"{c['currency']}{c['prev_price']:.2f} → "
            f"{c['currency']}{c['curr_price']:.2f} "
            f"({c['change_pct']:+.1f}%)"
        )
else:
    print("\nNo significant price changes.")

print(f"\nDone. {len(current)} URLs checked.")

Batch API reference

All batch parameters