If you have ever scraped a big list of URLs, you write a loop, add some retry logic, try to figure out which ones failed and which ones succeeded, and by the end of it half your code is just plumbing. None of that logic is unique to your product, you just end up writing it over and over.
We built batch scraping to take all of that off your plate. You hand us the list, we handle everything else.
Send up to 50 URLs in one request and Spidra processes them all in parallel. You get a batchId back immediately, and you poll that to check progress:
curl --request POST \
--url https://api.spidra.io/api/batch/scrape \
--header 'Content-Type: application/json' \
--header 'x-api-key: YOUR_API_KEY' \
--data '{
"urls": [
"https://example.com/product-1",
"https://example.com/product-2",
"https://example.com/product-3"
],
"prompt": "Extract product name, price, and availability",
"output": "json"
}'
When you poll the status, you can see exactly where each URL is and what came back from it:
{
"batchId": "b_01jrmxyz...",
"status": "running",
"totalItems": 3,
"completedItems": 2,
"failedItems": 0,
"items": [
{
"url": "https://example.com/product-1",
"status": "completed",
"data": { "name": "Widget Pro", "price": "$29.99", "available": true },
"creditsUsed": 2
},
{
"url": "https://example.com/product-2",
"status": "completed",
"data": { "name": "Widget Lite", "price": "$14.99", "available": false },
"creditsUsed": 2
},
{
"url": "https://example.com/product-3",
"status": "running"
}
]
}
Each URL lives in its own little world inside the batch. If one fails, the others keep going. The batch only reaches completed once every single item has settled, one way or another.
When some items fail, you do not have to redo the whole thing. There is a retry endpoint that picks up only the failed items and queues them again. Everything that already worked stays exactly as it was. You call it and Spidra figures out what still needs doing.
curl --request POST \
--url https://api.spidra.io/api/batch/scrape/BATCH_ID/retry \
--header 'x-api-key: YOUR_API_KEY'
If you want to stop a batch early, you can cancel it. Any URLs that have not started yet get cancelled, and their reserved credits come back to your account automatically. The ones already in progress will finish and you only pay for what actually ran.
curl --request DELETE \
--url https://api.spidra.io/api/batch/scrape/BATCH_ID \
--header 'x-api-key: YOUR_API_KEY'
On credits: when you submit a batch, 2 credits per URL are reserved upfront. If a URL fails, you are not charged for it. AI token costs, CAPTCHA solving, and proxy usage are all calculated per item once it finishes, same as a regular scrape.
Everything you can do in a regular scrape works here too. You can pass a schema and every URL in the batch returns data in the same shape. You can use stealth mode with proxy rotation, target a specific country with proxyCountry, pass cookies for pages that need a login session, and take viewport or full-page screenshots. All of it applies uniformly across every URL in the batch.
Read the full API reference in the docs.
