The Spidra API now includes a comprehensive set of crawling endpoints. You can submit crawl jobs, track progress, and retrieve results — all from your own code.
Submit a crawl by providing a starting URL, a natural-language instruction describing which pages to discover, and how content should be extracted:
curl --request POST \
--url https://api.spidra.io/api/crawl \
--header 'Content-Type: application/json' \
--header 'x-api-key: YOUR_API_KEY' \
--data '{
"baseUrl": "https://example.com/blog",
"crawlInstruction": "Crawl all blog post pages",
"transformInstruction": "Extract title, author, date, and content",
"maxPages": 5
}'
Spidra handles page discovery, navigation, captcha solving, and transformation automatically. The crawl runs asynchronously — you get back a job ID and can poll for progress or results:
{
"status": "completed",
"progress": {
"pagesCrawled": 5,
"maxPages": 10
},
"result": [
{
"url": "https://example.com/blog/post-1",
"status": "success",
"data": {
"title": "First Post",
"author": "John",
"date": "2025-01-01"
}
}
]
}
The API also includes endpoints for fetching crawled pages independently, viewing job configuration details, and listing crawl history with pagination. The API uses the same crawling engine as the Playground, including automatic CAPTCHA handling, optional stealth mode with proxy rotation, and retry logic for failed pages.
Read the full blog here.
