One platform to search, scrape, and crawl the web
Call the REST API directly, use the Node.js or Python SDK, or connect an AI agent over MCP. Spidra handles rendering, proxies, CAPTCHAs, and AI extraction so you never have to.
300 free credits · No credit card required
"urls": [{ "url": "eventbrite.com/houston" }],
"prompt": "Get organizers with emails",
"output": "json",
"useProxy": true,
"proxyCountry": "us",
"screenshot": true,
"extractContentOnly": true
}
Search
POST /api/search
Don't have a URL yet? Search the web, news, images, videos, research, and GitHub for a plain-text query and get real results back.
Scrape
POST /api/scrape
Turn any URL into structured data by passing a prompt to extract exactly what you need and let AI handle the rest.
Crawl
POST /api/crawl
Give a starting URL and instructions. Spidra discovers pages, crawls them, and returns transformed data. Re-run AI extraction on any existing crawl with a new transform instruction.
REST API and official SDKs
Three calls. Submit, poll, get data. Use the REST API directly, or the Node.js and Python SDKs. See all SDKs.
{
"status": "completed",
"result": {
"content": [
{
"event_name": "Houston's Funniest",
"organizer": "The Riot Comedy Club",
"organizer_url": "theriothtx.com"
},
{
"event_name": "Stand Up Comedy Night",
"organizer": "Comedy Showcase",
"organizer_url": "comedyshowcase.com"
}
]
}
}Give your AI assistant search and scraping tools
Connect Claude, Cursor, or any MCP-compatible client to https://mcp.spidra.io/mcp and your API key. Nothing to install, nothing to host, no build step. Your agent gets search, scrape, crawl, and batch tools in one connection.
{
"mcpServers": {
"spidra": {
"type": "http",
"url": "https://mcp.spidra.io/mcp",
"headers": {
"Authorization": "Bearer spd_YOUR_API_KEY"
}
}
}
}From URL to clean data
Whether you need one data point or entire datasets at scale, the Spidra API just works.
"event": "Houston's Funniest Comedy Show",
"organizer": "The Riot Comedy Club",
"email": "[email protected]",
"phone": "(713) 264-8664",
"website": "theriothtx.com",
"facebook": "fb.com/theriothtx"
}
Built for any use case
From lead generation to dataset curation, the API extracts structured data from even the most complex websites.
Lead Generation
Turn any directory or marketplace into a lead database. Extract verified emails, phone numbers, and business details from Eventbrite, Google Maps, or any source.
Price Monitoring
Track competitor pricing across Amazon, Shopify stores, and e-commerce platforms. Get structured product data including prices, sizes, and stock availability.
Market Research
Analyze reviews, ratings, and sentiment across G2, Trustpilot, and Reddit. Get market cap, P/E ratios, and financial metrics from any public source.
Dataset Building
Build structured datasets from any source at scale, including research papers from arXiv, job listings from LinkedIn, and property data from real estate platforms.
Real Estate
Extract property listings with pricing, photos, and agent details from Zillow, Redfin, or any MLS. Filter by location, price range, and property type.
Research & Academia
Extract clinical trial results, paper abstracts, and citation data from PubMed, Google Scholar, and academic databases for systematic reviews.
Scrape every item, not just the page
Most of the data you actually want is hidden behind clicks: product variant drawers, detail modals, accordion panels. forEach finds every matching element, interacts with each one, and returns a structured array in a single request.
One instruction.
Every item extracted.
Describe which elements to find in plain text. Spidra locates each one, interacts with it, runs your extraction prompt on whatever appears, and returns a single array.
{
"urls": [
{
"url": "https://example.com/products/wireless-headphones",
"actions": [
{
"type": "forEach",
"observe": "Find all color and size variant buttons",
"mode": "click",
}
]
}
],
"prompt": "Extract variant name, price, and stock status",
"output": "json"
}Everything built in
No external proxy subscriptions. No CAPTCHA solver accounts. No browser infrastructure to maintain. It's all in the API.
AI extraction
Describe what you want in plain text. The API uses AI to extract exactly those fields without needing any CSS selectors.
Browser rendering & actions
Full headless browser renders SPAs and executes sequences of actions like clicking, typing, scrolling, and iterating over element lists with forEach pagination.
CAPTCHA solving
Spidra automatically detects and solves popular CAPTCHAs including reCAPTCHA v2/v3, hCaptcha, and Cloudflare Turnstile.
Proxy rotation
Enterprise-grade residential proxies rotate per request. Target by region: US, EU, Asia, or global.
Cookie authentication
Pass session cookies to scrape pages behind logins like dashboards, member portals, and authenticated directories.
Multiple output formats
Get data in whatever format your pipeline needs, whether that's JSON, Markdown, or full-page screenshots.
FAQ
Common questions about the Spidra scraping API.
