A large portion of valuable web data resides on dashboards, internal tools, admin panels, analytics views, private documentation, customer portals, CRM systems and is not publicly accessible to traditional scrapers.

Until now, Spidra was limited to publicly available pages. If the content required signing in, there was no clean way to access it.

Today, we are excited to introduce authenticated scraping and crawling, which helps remove that limitation by allowing Spidra to operate inside a logged-in session, just like a real user’s browser.

How authenticated scraping and crawling work

, which is how browsers maintain login sessions across requests.

Instead of asking you to reimplement authentication flows or handle credentials directly, Spidra relies on cookies that already exist after you log in normally.

To do this, simply log in to the target website using your browser and open browser DevTools to copy the relevant cookies.

crucnbase-cookie.webp

You then provide those cookies to Spidra either as name-value pairs so that Spidra can inject them into its browser session before navigation:

auth-scrape.webp

Or by simply pasting them directly from your browser’s DevTools. Spidra automatically parses and applies them to the session before navigation.

raw-edit.webp

From that point on, all requests behave as if they are coming from your authenticated browser session.

 In some cases, including cookies related to prior verification steps (such as clearance or challenge cookies) can significantly reduce or eliminate CAPTCHA challenges during scraping or crawling. This can lead to faster runs and fewer interruptions, especially on heavily protected sites.

Authenticated scraping and crawling in the Spidra API

Authenticated access is available not only in the Spidra Playground but also through the API, making it easy to integrate into automated workflows.

You can pass cookies directly when submitting a scrape job:

curl --request POST \
  --url https://api.spidra.io/api/scrape \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: YOUR_API_KEY' \
  --data '{
    "urls": [
      { "url": "https://app.example.com/dashboard" }
    ],
    "cookies": "session_id=abc123; auth_token=xyz789"
  }'

 The API supports both standard cookie strings and raw DevTools pastes. Spidra automatically detects the format and parses it accordingly.

Authenticated crawling works the same way, with cookies applied at the start of the crawl:

curl --request POST \
  --url https://api.spidra.io/api/crawl \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: YOUR_API_KEY' \
  --data '{
    "baseUrl": "https://app.example.com",
    "crawlInstruction": "Find all report pages",
    "transformInstruction": "Extract report data",
    "maxPages": 10,
    "cookies": "session_id=abc123; auth_token=xyz789"
  }'

One key difference between scraping and crawling is 

For scraping, the provided cookies apply to the specific request being executed. For crawling, the same authenticated session is preserved across all discovered pages. This ensures consistent access as Spidra navigates links, pagination, and internal sections without re-authentication issues.

Authenticated scraping and crawling unlock a wide range of workflows, including:

They are used transiently for the duration of a scrape or crawl and destroyed/discarded immediately afterward. Spidra does not persist session data or reuse cookies across jobs.

Users are responsible for ensuring they have legal authorization to access and extract any content. Clear guidance and disclaimers are provided in both the UI and documentation.

Introducing authenticated scraping and crawling

About two weeks ago, we announced the launch of 

 to make crawling work reliably on protected and JavaScript heavy sites.

Today, we are happy to announce the introduction of a comprehensive set of crawling endpoints in the Spidra API, enabling you to run, track, and manage crawl jobs directly from your own code.

If you prefer automation over UI, this release is for you.

 To use the Spidra API, you need a valid API key. You can create and manage your API keys from your dashboard. All API requests are authenticated using the 

This endpoint mirrors what happens in the Playground. You provide a starting URL, describe which pages should be discovered, and define how content should be extracted from each page. Spidra handles discovery, navigation, captcha solving, and transformation automatically.

curl --request POST \
  --url https://api.spidra.io/api/crawl \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: YOUR_API_KEY' \
  --data '{
    "baseUrl": "https://example.com/blog",
    "crawlInstruction": "Crawl all blog post pages",
    "transformInstruction": "Extract title, author, date, and content from each post",
    "maxPages": 5,
    "useProxy": false
  }'

The response confirms the job is queued and returns a job ID.

From this point on, the crawl runs asynchronously. You can submit the job and come back later to inspect progress or results.

Full parameter details and edge cases are documented in the crawling section of the 

, where you can also test requests using our interactive docs.

Once a crawl is running, the job status endpoint becomes the primary way to understand what is happening.

curl --request GET \
  --url https://api.spidra.io/api/crawl/abc-123 \
  --header 'x-api-key: YOUR_API_KEY'

This endpoint serves two purposes. While the crawl is active, it reports progress such as how many pages have been visited so far. When the crawl completes, it returns the extracted results for each page.

Each page is returned as a separate object. This makes it easy to reason about partial successes and failures without rerunning the entire crawl.

For workflows that need more control, crawled pages can also be fetched independently.

curl --request GET \
  --url https://api.spidra.io/api/crawl/abc-123/pages \
  --header 'x-api-key: YOUR_API_KEY'

This endpoint is useful when you want to process results incrementally, retry individual pages, or store page-level data in your own systems. It exposes the same page objects used internally by the job status endpoint, without coupling them to job progress.

Sometimes you need to understand how a crawl was configured rather than what it returned.

The job details endpoint exposes the original instructions, limits, and usage information associated with a crawl job.

curl --request GET \
  --url https://api.spidra.io/api/crawl/job/abc-123 \
  --header 'x-api-key: YOUR_API_KEY'

This is helpful for auditing, debugging, and tracking how crawling fits into your overall usage.

To support longer-running or repeated workflows, we also added an endpoint for listing crawl history.

curl --request GET \
  --url 'https://api.spidra.io/api/crawl/history?page=1&limit=10' \
  --header 'x-api-key: YOUR_API_KEY'

This endpoint returns a paginated view of past crawl jobs along with basic status and usage information.

It is useful for monitoring activity, understanding credit usage, and building simple reporting around your crawls.

Additional query parameters are available for pagination and filtering. You can find the full list of supported parameters in the 

, where you can also test different combinations using the interactive docs.

The API uses the same crawling engine as the Playground. That includes automatic handling of CAPTCHA, optional stealth mode with proxy rotation, and retry logic for failed pages.

Introducing Crawling via the Spidra API

Crawling protected websites just got easier.

Spidra’s crawler can now solve captchas automatically and run in stealth mode using residential proxy rotation. This reduces blocks, improves reliability, and makes it possible to crawl sites that were previously difficult to access.

When the crawler encounters a captcha, it detects the type and solves it automatically in the background. No manual steps are required.

Some websites also block or limit repeated requests from the same IP address. When stealth mode is enabled, traffic is routed through rotating residential proxies to reduce detection and avoid rate limits during larger crawls.

 and test these features live in your dashboard. If you have questions or feedback, feel free to 

Captcha solving and stealth mode are now live for crawling

You can switch between light and dark mode anytime from the dashboard. The experience stays exactly the same while scraping, crawling, and reviewing results. It is simply easier on the eyes during long sessions.

darkmode-spidra-overview.webp

This update is part of our ongoing beta improvements as we continue refining Spidra based on real usage and feedback.

Dark mode Is now available in Spidra

Until now, Spidra focused on no-code scraping. This lets you extract clean content from specific URLs, even on complex sites protected by CAPTCHA or heavy JavaScript. 

Many users loved the simplicity, but one request kept coming up: 

“Can I crawl an entire website, not just individual pages?”

In real workflows, you rarely know every URL you need upfront.

Whether you’re auditing a school website, analyzing competitor content, or building datasets for research, manually listing pages doesn’t scale. Crawling solves this by automatically discovering pages and extracting content at scale.

With AI-powered crawling, Spidra now handles both discovery and extraction in one flow.

Spidra’s crawler combines browser-level rendering with AI-driven page selection and transformation.

crawl-instructions.webp

crawl.webp

All of this happens inside the UI. No code required.

 and try crawling in the dashboard. You can also explore 

 to share feedback or discuss your use case.

Introducing AI-powered crawling

We’re excited to announce that Spidra is officially in BETA. After months of building, refining, and testing behind the scenes, we're ready to welcome our first batch of early users.

If you’d like to try Spidra before the public launch and help shape the future of AI-powered web scraping, we’d love to have you on board. You can 

Spidra is an AI-first web scraping platform that lets you extract data from any website using real browser automation and natural language prompts. 

Instead of writing selectors, scripts, or dealing with brittle crawlers, you simply tell Spidra what you want, and it does the clicking, scrolling, typing, and waiting for you.

Under the hood, Spidra uses large language models and real browser automation to navigate pages like a human, extract clean data, solve captchas, and stay undetected by anti-bot systems.

Web scraping hasn’t evolved much in the last decade. Most tools still rely on selectors, brittle automations, and endless debugging. And if you’ve ever tried scraping a dynamic site, handling login flows, or getting past captchas… you know the pain.

We wanted to rethink scraping from the ground up.

What if you could simply tell a scraper, 

“Extract the latest 20 listings and send them to Slack,”

What if the scraper actually behaved like a real user — clicking, scrolling, typing, waiting — instead of firing thousands of requests and hoping anti-bot systems don’t notice?

What if AI could understand the page, locate the right data, and deliver it in clean, structured formats?

This is the problem we set out to solve.

And this is what Spidra is built to do.

If you’re working with web data, whether you’re monitoring prices, scraping jobs, aggregating content, collecting research, or building lead lists, we’d love to have you test Spidra.

 to learn more about how everything works in detail.

We’re genuinely excited to put Spidra into your hands and see what you build with it.

Thank you for being part of this early chapter.

More updates and plenty of improvements are on the way!

Your feedback is extremely valuable at this stage. It helps us refine features, improve accuracy, and shape the roadmap.

feedback-spidra.webp


Whenever you encounter something confusing, have a suggestion, or want to share an idea, just click it and send your thoughts directly to us.

feedback-form.webp

We read every submission that comes in, and many of the improvements you’ll see in upcoming updates will come directly from beta testers like you.

Blog

Introducing authenticated scraping and crawling

Explore Articles

Introducing Crawling via the Spidra API

Captcha solving and stealth mode are now live for crawling

Dark mode Is now available in Spidra

Introducing AI-powered crawling

Spidra beta is now live 🚀