Introducing Spidra Search: Search the web before you scrape it

Most scraping tools start by asking you for a URL.

That makes sense if you already know exactly where the data lives. For example, if you have a product page, a documentation site, a list of company URLs, or a domain you want to crawl, then it makes sense to give the scraper the URL and let it get to work.

But that is not always where the work starts.

Sometimes you know the information you need without knowing the exact page or website where it lives. You may want to find "open source alternatives to Datadog", look for the "latest SEC filings from Nvidia", find discussions about a "Playwright authentication issue", or search for research papers on "retrieval-augmented generation".

In cases like these, you first need to find the relevant pages before you can do anything useful with them.

Until now, you had to handle that step somewhere else, whether that meant searching manually, using another search API, or collecting the URLs yourself before bringing them into Spidra.

We have now added Search to Spidra so you can handle that part of the workflow in the same place.

You can use Search directly from the dashboard or in code, choose the results you want, and keep working with the pages you find instead of stopping at a list of links.

Start with what you are looking for

Suppose you want to find "open source alternatives to Datadog". You can search for them directly:

const result = await spidra.search({
  query: "open source alternatives to Datadog",
  sources: ["web"],
});

Spidra returns the search results as structured data, including the information you would expect such as titles, URLs, and descriptions.

This gives Search a different job from Scrape.

Note: If you already know the exact URL you need, there is no reason to search for it again. You can scrape the page directly.

Search the part of the web you actually need

Not every search has the same intent.

If you are trying to solve a programming issue, developer-focused results will usually be more useful than a generic web search. If you are researching something that happened recently, you may want news results. If you need an academic paper or an image, those are different searches again.

Spidra Search supports web, news, images, videos, research, and developer sources, so you can specify where you want results to come from.

For example, if you are trying to troubleshoot Playwright authentication state, you can search developer sources:

const result = await spidra.search({
  query: "Playwright authentication state issue",
  sources: ["developer"],
});

If you are researching the latest Nvidia SEC filings, you might want both web and news results:

const result = await spidra.search({
  query: "latest SEC filings from Nvidia",
  sources: ["web", "news"],
});

This is useful because the web results may contain the filing or official company material, while the news results can help you find recent reporting around it.

You can do the same thing for research:

const result = await spidra.search({
  query: "retrieval augmented generation",
  sources: ["research"],
});

Or images:

const result = await spidra.search({
  query: "mountain wallpaper larger:1920x1080",
  sources: ["images"],
});

For images, you can download them as a zipped folder:

The useful part here is not simply that Spidra offers different result types. It means you can tell Search what kind of material you need instead of taking a generic set of results and filtering it yourself afterward.

Sometimes the file itself matters

Search also supports filters that become useful when you know more about the result you need.

For example, if you are looking for Apple’s annual report, you probably want the actual report rather than articles discussing it.

You can limit a web search to PDF files:

const result = await spidra.search({
  query: "Apple annual report 2026",
  sources: ["web"],
  filetype: "pdf",
});

The same idea applies to domain and recency filtering.

If you are building a workflow around company filings, you may want results only from a trusted domain. If you are tracking something that changes quickly, you may only care about results from the past day, week, or month.

These controls matter more once Search becomes part of an application instead of something a person runs once in the dashboard.

Search does not have to stop at the result page

A search result is useful when all you need is the title, URL, description, or other result metadata.

But many workflows need the content behind those URLs.

For example, if you search for open source alternatives to Datadog because you want to compare their features, the URLs themselves are only the first step. You still need to visit the pages and read the relevant content.

You can ask Spidra to scrape web results as part of the same search:

const result = await spidra.search({
  query: "open source alternatives to Datadog",
  sources: ["web"],
  scrapeOptions: {
    formats: ["markdown"],
    maxResults: 5,
  },
});

Instead of returning only the search metadata, Spidra can also fetch the page content for the returned web results.

This is useful for applications that need the contents of the pages rather than a list of links, such as research tools, RAG pipelines, lead enrichment systems, monitoring workflows, or AI agents.

It also means you do not have to build one search integration to discover URLs and another scraping integration just to read them.

Search and scraping still solve different problems

Adding page content to Search does not make Scrape unnecessary.

If you already have a URL and need to work deeply with that page, Scrape gives you much more control.

Spidra can open the page in a real browser, render JavaScript, perform browser actions, use authentication sessions, work with proxies and stealth features when needed, and extract structured information from the page.

That becomes important when getting the data involves more than simply downloading the page.

For example, a search might help you find a product page, but you may still need to click a button, choose an option, open a modal, or loop through several items before the information you need appears.

That is where the rest of Spidra comes in.

You could:

• Search to find the relevant pages.
• Scrape one of them if you need to work with a single page.
• Batch the results if you already have several URLs.
• Crawl one of the websites if you discover that the useful information is spread across multiple pages.
• Use browser actions if the site needs interaction before the data appears.

Search gives those workflows an earlier starting point.

Search is also available to agents through MCP

Search has also been added to Spidra’s MCP server, which means an agent using Spidra does not need you to provide every URL before it can start working.

For example, I asked Claude:

I’m thinking of visiting Nairobi later this year. Use Spidra to find recent recommendations for where to stay, what to do, and good restaurants, then put together a 3-day plan for me.

The agent used Search to find the pages first, then used Spidra’s other tools when it needed the full page content.

Search moves Spidra one step earlier in the workflow

We originally built Spidra for what happens after you already have a website or URL.

You give Spidra a page, and it handles the work required to turn that page into useful data, including browser rendering, interaction, crawling, and structured extraction.

But many workflows people build don't actually begin with a URL.

They begin with something like:

• Find companies that match these criteria.
• Find the latest filings from this company.
• Find discussions about this bug.
• Find papers related to this topic.
• Find pages worth extracting data from.

That is the part Search adds.

You can now start with what you are looking for, find the relevant pages, and continue into the same scraping, crawling, browser interaction, or extraction workflow without first building a separate discovery step. Try Search in Spidra or read the Search docs.