Skip to main content
All releases
New

Find the pages you need with Map and an improved Crawl

Map lists a site's URLs from its sitemap and links, up to 5,000 in one request. Crawl now returns only the pages you describe, with path filters and a report of what it skipped.

We've made it easier to discover pages on a website and collect data from the ones that matter. The new Map feature helps you find URLs before scraping, while improvements to Crawl give you more control over which pages are returned and better visibility into the results.

Discover URLs with Map

Map lists the pages it finds through a website's sitemap and links, without scraping the content of every page.

You can search for relevant URLs, include subdomains, or choose whether to discover pages from sitemaps, page links, or both.

For example, map the Python documentation and search for asyncio. Spidra returns relevant documentation URLs, including the main asyncio reference and related guides.

Map can return up to 5,000 URLs in one request. From the dashboard, you can review the links, export them as CSV or TXT, or use them to start another scraping workflow.

Map searching docs.python.org/3/ for asyncio: 21 of 595 URLs match, found through the sitemap and page links, for 1 credit

Crawl the pages that match your goal

Crawl now takes a more focused approach to website discovery. Describe the pages you want in plain language, such as individual product pages in a category or articles in a documentation section, and Spidra follows relevant links to find them.

Pages visited only to discover other pages are treated as navigation pages. They aren't included in the returned dataset or charged as crawled results.

You also have more control over the process:

  • Path filtering: Include or exclude specific sections of a website.
  • Crawl reports: See how Spidra interpreted your request and why certain URLs were skipped.
  • Larger crawls: Retrieve results through paginated responses and download them for further processing.

The dashboard also makes it easier to inspect completed jobs and their results.

Map and the improved Crawl are available through the Spidra dashboard, API, SDKs, and MCP server.

Read the Map documentation and Crawl documentation, or open Spidra to try them.

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.