Find the pages you need before you scrape
Discover a site's URLs from its sitemap and links, filtered to your topic. Up to 5,000 in one request, for 1 credit.
0 URLs found
Sitemap 0 Page links 0
Waiting to run
A real map of docs.python.org, replayed. Only the pace is staged.
Stop scraping pages you don't need
A site can have hundreds of pages when you need a handful. Map shows you which ones first, for 1 credit.
Read every page to find them
0credits
Reading page 1 of 595…
Map first, then scrape the matches
0credits
Mapping…
Same 21 pages, 573 fewer credits, on one docs site.
From one URL to the pages that matter
One real run on docs.python.org: where the URLs came from, and how a search narrowed them down. 9.6 s, 1 credit.
Read the sitemap
0URLs
robots.txt Sitemap lines, then /sitemap.xml and sitemap indexes, gzipped ones included.
Follow page links
0URLs
The start page's links and their text, then up to 40 more pages. With a search, pages about it go first.
Clean up
0kept
Same site only, no duplicates or log-out links, current pages before old versions and translations.
Rank for "asyncio"
0match
Best matches first, by the words in the URL and in the link text.
The request
curl -X POST https://api.spidra.io/api/map \-H "Authorization: Bearer $SPIDRA_API_KEY" \-H "Content-Type: application/json" \-d '{ "url": "https://docs.python.org/3/", "search": "asyncio", "limit": 20 }'
Map answers in the same request: no job to poll. Copy the list, or export it as CSV or TXT from the dashboard.
21 of 595 URLs match “asyncio”
Found through the sitemap (7) and page links (588). The first 20, as limit asked:
asyncio
/3/library/asyncio.html
asyncio and free-threaded Python
/3/library/asyncio-threading.html
A conceptual overview of asyncio
/3/howto/a-conceptual-overview-of-asyncio.html
Create asynchronous generators only when the event loop is running
/3/library/asyncio-dev.html
Scheduling from other threads
/3/library/asyncio-task.html
Synchronization Primitives
/3/library/asyncio-sync.html
Priority Queue
/3/library/asyncio-queue.html
Low level utility functions
/3/library/asyncio-graph.html
Command-line introspection tools
/3/library/asyncio-tools.html
Handling Keyboard Interruption
/3/library/asyncio-runner.html
Register an open socket to wait for data using streams
/3/library/asyncio-stream.html
Future Functions
/3/library/asyncio-future.html
Getting and Setting the Policy
/3/library/asyncio-policy.html
loop.subprocess_exec() and SubprocessProtocol
/3/library/asyncio-protocol.html
Display the current date with call_later()
/3/library/asyncio-eventloop.html
Subprocess Support on Windows
/3/library/asyncio-platforms.html
Future and Task private constructors
/3/library/asyncio-extending.html
High-level API Index
/3/library/asyncio-api-index.html
Interacting with Subprocesses
/3/library/asyncio-subprocess.html
Exceptions
/3/library/asyncio-exceptions.html
Find more than what's in the sitemap
A sitemap doesn't always tell the whole story. Here's the same map of docs.python.org, three ways.
docs.python.org, search “asyncio”
0matches
Out of 595 URLs found.
The first 6 matches
asyncio
/3/library/asyncio.html
asyncio and free-threaded Python
/3/library/asyncio-threading.html
A conceptual overview of asyncio
/3/howto/a-conceptual-overview-of-asyncio.html
Create asynchronous generators only when the event loop is running
/3/library/asyncio-dev.html
Scheduling from other threads
/3/library/asyncio-task.html
Synchronization Primitives
/3/library/asyncio-sync.html
The most complete: the sitemap and the links on the site's pages together.
includeSubdomains
Also list subdomains, like docs.example.com from example.com.
ignoreQueryParams
On by default: URLs that differ only by query string count as one.
cookies
For a site whose pages need your sign-in.
A list is where the real work starts
Map tells you what a site has, so the next tool only touches the pages you want. Pick a step to watch it run.
One credit a map, however big the site
A map costs the same whether it returns 21 URLs or 5,000. One that finds nothing is free.
Maps from this page, and what each cost
Sitemap only, “asyncio” · found nothing
FreeSearch “asyncio” · out of 595
1 creditEvery URL on the site · no search
1 creditThe most one map returns · the limit
1 creditWhat a plan buys, in maps
Credits are shared by every tool, so the same balance pays for the pages you scrape after. 60 maps a minute on every plan.
See plans and top-upsQuestions, answered
Map
It reads the site's sitemaps first, then the links on the page you gave, then up to 40 more pages for links the start page doesn't show. With a search term, pages about that term are read first.
You build the product.
Spidra handles the data work.
Start with 300 free credits, about 300 pages. No credit card required.