Imagine trying to collect documentation about one feature from a website with hundreds of pages. You could scrape everything and filter the content afterward, but that means processing installation guides, unrelated tutorials, changelogs, and even translated versions of pages you don't need.
We ran into this while exploring FastAPI's documentation. A single website mapping request discovered 619 URLs, with 36 matching our search for dependency-related content. We eventually needed just two pages.
That's the idea behind mapping a website before scraping it. Instead of collecting all the content and figuring out what's useful later, you find the relevant URLs first and scrape only those pages.
In this tutorial, we'll use Python and Spidra to map FastAPI's documentation, find pages about dependency injection, filter unwanted URLs, and save the selected pages as Markdown files.
This approach is useful when you know what information you're looking for but don't have all the URLs. If you already have the exact links, you can skip mapping and scrape them directly.
Find the URLs before scraping the content
A scraper needs a URL to know which page to fetch. But when you're working with an unfamiliar website, finding the right URLs can be half the job.
One option is to check the site's XML sitemap, usually available at /sitemap.xml or referenced in robots.txt. A sitemap lists URLs that the website owner wants search engines and other tools to discover.
You can parse that file using Python's XML libraries and filter the links yourself.
The problem is that not every website has a complete or reliable sitemap. Some pages might be missing from it but still reachable through internal links. Others might appear in a sitemap even though they're no longer useful.
As Google explains in its sitemap documentation, sitemaps help crawlers discover URLs, but they're not a guarantee that every page will be found.
For this workflow, we'll use Spidra's Map feature, which discovers URLs from a site's sitemap and internal links.
Map can also search for URLs related to a particular topic. It looks at URLs and link text to identify relevant matches and returns the list without extracting the full page content.
For example, you could map a documentation website and search for authentication-related pages, discover product URLs from an online store, or identify articles about a particular topic on a blog.
The result is a list of candidate URLs that you can inspect before deciding which ones to scrape.
Map FastAPI's documentation with Python
Let's say we want to collect FastAPI's documentation about dependency injection.
FastAPI has tutorials, API references, deployment guides, and translations. We're interested in the English documentation explaining how dependencies work.
First, install the Spidra Python SDK:
pip install spidraYou'll also need a Spidra API key. Create one from your account and set it as an environment variable:
export SPIDRA_API_KEY="your_api_key"Now create a Python file and initialize the client:
# map_fastapi.py
import os
from spidra import Spidra
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
site = client.map(
"https://fastapi.tiangolo.com/",
search="dependencies",
limit=10,
)
print(site.total, "URLs matched,", site.scanned, "scanned,", site.credits_used, "credit")
for link in site.links[:5]:
print(link.url)Here, client.map() takes FastAPI's documentation homepage as the starting URL.
We pass search="dependencies" to find URLs related to that topic and limit=10 to return up to ten matches.
When a search is provided, Map ranks the URLs by relevance rather than simply returning every discovered page.
Here's what came back during our recorded test:
36 URLs matched, 619 scanned, 1 credit
https://fastapi.tiangolo.com/tutorial/dependencies/
https://fastapi.tiangolo.com/reference/dependencies/
https://fastapi.tiangolo.com/de/reference/dependencies/
https://fastapi.tiangolo.com/es/reference/dependencies/
https://fastapi.tiangolo.com/fr/reference/dependencies/That's quite a useful result from one request.
Map discovered 619 URLs from the website and identified 36 related to our search. We asked for ten results, and the output above shows the first five.
The request cost one credit, regardless of how many URLs it discovered.
More importantly, we haven't scraped any documentation content yet. We only have the URLs, which means we can still decide exactly what we want to collect.
And looking at the output, there's already something we should fix before scraping.
The third result is a page we don't need
The first two URLs are exactly what we're looking for:
https://fastapi.tiangolo.com/tutorial/dependencies/
https://fastapi.tiangolo.com/reference/dependencies/One points to FastAPI's dependency injection tutorial, while the other contains the API reference.
But the next three URLs point to German, Spanish, and French translations of the reference page.
Map didn't make a mistake. These pages are relevant to dependency injection, which is what we searched for.
We simply haven't told it that we only want English documentation.
To see why this matters, let's scrape the first three results without filtering them.
Add this to map_fastapi.py:
picked = [link.url for link in site.links[:3]]
batch = client.batch_scrape(
picked,
output="markdown",
extract_content_only=True,
)
for item in batch.items:
print(item.url, item.status, len(item.result or ""), "characters")We're using Spidra's Batch Scrape API to process the URLs together.
The output="markdown" option returns the page content as Markdown, while extract_content_only=True asks Spidra to remove elements such as navigation, headers, and sidebars from the extracted content.
Here's the actual output from our test:
https://fastapi.tiangolo.com/tutorial/dependencies/ completed 20380 characters
https://fastapi.tiangolo.com/reference/dependencies/ completed 20153 characters
https://fastapi.tiangolo.com/de/reference/dependencies/ completed 20283 charactersAll three pages were successfully scraped, but the third one was the German reference page.
Nothing broke. The API did exactly what we asked.
Still, we've collected more content than we needed.
That's especially annoying when you're preparing documentation for a search index or a RAG application. You could end up storing multiple language versions of the same information, creating unnecessary duplication.
And there's a cost to that.
The recorded Map and Batch Scrape requests used four credits altogether: one to discover the URLs and three to scrape the pages.
We could have avoided that third scrape entirely.
The fix is straightforward: filter the URLs before sending them to Batch Scrape.
Filter the URLs to keep only the pages you need
There are several ways to decide which URLs to scrape.
For example, you might want every URL under /products/, only articles published under /blog/, or documentation pages containing a particular word.
In our case, we already know the two English documentation paths we want:
/tutorial/dependencies/
/reference/dependencies/We can use Python's urllib.parse module to check the paths and remove everything else.
from urllib.parse import urlsplit
wanted_paths = {
"/tutorial/dependencies/",
"/reference/dependencies/",
}
selected = [
link.url for link in site.links
if urlsplit(link.url).hostname == "fastapi.tiangolo.com"
and urlsplit(link.url).path in wanted_paths
]
print("\n".join(selected))Applied to our recorded Map results, this selects:
https://fastapi.tiangolo.com/tutorial/dependencies/
https://fastapi.tiangolo.com/reference/dependencies/The German, Spanish, and French pages are excluded because their paths don't match either entry in wanted_paths.
We're also checking the hostname, so URLs outside the intended site won't accidentally enter our selection.
This example uses exact path matching because we know which pages we need. If you wanted an entire documentation section, you could use startswith() to select URLs sharing a particular path prefix.
For example, checking whether a path starts with /tutorial/dependencies/ would keep pages within that tutorial section. You would still need to consider the website's language prefixes, query parameters, and URL structure.
The important thing is to make URL selection deliberate.
A keyword search helps you find likely matches, but your own filtering rules decide which pages actually get scraped.
Put mapping, filtering, and scraping into one script
Now let's combine everything into a script that maps the website, selects the two English pages, scrapes them, and saves their content.
We'll also handle failed pages so the script doesn't try to save an empty or unsuccessful result.
Create a new file named map_and_scrape.py:
# map_and_scrape.py
import os
from pathlib import Path
from urllib.parse import urlsplit
from spidra import Spidra
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
site = client.map(
"https://fastapi.tiangolo.com/",
search="dependencies",
limit=10,
)
wanted_paths = {
"/tutorial/dependencies/",
"/reference/dependencies/",
}
selected = []
for link in site.links:
parsed = urlsplit(link.url)
if parsed.hostname == "fastapi.tiangolo.com" and parsed.path in wanted_paths:
if link.url not in selected:
selected.append(link.url)
if not selected:
raise SystemExit("No matching URLs found. Check the Map results.")
output_dir = Path("fastapi-dependencies")
output_dir.mkdir(exist_ok=True)
batch = client.batch_scrape(
selected,
output="markdown",
extract_content_only=True,
)
for item in batch.items:
if (
item.status != "completed"
or not isinstance(item.result, str)
or not item.result.strip()
):
print("SKIPPED", item.url, item.error or "Empty result")
continue
filename = urlsplit(item.url).path.strip("/").replace("/", "_") + ".md"
(output_dir / filename).write_text(item.result, encoding="utf-8")
print("SAVED", filename)
print("Completed:", batch.completed_count, "Failed:", batch.failed_count)Run the script:
python map_and_scrape.pyThe script starts by mapping the documentation website and finding URLs related to dependencies. It then applies our path filter and removes duplicate URLs.
After that, it sends the selected URLs to Batch Scrape and saves each successful result as a Markdown file.
Assuming both selected pages succeed, you'll get this directory:
fastapi-dependencies/
├── reference_dependencies.md
└── tutorial_dependencies.mdEach file contains the extracted content of its corresponding page.
You can now use those Markdown files for whatever you're building, whether that's a searchable documentation collection, a local reference tool, or a RAG pipeline.
One detail worth pointing out is the way the script handles scraping failures.
We check item.status and verify that the returned content is a non-empty string before writing anything to disk.
That's necessary because a completed batch doesn't always mean every page succeeded.
A successful batch can still contain failed pages
When Batch Scrape processes several URLs, each URL gets its own result.
Some pages might work while others fail because the page no longer exists, the server returns an error, or the site blocks automated access.
Spidra can report the overall batch as completed even when individual pages fail.
For example, in a separate recorded test involving four URLs, two pages completed while the other two failed. One returned HTTP 404, and the other had a DNS lookup failure.
Both failed items used zero credits.
That's why our script checks every item's status rather than relying only on the batch's overall status.
It also prints the failure information, which helps identify pages that need another attempt.
For a larger pipeline, you could save the failed URLs to a file and retry them later. Spidra also supports retrying failed batch items without repeating successful ones.
You can find those options in the Batch Scrape documentation.
Batch Scrape accepts up to 50 URLs per request. If your filtering step returns more than that, you'll need to divide the selected URLs into batches of 50 or fewer.
What if Map doesn't find the pages you're looking for?
Mapping works well when a website exposes its URLs through sitemaps or internal links, but it won't always discover everything.
A page might exist without appearing in the sitemap or being linked from another discoverable page. Some websites also generate links dynamically, which can make URL discovery more difficult.
If Map doesn't find the pages you expect, start by checking the site's URL structure and sitemap.
By default, Spidra Map uses both sitemap data and page links. You can change that behavior with the sitemap option.
Setting it to "only" uses the sitemap, which can be quicker when the website maintains a reliable one. Setting it to "skip" discovers URLs through links instead, which can help when the sitemap is missing or outdated.
You can also increase the result limit if a search returns more matches than you requested.
Map supports returning up to 5,000 URLs per request, but increasing that limit won't reveal pages that aren't discoverable from the sources it examines.
If you already know a page's exact URL, there's no reason to map the entire site just to retrieve it. You can pass the URL directly to Scrape or Batch Scrape.
Before collecting content from a third-party website, review its terms and crawling policies. Spidra Map also provides a respectRobots option for excluding URLs disallowed by robots.txt. Our guide to web scraping legality covers the broader considerations.
But what if you don't know the exact URLs, and finding them requires moving through categories or other navigation pages?
That's where crawling becomes useful.
When should you crawl instead of mapping?
Map and Crawl solve related problems, but they're not interchangeable.
Map is useful when you want to discover a website's URLs, inspect them, and decide what to scrape.
That's what we did with FastAPI's documentation.
We discovered the available URLs, searched for dependency-related pages, filtered the results, and scraped the two that matched our requirements.
Crawl works differently.
With Spidra's Crawl feature, you provide a starting URL and describe the kind of pages you want. The crawler follows links, uses navigation pages to discover matching content, and returns the pages that fit your instruction.
For example, suppose you want individual product pages from an online store.
You might know the store's category page but not the URLs of all its products. Instead of mapping the site and writing URL filters, you could start a crawl from the category page with an instruction to find individual product detail pages.
The crawler can follow category and listing links to discover those pages.
You can also use include and exclude path filters when you need stricter boundaries.
The choice depends on how much control you need over page selection.
If you want to inspect and approve the exact URLs before fetching content, Map followed by Batch Scrape is a good fit.
If you want to describe the target pages and have the system follow navigation links to find them, Crawl is usually a better starting point.
And when you already have the URLs, use Scrape or Batch Scrape directly.
