How to list all discoverable URLs on a website with Python

You're about to extract data from a website, and you don't know what's on it yet. By the end of this article, you'll have one script that can list the discoverable URLs on a website with the Python Programming Language, remove the duplicates, sort them into groups and write a CSV. This article uses Spidra's Map, and it is particularly for developers who need that inventory before they scrape anything.

A URL is discoverable if the site's sitemap lists it or another page links to it. A page with neither is invisible to every tool that works from outside, and we'll get to what that means.

The sitemap is where everyone starts, and some sites don't have one

The first attempt is always /sitemap.xml. When it exists, it's usually the cleanest list you'll get. ultimate-sitemap-parser reads sitemaps from robots.txt, follows sitemap indexes and tries to find sitemaps that aren't listed there. If your site has a good sitemap, use that and stop reading.

Then there's books.toscrape.com, which has none. Ask Map to read only the sitemap there:

result = client.map("https://books.toscrape.com/", sitemap="only")print(result.links, result.total, result.credits_used)print(result.message)

Output:

[] 0 0
This site doesn't publish a sitemap we could read. Try sitemap: "include" to read its links instead.

Nothing found costs nothing, and the message tells you what to try. A sitemap can be missing, stale, or list only part of a site. The other source is the site's own links.

One Map call returns URLs when it has them

Map reads the sitemap and the links on the site's pages, and it answers in the same request. No job, no polling. We'll use fastapi.tiangolo.com: it has a sitemap, and it's big enough to show where the interesting problems are.

# try_map.pyimport os
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"]) 
site = client.map("https://fastapi.tiangolo.com/", search="dependencies", limit=10)
print(site.total, "URLs matched,", site.scanned, "scanned,", site.credits_used, "credit")
for link in site.links[:5]:    
    print(link.url)

Output:

36 URLs matched, 619 scanned, 1 credit
https://fastapi.tiangolo.com/tutorial/dependencies/
https://fastapi.tiangolo.com/reference/dependencies/
https://fastapi.tiangolo.com/de/reference/dependencies/
https://fastapi.tiangolo.com/es/reference/dependencies/
https://fastapi.tiangolo.com/fr/reference/dependencies/

Two things to notice. scanned counts every URL Map found before the search filter, so this site has at least 619. And the German, Spanish and French copies of the same page sit right next to the English one. We'll deal with them in a minute.

Each link also has a title, but only when Map found the URL as a link on a page: it's the link text. A URL that came only from the sitemap has none. That's why the CSV we're building says "titles when available".

Sitemap, links or both? Run both and compare

Map's sitemap option has three settings. "only" reads the sitemap, "skip" reads only page links, and "include" (the default) does both. Running the first two on the same site shows how much each source contributes. Start a new file for the real script:

# url_inventory.pyimport csv
import os
from collections import Counter
from datetime import date
from urllib.parse import urlsplit, urlunsplit

from spidra import Spidra
SITE = "https://fastapi.tiangolo.com/"
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

Then a small function that runs one Map call and checks it wasn't cut off:

def run_map(mode):    
    result = client.map(SITE, limit=5000, sitemap=mode)    
    print(mode, "| total", result.total, "| returned", len(result.links), "| from", result.sources)    
    if result.total > len(result.links):        
        print("  Cut off: raise limit.")    
    return result

For a sense of the other extreme, quotes.toscrape.com with sitemap="skip" returned 154 URLs, none from a sitemap, in about 4 seconds. That's a site where links do all the work.

The default limit stops at 500

limit defaults to 500 and goes up to 5,000. On a site with 619 URLs, a call that leaves it alone returns a list that looks complete and isn't. That's why run_map compares total with the number of links it got back, and asks for 5,000.

site = client.map(SITE)   # default limit
print(site.total, "found,", len(site.links), "returned")

Counts also drift a little between runs. On docs.python.org, two runs the same night scanned 556 and 554 URLs. Stamp the CSV with the date, and don't promise anyone an exact number.

Remove duplicates twice

Map already treats URLs that differ only by query string as one, and it drops duplicates and action links such as log-out. That's the first pass. The second one is yours, because inventories get merged and compared later, and https://example.com/page and http://www.example.com/page/ should be one row.

def normalize(url):    
    p = urlsplit(url)    
    host = p.netloc.lower().removeprefix("www.")    
    return urlunsplit(("https", host, p.path.rstrip("/") or "/", "", "")) 

def dedupe(links):    
    seen = {}    
    for link in links:        
        key = normalize(link.url)        
        if key not in seen or (not seen[key] and link.title):            
            seen[key] = link.title
    return seen  # {url: title or None}

The last condition keeps a title if any copy of the URL has one. A bare sitemap URL and a www copy with a link title collapse into one row, with the title. Trailing slashes and #fragments collapse too. We merge both runs this way, so a URL found by the sitemap and by links appears once.

Group the list by path

A flat list of 600 URLs is hard to look at. The first path segment usually names the section, so group on that. Translated copies and files get their own groups, or they pollute every count.

LANGS = {"de", "es", "fr"}   # add the codes you see in your own list 

def category(url):    
    parts = [p for p in urlsplit(url).path.split("/") if p]    
    if not parts:        
        return "home"    
    if parts[0] in LANGS:        
        return "translation"    
    if "." in parts[-1] and not parts[-1].endswith(".html"):        
        return "file"    
    return parts[0]

The language list is the one thing to check by eye. Print the first path segments, spot the two-letter ones, and add them.

Choose the URLs on purpose

Most inventories skip this step, and it decides what you pay for later. Write down which groups you're keeping, then print what the rules dropped, so the choice shows up in the output and isn't buried in a filter.

KEEP = {"tutorial", "reference"}   # the sections you want to scrape later 
only, skip = run_map("only"), run_map("skip")
inventory = dedupe(only.links + skip.links)
in_sitemap = {normalize(link.url) for link in only.links}
in_links = {normalize(link.url) for link in skip.links} 

picked = {u for u in inventory if category(u) in KEEP}
dropped = Counter(category(u) for u in inventory if u not in picked)
print(len(picked), "kept. Dropped:", dict(dropped))

Now write the inventory: one row per URL, with its group, whether your rules selected it, and where it was found.

filename = f"url_inventory_{date.today()}.csv"
with open(filename, "w", newline="", encoding="utf-8") as f:    
    out = csv.writer(f)    
    out.writerow(["url", "title", "category", "selected", "found_in"])    
    for url, title in sorted(inventory.items()):        
        found = "both" if url in in_sitemap and url in in_links else "sitemap" if url in in_sitemap else "links"        
        out.writerow([url, title or "", category(url), url in picked, found])

That's the file: a categorized inventory with titles where Map had them, no duplicate rows, and a column saying which source saw each URL.

What Map misses, and why

Map finds what the sitemap lists and what the site's links expose. On a site with no sitemap, it reads the start page's links and then a limited number of further pages, so a page that's buried several clicks deep can be missed. Other misses are structural:

• A page nothing links to and the sitemap doesn't list. Nobody outside can find it, Map included.
• Links built by scripts, or sites that block automated visits. These can come back empty, and an empty map is free.
• Pages behind a login, unless you pass cookies.
• Anything respect_robots=True leaves out, if you turn it on.

Hand the picked URLs straight to a batch

The list is only useful if the next step is cheap to take. Batch scraping takes up to 50 URLs per request, so send the picked pages in chunks:

urls = sorted(picked)
for i in range(0, len(urls), 50):    
    batch = client.batch_scrape(urls[i:i + 50], output="markdown", extract_content_only=True)    
    print(batch.completed_count, "completed,", batch.failed_count, "failed")

A batch is completed even when some items fail, so read failed_count. Failed pages cost nothing.

Check the site's terms first

Map reads a public site's sitemap and links, and you're responsible for following that site's terms and the law. It doesn't apply robots.txt unless you set respect_robots=True, which is the setting to use for audits.

Read our guide to the legal questions before you scrape anything at scale.

Run it on a site you own first

Your own site is the best test, because you already know which pages should appear, so you can see exactly which ones Map missed.

Create a free account at app.spidra.io and run url_inventory.py on it. The Python SDK docs list the Map options we skipped.