Turning a website into Markdown with Python sounds like one library call. It isn't, because a page is full of text you don't want: the menu, a cookie banner, a sidebar, related posts, a footer. A script or a model that reads the page as it is reads all of that too.
This tutorial builds a command that takes a public URL and writes the page's text to a file as clean Markdown, with the URL it came from. When a page can't be read, it says so and saves nothing. First you'll see what the simple route produces and where it fails, so you know what the finished command has to handle.
Install the packages, and put your key in the environment as SPIDRA_API_KEY:
# pip install spidra requests markdownify beautifulsoup4 trafilatura
import argparse
import hashlib
import json
import os
import re
import sys
import threading
from datetime import date
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from types import SimpleNamespace
from urllib.parse import urlsplit
import requests
import trafilatura
from bs4 import BeautifulSoup
from markdownify import markdownify
from spidra import (
Spidra,
SpidraAuthenticationError,
SpidraError,
SpidraJobFailedError,
SpidraPaymentRequiredError,
SpidraRateLimitError,
SpidraServerError,
SpidraTimeoutError,
SpidraValidationError,
)
The simple route saves the whole page
The ordinary route downloads the HTML and converts all of it:
def simple_markdown(url):
"""Convert everything the server sent back."""
return markdownify(requests.get(url, timeout=30).text)To run the examples without a network, a tiny local site stands in for a real one. This page has the usual furniture: a cookie banner, a menu, a sidebar, an article with a code block and a table, related posts and a footer.
ACME = (
"<html><head><title>Retry failed requests - Acme Docs</title>"
"<style>.x{color:red}</style><script>window.dataLayer=[];</script></head><body>"
'<div id="cookie-banner">We use cookies. <button>Accept all</button></div>'
'<nav><a href="/">Acme</a> <a href="/docs">Docs</a> <a href="/login">Sign in</a></nav>'
'<aside><h3>In this section</h3><ul><li><a href="/docs/a">Install</a></li></ul></aside>'
'<article><h1>Retry failed requests</h1>'
'<p>Wait between attempts. Retry a <a href="/e">503</a>.</p><h2>Example</h2>'
'<pre><code class="language-python">for n in range(3):\n time.sleep(2 ** n)\n</code></pre>'
"<h2>Limits</h2><table><tr><th>Plan</th><th>Retries</th></tr>"
"<tr><td>Free</td><td>3</td></tr></table></article>"
'<section class="related"><h3>Related posts</h3><a href="/blog/1">Ten API tips</a></section>'
"<footer>(c) 2026 Acme Inc. Terms Privacy. Subscribe to our newsletter</footer></body></html>"
)
class Site(BaseHTTPRequestHandler):
pages = {}
def do_GET(self):
status, html = self.pages.get(self.path, (404, "<h1>Not found</h1>"))
body = html.encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
passdef serve(pages, port=8795):
"""{path: (status, html)} -> the base URL of a site that serves those pages."""
Site.pages = pages
threading.Thread(
target=ThreadingHTTPServer(("127.0.0.1", port), Site).serve_forever, daemon=True
).start()
return f"http://127.0.0.1:{port}"JS_PAGE = '<div id="app"></div><script>app.innerHTML = "<p>Be kind.</p>"</script>'
SITE = {
"/docs/retry": (200, ACME),
"/js": (200, JS_PAGE),
"/empty": (200, "<html><body></body></html>"),
"/forbidden": (403, "<h1>Access denied</h1>"),
}
base = serve(SITE)
print(simple_markdown(base + "/docs/retry"))Output:
Retry failed requests - Acme Docs
We use cookies. Accept all
[Acme](/) [Docs](/docs) [Sign in](/login)
### In this section
* [Install](/docs/a)
Retry failed requests
=====================
Wait between attempts. Retry a [503](/e).
Example
-------for n in range(3): time.sleep(2 ** n)
Limits
------
| Plan | Retries |
| --- | --- |
| Free | 3 |
### Related posts
[Ten API tips](/blog/1)
(c) 2026 Acme Inc. Terms Privacy. Subscribe to our newsletter
Everything the server sent is in the file. The cookie banner, the menu, the sidebar, the related posts and the footer all came along, and the title leads the document as a stray line. The code block lost its language, and the headings are underlined instead of using #.
The usual cleanup advice makes it worse
The common fix is strip=["script", "style"]. On a page with a script and a style block, it does the opposite of what you want:
head = "<head><style>.x{color:red}</style><script>window.dataLayer=[];</script></head>"
html = f"<html>{head}<body><p>Hi</p></body></html>"
print(repr(markdownify(html)))
print(repr(markdownify(html, strip=["script", "style"])))Output:
'Hi'
'.x{color:red}window.dataLayer=[];\n\nHi'strip removes the tags but keeps the text inside them, so the CSS and JavaScript end up in your Markdown. To drop them, remove the elements before converting. Beautiful Soup can do that, along with the page's furniture:
BOILERPLATE = [
"script",
"style",
"head",
"nav",
"header",
"footer",
"aside",
"#cookie-banner",
".related",
]def souped_markdown(html):
"""Remove what isn't content first, then convert. The selectors are the part that breaks."""
soup = BeautifulSoup(html, "html.parser")
for unwanted in soup.select(", ".join(BOILERPLATE)):
unwanted.decompose()
return markdownify(str(soup), heading_style="ATX", code_language="python")
# The same page, with the boilerplate removed before converting
cleaned = souped_markdown(ACME)
print(cleaned)Output:
# Retry failed requests
Wait between attempts. Retry a [503](/e).
## Example
```python
for n in range(3):
time.sleep(2 ** n)
## Limits
| Plan | Retries |
| --- | --- |
| Free | 3 |That's clean. It's also written for this one page: #cookie-banner and .related mean nothing on another site, so every site needs its own list.
An extraction library finds the main content without a list. trafilatura does this well on article-like pages:
article = trafilatura.extract(
ACME, output_format="markdown", include_tables=True, include_links=True
)
print(article)Output:
# Retry failed requests
Wait between attempts. Retry a [503](/e).
## Examplefor n in range(3): time.sleep(2 ** n)
## Limits
| Plan | Retries |
|---|---|
| Free | 3 |It dropped the boilerplate and kept the table, the link and the code. It also dropped the code block's language, and it makes a judgement about what counts as the main content on every page. When that judgement is wrong, the content is gone, and nothing tells you.
Plain requests saves error pages as content
Cleanup is the easy part. The harder problem is the page that isn't the page. Here's what the same route does with four pages that go wrong:
for path in ("/forbidden", "/empty", "/js", "/nothing-here"):
response = requests.get(base + path, timeout=10)
print(response.status_code, repr(souped_markdown(response.text).strip()))Output:
403 '# Access denied'
200 ''
200 ''
404 '# Not found'A 403 and a 404 become documents, because nothing checked the status. A blank page and a page drawn by JavaScript both return 200 and no text, so you'd save an empty file and think it worked. Real sites add cookie walls, logins and bot checks to this list, and a plain request can't tell you which one it hit.
Fetch the page with Spidra Scrape
Scrape loads the page in a browser, so content that JavaScript draws is part of the result. With extract_content_only=True, it strips the navigation, header, footer and sidebars. The Markdown comes back in result.content, exactly as the page's text, with no extra lines added.
# needs SPIDRA_API_KEY
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
result = client.scrape("https://example.com/docs/page", extract_content_only=True)
markdown = result.contentThat's the whole fetch. The rest of the command is what you do with markdown, and what you do when there isn't any.
Clean the Markdown
Content-only extraction removes menus and footers on most sites. It doesn't make every page tidy. A long reference page keeps most of its length, and some pages carry small oddities: Wikipedia's Markdown still starts with the list of other-language links, headings like # 3. Title come back with an escaped period, and Python's >>> prompts come back escaped too. These rules deal with them without touching your code blocks:
FENCE = "`" * 3 # the marker that opens and closes a code block
LINK_ITEM = re.compile(r"^\s*[*-]\s+\[[^\]]+\]\([^)]+\)\s*$")
ESCAPED_NUMBER = re.compile(r"^(#{1,6} \d+(?:\.\d+)*)\\\.", re.M)
def drop_source_line(lines):
"""Older output starts with 'Content from: <url>'. The CLI records the source itself."""
return lines[1:] if lines and lines[0].startswith("Content from:") else lines
def drop_leading_links(lines, minimum=8):
"""A run of bare links at the top, such as Wikipedia's language list, is not content."""
end = 0
while end < len(lines) and (LINK_ITEM.match(lines[end]) or not lines[end].strip()):
end += 1
links = [ln for ln in lines[:end] if ln.strip()]
return lines[end:] if len(links) >= minimum and end < len(lines) else lines
def unescape(text):
"""Undo escapes that only make text harder to read: '3\\.' in headings and '\\>>>' prompts."""
return ESCAPED_NUMBER.sub(r"\1.", text).replace("\\>>>", ">>>")
def tidy(text):
"""Trim trailing spaces and collapse blank runs, leaving code blocks exactly as they are."""
out, fenced, blanks = [], False, 0
for line in text.splitlines():
marker = line.startswith(FENCE)
fenced ^= marker
if fenced or marker or line.strip():
out += [""] * min(blanks, 1) + [line if fenced or marker else line.rstrip()]
blanks = 0
else:
blanks += 1
return "\n".join(out).strip("\n") + "\n"
def clean_markdown(markdown):
lines = drop_leading_links(drop_source_line(markdown.splitlines()))
return tidy(unescape("\n".join(lines)))
Here's a sample shaped like the top of a Wikipedia page, with nine language links before the content:
languages = ["العربية", "Deutsch", "Español", "Français", "हिन्दी"]
languages += ["日本語", "Português", "Русский", "中文"]
links = [f"* [{name}](https://{name[:2]}.wikipedia.org/wiki/X)" for name in languages]
sample = "\n".join(links + ["", "# 3\\. An Informal Introduction", "", "\\>>> 1 + 1", "2"])
print(clean_markdown(sample))
Output:
# 3. An Informal Introduction
>>> 1 + 1
2The language list needs at least eight links in a row at the top, and something after it, so a page that really is a list of links is left alone.
Save the file with its source URL
A saved file should say where it came from, and a name should stay readable and unique. Spidra returns the page's Markdown and nothing else, so the file has to record the source itself. YAML front matter does it, and json.dumps quotes each value safely, even a title with a colon or a quotation mark:
def slug_for(url, limit=60):
"""A readable name that stays unique: host and path, plus a short hash of the full URL."""
parts = urlsplit(url)
words = re.sub(r"[^a-z0-9]+", "-", f"{parts.hostname}{parts.path}".lower()).strip("-")
return f"{words[:limit].rstrip('-')}-{hashlib.sha1(url.encode('utf-8')).hexdigest()[:8]}"
def title_of(markdown, page_title, url):
"""The first heading, else the page's own title, else the host."""
heading = re.search(r"^#\s+(.+)$", markdown, re.M)
return (heading.group(1) if heading else page_title or urlsplit(url).hostname).strip()
def front_matter(url, title, fetched_on):
"""YAML with the source first. json.dumps gives strings that are safely quoted."""
return (
f"---\nsource_url: {json.dumps(url)}\ntitle: {json.dumps(title, ensure_ascii=False)}\n"
f"fetched_on: {fetched_on}\n---\n"
)
def build_document(url, title, markdown, fetched_on):
return front_matter(url, title, fetched_on) + "\n" + markdown
def write_atomic(path, text):
"""Write to a temporary file, then rename it, so a crash never leaves a partial file."""
temporary = path.with_name(path.name + ".tmp")
temporary.write_text(text, encoding="utf-8")
os.replace(temporary, path)
The name is the host and path, plus a short hash of the full URL, so two URLs that differ only by a query string never overwrite each other:
urls = [
"https://docs.python.org/3/tutorial/introduction.html",
"https://docs.python.org/3/tutorial/introduction.html?highlight=prompt",
"https://example.com",
]
for url in urls:
print(slug_for(url) + ".md")Output:
docs-python-org-3-tutorial-introduction-html-87070e86.md
docs-python-org-3-tutorial-introduction-html-bedd6686.md
example-com-327c3fda.md
And the file itself, for a made-up page:
url = "https://example.com/docs/retry"
markdown = "# Retry failed requests\n\nWait between attempts.\n"
print(build_document(url, "Retry failed requests", markdown, "2026-11-16"))Output:
---
source_url: "https://example.com/docs/retry"
title: "Retry failed requests"
fetched_on: 2026-11-16
---
# Retry failed requests
Wait between attempts.write_atomic writes to a temporary file and renames it, so a crash never leaves half a file where a good one used to be.
Report errors a script can act on
A command that other scripts call has to fail clearly. The SDK raises a specific exception for each kind of failure, so map each one to a message a person can read and an exit code a script can test:
RULES = [
(SpidraAuthenticationError, 3, "the API key was rejected. Check SPIDRA_API_KEY."),
(SpidraPaymentRequiredError, 4, "the account is out of credits."),
(SpidraRateLimitError, 5, "too many requests. Wait a moment and run it again."),
(SpidraTimeoutError, 6, "the page took too long to read."),
(SpidraJobFailedError, 7, None), # its own message names the page and the reason
(SpidraValidationError, 8, "the request was rejected as invalid."),
(SpidraServerError, 9, "Spidra had a problem on its side. Try again."),
]
EMPTY_PAGE = 10 # the page loaded but had no content
BAD_URL = 11 # the argument is not a web address
UNKNOWN = 1 # any other error
class Failure(Exception):
def __init__(self, message, code):
super().__init__(message)
self.message, self.code = message, code
def explain(error):
"""The first rule that matches wins, so list the specific errors before the general one."""
for kind, code, text in RULES:
if isinstance(error, kind):
return Failure(text or str(error), code)
return Failure(str(error), UNKNOWN)
A page that doesn't exist raises SpidraJobFailedError, and the message already names the page and the reason, so it's passed through as it is. This is the exact message Spidra returns for a missing page:
error = SpidraJobFailedError(
"Failed to scrape https://books.toscrape.com/this-page-does-not-exist: The page was not "
"delivered: the page answered HTTP 404. Nothing was charged. Check the URL."
)
failure = explain(error)
print(failure.code, "|", failure.message)A page can also load and still be empty: a cookie wall, a login prompt or a blank shell. Spidra marks these with a low_content flag. The flag alone would also catch short pages that are real, so the command treats a page as empty when the text is blank, or when the flag is set and there's very little of it:
def check_url(url):
parts = urlsplit(url)
if parts.scheme not in ("http", "https") or not parts.hostname:
raise Failure(
f"{url!r} is not a web address. Use one that starts with http:// or https://", BAD_URL
)
def fetch(client, url, full=False, proxy=False):
"""One URL in, the page's Markdown out. Content only strips menus, footers and sidebars."""
options = {"extract_content_only": not full} | ({"use_proxy": True} if proxy else {})
try:
return client.scrape(url, **options)
except SpidraError as error:
raise explain(error) from error
def check_content(result, url):
"""A page can load and still be empty: a cookie wall, a login prompt or a blank shell."""
text, page = result.content.strip(), (result.data or [None])[0]
if not text or (page and page.low_content and len(text) < 200):
raise Failure(f"{url} came back empty or nearly empty. Nothing was saved.", EMPTY_PAGE)
blank = SimpleNamespace(content=" \n", data=[])
try:
check_content(blank, "https://example.com/cookie-wall")
except Failure as failure:
print(failure.code, "|", failure.message)
Output:
10 | https://example.com/cookie-wall came back empty or nearly empty. Nothing was saved.
Failed pages aren't charged. A page that loads a cookie wall can come back nearly empty and still be charged, which is one more reason to check for it.
The finished command
Now put the pieces together. convert fetches, checks, cleans and saves. main reads the arguments, turns every failure into one line on stderr and an exit code, and prints the path of the saved file.
def convert(client, url, out_dir=".", full=False, proxy=False, today=None):
result = fetch(client, url, full, proxy)
check_content(result, url)
markdown = clean_markdown(result.content)
page = (result.data or [None])[0]
title = title_of(markdown, page.title if page else None, url)
path = Path(out_dir) / f"{slug_for(url)}.md"
path.parent.mkdir(parents=True, exist_ok=True)
write_atomic(path, build_document(url, title, markdown, (today or date.today()).isoformat()))
return path
def make_client():
if not os.environ.get("SPIDRA_API_KEY"):
raise Failure("set SPIDRA_API_KEY to your key first.", 3)
return Spidra(api_key=os.environ["SPIDRA_API_KEY"], api_url=os.environ.get("SPIDRA_API_URL"))
def main(argv=None, client=None):
ask = argparse.ArgumentParser(description="Save a web page as Markdown with its source URL.")
ask.add_argument("url")
ask.add_argument("-o", "--out-dir", default=".", help="folder for the file (default: here)")
ask.add_argument("--full", action="store_true", help="keep menus and footers")
ask.add_argument("--proxy", action="store_true", help="read the page through a proxy")
args = ask.parse_args(argv)
try:
check_url(args.url)
path = convert(client or make_client(), args.url, args.out_dir, args.full, args.proxy)
except Failure as failure:
print(f"error: {failure.message}", file=sys.stderr)
return failure.code
print(f"Saved {path}")
return 0
Save the pieces as web2md.py and run it with a URL and a folder:
# web2md.py ... the functions above, then:
if __name__ == "__main__":
sys.exit(main())
python web2md.py https://example.com/docs/page -o notes writes the Markdown to notes/, and prints its path. --full keeps the menus and footers, and --proxy reads the page through a proxy, which is worth trying when a page comes back empty.
If the page can't be read, nothing is written, the reason goes to stderr, and the exit code says which kind of failure it was. A script can tell a missing key from a missing page.
Choosing an approach
Situation | Use |
|---|---|
HTML you already have, or pages you control that don't change |
|
Articles from many sites, as static HTML | An extraction library such as |
Pages that need JavaScript, or sites with banners, walls and layouts that change | Scrape with |
Many pages | A batch scrape, or Map to find the pages first. |
If the HTML is yours and stable, keep requests. The cost of the other routes only pays for itself when pages change under you.
What this doesn't handle
Some pages stay out of reach. Pages behind a login aren't available to a plain scrape, and some sites block automated access whatever you do. The command reports them, which is the point of the error codes, but it can't read them.
Content-only extraction is not a summary: a long reference page stays long. Markdown is also a lossy format, so a complicated table or an unusual layout may not survive the trip. And the command handles one URL per run. For a list of pages, our post on scraping multiple URLs in Python covers batches and failures, and Map finds the pages on a site first.
Check each site's terms before you collect from it, and read our guide on whether web scraping is legal. For why Markdown is the format to feed a model, see HTML vs Markdown for LLMs.
From one page to a pipeline
You now have a command that turns a public URL into a Markdown file with its source, and fails clearly when it can't. The cleanup is a few rules instead of a selector list per site, and it stays that way as you add sites.
That's why Spidra is worth reaching for once a page stops being simple. Here is what it does for this job:
- It reads pages like a browser. Content that JavaScript draws after the page loads is in the Markdown, where a plain request sees nothing.
- It strips the furniture before you see it. Content-only extraction removes menus, headers, footers and sidebars, so your code handles the exceptions, not every page.
- It says when a page can't be read. You get a specific error, and a failed page isn't charged.
- It reads documents too. The same call turns PDF, Word, Excel, PowerPoint, ODT, RTF, EPUB and CSV files into Markdown.
- It scales past one URL. A single request takes up to 50 URLs and returns a separate result for each. Map finds the pages first, a crawl reads a whole docs site, and Search finds pages by query.
- It keeps working after you build it. Monitors check a page on a schedule and alert you when it changes, so the Markdown you ingested doesn't go stale.
That's the step from a one-off converter to recurring website ingestion: documentation for a retrieval pipeline, pages for a model to read, content that stays current. If you only need Markdown from pages you control, requests and a few selectors are enough. The moment pages change, load with JavaScript or block your script, a scrape call is the cheaper thing to maintain.
Create a free account, run web2md.py on a page your requests script can't read, and then point it at your own list of URLs.