You fetch a page with requests, and the HTML is missing the content you can see in your browser. A visitor sees it because a script builds the page after it loads, and Requests doesn't run scripts. If you need to scrape a JavaScript website with Python, the first job is finding out where the content lives, so you only reach for a browser when you have to.
This tutorial does that, then retrieves a page's quotes, authors and tags with a script that checks what it got. You'll start with the failed request and read the response before choosing a tool.
Install the packages, and put your key in the environment as SPIDRA_API_KEY:
# pip install spidra requests playwright
import json
import os
import re
import sys
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import requests
from playwright.sync_api import sync_playwright
from spidra import BrowserAction, ScrapeParams, ScrapeUrl, Spidra
Requests returns an empty shell
Here is the failure on a real demo page, quotes.toscrape.com/js/. It shows ten quotes in a browser. Count them in the response:
# needs network
url = "https://quotes.toscrape.com/js/"
html = requests.get(url, timeout=30).text
print("quotes in the raw HTML:", html.count('class="quote"'))Output:
quotes in the raw HTML: 0Zero. The quotes are drawn by JavaScript after the page loads. There are several ways a page can do that, and they aren't equally hard to beat. To try each one offline, a tiny local site stands in for a real one. It's a made-up quotes site that builds its page with scripts in three ways, plus one ordinary page:
QUOTES = [
{"text": "Be kind.", "author": "A. Writer", "tags": ["kindness"]},
{"text": "Write code.", "author": "B. Maker", "tags": ["code", "craft"]},
]
DRAW = (
"for (const q of data) { const d = document.createElement('div'); d.className = 'quote';"
" d.textContent = q.text; app.appendChild(d); }"
)STATIC = '<div class="quote">Be kind.</div><div class="quote">Write code.</div>'
EMBEDDED = f'<div id="app"></div><script>var data = {json.dumps(QUOTES)}; {DRAW}</script>'
FETCHED = (
'<div id="app"></div><script>fetch("/api/quotes?page=1").then(r => r.json())'
".then(data => { " + DRAW + " });</script>"
)
COMPUTED = (
'<div id="app"></div><script>const w = ["Be", "kind."]; const data = [{text: w.join(" ")}];'
" " + DRAW + "</script>"
)
SITE = {
"/static": (200, "text/html", STATIC),
"/embedded": (200, "text/html", EMBEDDED),
"/fetched": (200, "text/html", FETCHED),
"/computed": (200, "text/html", COMPUTED),
"/api/quotes": (200, "application/json", json.dumps(QUOTES)),
}class Site(BaseHTTPRequestHandler):
pages = {}
def do_GET(self):
status, kind, body = self.pages.get(
self.path.split("?")[0], (404, "text/html", "Not found")
)
data = body.encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", f"{kind}; charset=utf-8")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
log_message = lambda *args: Nonedef serve(pages, port=8801):
"""{path: (status, content type, body)} -> the base URL of a site serving those pages."""
handler = type("Handler", (Site,), {"pages": pages})
threading.Thread(
target=ThreadingHTTPServer(("127.0.0.1", port), handler).serve_forever, daemon=True
).start()
return f"http://127.0.0.1:{port}"base = serve(SITE)
for path in ("/static", "/embedded", "/fetched", "/computed"):
html = requests.get(base + path, timeout=10).text
print(f"{path:10s}", html.count('class="quote"'), "quotes in the raw HTML")Output:
/static 2 quotes in the raw HTML
/embedded 0 quotes in the raw HTML
/fetched 0 quotes in the raw HTML
/computed 0 quotes in the raw HTMLThree of these pages look the same to Requests: no quotes. They aren't the same problem.
Read the response before you reach for a browser
A script can put content in a page in three ways, and you can tell them apart by reading the response you already have:
- The data ships inside the page, as JSON in a
<script>tag. The script only turns it into HTML. - The page asks an API for the data. The request is in the script, and you can make it yourself.
- The content only exists once the scripts run. There's nothing to parse and no request to copy.
Only the third needs a browser. This code checks for each in order, cheapest first. It finds JSON assigned in scripts, and requires something shaped like records, so a list of two words doesn't count:
SCRIPT = re.compile(r"<script([^>]*)>(.*?)</script>", re.S | re.I)
ASSIGNMENT = re.compile(r"(?:var|let|const)\s+(\w+)\s*=\s*|window\.(\w+)\s*=\s*")
CALLS = [
r"fetch\(\s*[\"'`]([^\"'`]+)",
r"axios\.(?:get|post)\(\s*[\"']([^\"']+)",
r"\$\.(?:get|getJSON|post|ajax)\(\s*[\"']([^\"']+)",
r"\.open\(\s*[\"'](?:GET|POST)[\"']\s*,\s*[\"']([^\"']+)",
]def skip_string(text, i):
"""The index after the string opening at text[i]; a backslash escapes the next character."""
quote, i = text[i], i + 1
while i < len(text) and text[i] != quote:
i += 2 if text[i] == "\\" else 1
return i + 1def literal_at(text, start):
"""The bracketed literal at text[start], or None. Brackets inside strings are skipped."""
depth, i = 0, start
while i < len(text):
if text[i] in "\"'":
i = skip_string(text, i)
continue
depth += (text[i] in "[{") - (text[i] in "]}")
i += 1
if depth == 0:
return text[start:i]
return None
def json_values(body):
"""Every assignment in a script whose value is valid JSON: [(name, value)]."""
found = []
for match in ASSIGNMENT.finditer(body):
start = match.end()
text = literal_at(body, start) if body[start : start + 1] in ("[", "{") else None
try:
found.append((match.group(1) or match.group(2), json.loads(text))) if text else None
except ValueError:
continue # a JavaScript object literal is not always JSON
return found
def has_records(value, minimum=2, depth=3):
"""True for a list of objects with several fields, however deep it sits inside the JSON."""
if isinstance(value, list):
rows = [v for v in value if isinstance(v, dict) and len(v) >= 2]
return len(rows) >= minimum or (
depth > 0 and any(has_records(v, minimum, depth - 1) for v in value)
)
if isinstance(value, dict) and depth > 0:
return any(has_records(v, minimum, depth - 1) for v in value.values())
return False
def embedded_json(html, minimum=2):
"""JSON the page ships inside script tags that looks like data: [(name, value)]."""
found = []
for attributes, body in SCRIPT.findall(html):
if "json" in attributes.lower():
try:
found.append(("script", json.loads(body)))
except ValueError:
pass
found += json_values(body)
return [(name, value) for name, value in found if has_records(value, minimum)]
def api_urls(html):
"""URLs that the page's own scripts request, in order, without repeats."""
urls = [u for pattern in CALLS for u in re.findall(pattern, html)]
return list(dict.fromkeys(urls))
def visible_html(html):
"""The response without script bodies, so markup inside a JavaScript string is ignored."""
return SCRIPT.sub("<script></script>", html)
def diagnose(html, marker='class="quote"'):
"""Where is the content? Check the response in the order that costs you least."""
visible = visible_html(html)
if marker in visible:
return {"verdict": "content_in_html", "evidence": f"{visible.count(marker)} x {marker}"}
if blobs := embedded_json(html):
return {"verdict": "embedded_json", "evidence": [n for n, _ in blobs]}
if calls := api_urls(html):
return {"verdict": "api_call", "evidence": calls}
return {
"verdict": "needs_browser",
"evidence": "nothing to parse and nothing to copy",
}
diagnose ignores script bodies when it looks for the content, because a script can contain markup in a string ('<div class="quote">') without the page containing any. Here's the verdict for each stand-in page:
for path in ("/static", "/embedded", "/fetched", "/computed"):
html = requests.get(base + path, timeout=10).text
print(f"{path:10s}", diagnose(html))Output:
/static {'verdict': 'content_in_html', 'evidence': '2 x class="quote"'}
/embedded {'verdict': 'embedded_json', 'evidence': ['data']}
/fetched {'verdict': 'api_call', 'evidence': ['/api/quotes?page=1']}
/computed {'verdict': 'needs_browser', 'evidence': 'nothing to parse and nothing to copy'}The embedded case needs no browser. The data is already in the response, so parse it:
html = requests.get(base + "/embedded", timeout=10).text
name, records = embedded_json(html)[0]
print(name, "->", records[0])Output:
data -> {'text': 'Be kind.', 'author': 'A. Writer', 'tags': ['kindness']}So does the API case. Make the request the script makes:
html = requests.get(base + "/fetched", timeout=10).text
url = api_urls(html)[0]
print(url, "->", requests.get(base + url, timeout=10).json()[0])Output:
/api/quotes?page=1 -> {'text': 'Be kind.', 'author': 'A. Writer', 'tags': ['kindness']}Both are faster and cheaper than a browser, and they're the right answer when diagnose finds them. This is a heuristic, not a guarantee. A page can build its data in ways it doesn't see, so treat needs_browser as "look closer" and the other verdicts as shortcuts worth trying first.
When the content only exists after scripts run
The last page, /computed, is the case that needs a browser. Its data is built by the script from pieces, so there's no JSON to find and no request to copy. Something has to run the script and read the page that results.
You can run that browser yourself with Playwright. This is its standard flow: launch a browser, open the page, wait for the content to appear, read it, and close the browser even if waiting fails:
def read_quote(card):
return {
"text": card.locator(".text").inner_text(),
"author": card.locator(".author").inner_text(),
"tags": card.locator(".tag").all_inner_texts(),
}
def scrape_with_playwright(url, selector=".quote"):
"""Launch a browser, wait for the content to appear, read it, and always close the browser."""
with sync_playwright() as p:
browser = p.chromium.launch()
try:
page = browser.new_page()
page.goto(url)
page.wait_for_selector(selector)
cards = page.locator(selector)
return [read_quote(cards.nth(i)) for i in range(cards.count())]
finally:
browser.close()
It needs two installs: the package, and the browser it drives (playwright install chromium). It works well for a few pages. Running browsers yourself also means keeping their binaries current, installing their system libraries on every server and in CI, waiting for the right moment on every page, and watching their memory as you scale. When a site starts blocking you, proxies and bot checks are yours to solve.
Scrape it with Spidra
Scrape opens the page in a browser for you. You send the URL and a schema for the fields you want, and the quotes come back as Python data:
URL = "https://quotes.toscrape.com/js/"
QUOTE = {
"type": "object",
"properties": {
"text": {"type": "string"},
"author": {"type": "string"},
"tags": {"type": "array", "items": {"type": "string"}},
},
}
SCHEMA = {"type": "object", "properties": {"quotes": {"type": "array", "items": QUOTE}}}
# needs SPIDRA_API_KEY
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
page = ScrapeUrl(url=URL)
params = ScrapeParams(urls=[page], prompt="Extract every quote on the page", schema=SCHEMA)
quotes = client.scrape(params).content["quotes"]
print(len(quotes), "quotes")
for quote in quotes[:3]:
print(quote["author"], "|", quote["text"][:60], "|", quote["tags"])Output:
10 quotes
Albert Einstein | The world as we have created it is a process of our thinking | ['change', 'deep-thoughts', 'thinking', 'world']
J.K. Rowling | It is our choices, Harry, that show what we truly are, far m | ['abilities', 'choices']
Albert Einstein | There are only two ways to live your life. One is as though | ['inspirational', 'life', 'live', 'miracle', 'miracles']Ten quotes, where Requests found none. A browser is the default, and you can skip it when you don't need one. scrape_mode="fast" fetches the HTML without a browser, which suits static pages. If the HTML turns out to be an empty shell, Fast Mode falls back to a browser, so on a JavaScript page it won't save you anything.
Use actions when the page needs interaction
Some content appears only after you do something: scroll, click, wait. Actions run in order before the page is read. quotes.toscrape.com/scroll loads more quotes as you scroll, so scroll to the bottom, give the new quotes a moment to arrive, and ask for a count:
SCROLL_TO_END = [
BrowserAction(type="scroll", to="100%"),
BrowserAction(type="wait", duration=1500),
]
WAIT_FOR_QUOTES = [BrowserAction(type="waitFor", selector=".quote", timeout=15000)]
# needs SPIDRA_API_KEY
page = ScrapeUrl(url="https://quotes.toscrape.com/scroll", actions=SCROLL_TO_END)
prompt = 'How many quotes are on the page? Return {"count": number}'
result = client.scrape(ScrapeParams(urls=[page], prompt=prompt))
print(result.content)Output:
{'count': 20}Twenty quotes after the scroll. waitFor waits until an element appears, which suits content that arrives late. scrollUntilEnd keeps scrolling, or clicking a "Load more" button, until the page stops growing, and evaluate runs JavaScript in the page and returns the value. The actions docs list them all.
A step that fails stops the job, and a failed job costs nothing. The error names the step. Mark a step optional when the page might not have what it needs, such as a cookie banner, and it's skipped with a warning instead. Each URL then reports every step:
{ "index": 1, "type": "click", "status": "skipped", "error": "no element matches \"#accept-cookies\" (waited 2 seconds)." }
{ "index": 2, "type": "click", "status": "ok" }
Actions work on Scrape, not on batch, which takes plain URLs. A URL can have up to 50 actions, and the waits in them add up to 60 seconds.
The finished script
Put it together with a check that matters here: a page that hasn't finished rendering returns an empty list, and an empty list shouldn't be saved as a result.
def fetch_quotes(client, url=URL, actions=None):
"""Open the page in a browser, run the actions in order, then read the quotes."""
params = ScrapeParams(
urls=[ScrapeUrl(url=url, actions=actions)],
prompt="Extract every quote on the page",
schema=SCHEMA,
)
return client.scrape(params).content["quotes"]
def check_quotes(quotes):
"""An empty list is a failure, not a result. Every quote needs its text and its author."""
if not quotes:
raise ValueError("no quotes came back: the page may not have finished rendering")
bad = [q for q in quotes if not (q.get("text") and q.get("author"))]
if bad:
raise ValueError(f"{len(bad)} of {len(quotes)} quotes have no text or no author")
if len({q["text"] for q in quotes}) < len(quotes):
raise ValueError("some quotes appear twice: a scroll may have loaded the same page again")
return quotes
def main(url=URL):
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
quotes = check_quotes(fetch_quotes(client, url))
with open("quotes.json", "w", encoding="utf-8") as f:
json.dump(quotes, f, ensure_ascii=False, indent=2)
print(f"Saved {len(quotes)} quotes to quotes.json")
Save the schema, the actions and these functions as js_quotes.py, and run python js_quotes.py to write quotes.json. It stops with a clear message instead of saving an empty file, and a missing page raises an error that says why. Pass a different URL as the first argument, and swap in SCROLL_TO_END or WAIT_FOR_QUOTES when the page needs them.
Choosing an approach
Situation | Use |
|---|---|
The content is in the response: static HTML, embedded JSON, or an API the page calls |
|
A few pages, and you're comfortable running a browser | Playwright directly. |
Static pages, at scale | Scrape with |
Pages that need scripts, scrolling or clicking, many different sites, or sites that block scripts | Scrape with actions, so you don't run the browsers. |
What this doesn't handle
Pages behind a login need a session, and the cookies have to be your own. Some sites block automated access even with a browser, proxies and CAPTCHA solving. Spidra handles most bot protection, not all of it, and a page it can't read comes back as an error that isn't charged. Proxies and CAPTCHA solving are options on a scrape when a site pushes back.
Extraction reads what the browser rendered, and it can misread. That's why check_quotes exists, and why you should compare a sample against the page.
The demo site is a practice site built for scraping. Check the terms of any other site before you collect from it, and read our guide on whether web scraping is legal. For how headless browsers work, see what a headless browser is, and for more on actions, our browser actions guide.
Skip the browser infrastructure
You now have a way to scrape a JavaScript website with Python that starts with the response. You can tell when requests is enough, when a browser is unavoidable, and a script that retrieves a page's content and refuses to save an empty result.
That last step is where running browsers yourself gets expensive, and where Spidra earns its place. Here is what it takes off your plate:
- No browser to install or keep running. There are no binaries to update, no system libraries to add to every server, and no memory to watch as you scale.
- A browser when you need one, and none when you don't. The default renders the page. Fast Mode skips the browser for static pages.
- Interaction without writing a script for it. Scroll, click, wait, type and run JavaScript with actions. A failed step fails clearly and costs nothing.
- Bot protection handled for you. Spidra handles most bot protection, with proxies and CAPTCHA solving available when a site pushes back.
- Structured data, not parsing code. Send a schema and get the fields back as Python data, ready for a check like the one above.
- A path past one page. A batch takes up to 50 URLs and gives each its own result. Map finds the pages first, a crawl reads a whole site, and monitors alert you when a page changes.
If the response already holds your data, use requests and skip all of this. If you scrape a handful of pages and enjoy running Playwright, that's a fair choice too. But the moment pages need scripts, scrolling or clicking, or many sites with their own quirks, a managed browser is the cheaper thing to maintain.
Create a free account, run js_quotes.py on the page your Requests script came back empty from, and then point it at your own.