Every store describes a product its own way: a price in a paragraph on one site, in a data attribute on another, in a meta tag on a third. If you need to extract consistent JSON from different websites, reading one page isn't the hard part. Getting the same field names, the same types, and the same meaning from all of them is.
This tutorial builds that. One schema returns the same shape from three different stores, a normalizer makes prices and currencies mean the same thing, two layers of validation catch records you shouldn't trust, and every record keeps the page it came from. You'll also learn how to measure its accuracy on your own sites before you rely on it.
Install the packages, and put your key in the environment as SPIDRA_API_KEY:
# pip install spidra beautifulsoup4 jsonschema
import json
import os
import re
from collections import Counter
from decimal import ROUND_HALF_UP, Decimal
from math import sqrt
from types import SimpleNamespace
from urllib.parse import urlsplit
from bs4 import BeautifulSoup
from jsonschema import Draft202012Validator, FormatChecker
from spidra import Spidra
A parser per site works until a page changes
The ordinary route is one parser for each site. Here are three made-up stores, on reserved .test domains, that mark up the same facts three different ways:
NORTH = (
'<h1>The Quiet Orchard</h1><p class="price">£12.99</p><p class="stock">In stock</p>'
"<table><tr><th>UPC</th><td>q1w2e3</td></tr></table>"
)
GADGET = (
'<div class="card"><h2 class="title">Aero Laptop 14</h2>'
'<span class="p" data-amount="499.00" data-cur="USD"></span></div>'
)
MOSSLING = (
'<meta itemprop="name" content="Mossling"><meta itemprop="price" content="63.00">'
'<meta itemprop="priceCurrency" content="GBP"><p>4 in stock</p><span class="sku">M-204</span>'
)Each store gets its own parser, and all three return the same five keys:
def parse_north(html):
page = BeautifulSoup(html, "html.parser")
price = page.select_one("p.price").text
return {
"name": page.h1.text,
"price": float(price[1:]),
"currency": price[0],
"in_stock": "in stock" in page.select_one("p.stock").text.lower(),
"sku": page.find("th", string="UPC").find_next("td").text,
}
def parse_gadget(html):
page = BeautifulSoup(html, "html.parser")
amount = page.select_one("span.p")
return {
"name": page.select_one("h2.title").text,
"price": float(amount["data-amount"]),
"currency": amount["data-cur"],
"in_stock": None,
"sku": None,
}def parse_mossling(html):
page = BeautifulSoup(html, "html.parser")
meta = lambda key: page.find("meta", itemprop=key)["content"]
return {
"name": meta("name"),
"price": float(meta("price")),
"currency": meta("priceCurrency"),
"in_stock": "in stock" in page.text.lower(),
"sku": page.select_one("span.sku").text,
}
pages = ((parse_north, NORTH), (parse_gadget, GADGET), (parse_mossling, MOSSLING))
for parse, html in pages:
print(parse(html))Output:
{'name': 'The Quiet Orchard', 'price': 12.99, 'currency': '£', 'in_stock': True, 'sku': 'q1w2e3'}
{'name': 'Aero Laptop 14', 'price': 499.0, 'currency': 'USD', 'in_stock': None, 'sku': None}
{'name': 'Mossling', 'price': 63.0, 'currency': 'GBP', 'in_stock': True, 'sku': 'M-204'}Now the first store renames a CSS class in a redesign:
try:
parse_north(NORTH.replace('class="price"', 'class="amount"'))
except AttributeError as error:
print(f"{type(error).__name__}: {error}")Output:
AttributeError: 'NoneType' object has no attribute 'text'The parser doesn't return a wrong answer. It stops. With three sites, that's a small repair. With fifty, it's a standing job, and every new site means a new parser. The alternative is to describe the data you want once and let the extractor find it, which is what Spidra's AI scraper does when you send it a schema.
Send one schema with every URL
Describe the fields once. A type list with "null" marks a field a page may not show, and required marks the fields every record needs:
schema = {
"type": "object",
"required": ["name", "price"],
"properties": {
"name": {"type": "string"},
"price": {"type": "number"},
"currency": {"type": ["string", "null"]},
"in_stock": {"type": ["boolean", "null"]},
"sku": {"type": ["string", "null"]},
},
}
Send every URL in one batch with that schema. Each URL gets its own result:
# extract.py
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
urls = [
"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
"https://scrapeme.live/shop/Bulbasaur/",
"https://webscraper.io/test-sites/e-commerce/allinone/product/40",
]
batch = client.batch_scrape(urls, prompt="Extract the product", schema=schema)
for item in batch.items:
print(item.status, item.result)Output:
completed {'name': 'A Light in the Attic', 'price': 51.77, 'currency': '£', 'in_stock': True, 'sku': 'a897fe39b1053632'}
completed {'name': 'Bulbasaur', 'price': 63, 'currency': '£', 'in_stock': True, 'sku': '4391'}
completed {'name': 'Dell XPS 13', 'price': 1281.99, 'currency': '$', 'in_stock': None, 'sku': None}Three layouts, one shape. Prices are numbers. The laptop page shows no stock status or SKU, so those come back as None instead of a guess.
Without a schema, every page answers differently
Here are the same three pages with only a prompt:
batch = client.batch_scrape(
urls,
prompt="Extract the product name, price and whether it's in stock",
output="json",
)
for item in batch.items:
print(item.result)Output:
{'product_name': 'A Light in the Attic', 'price': '£51.77', 'in_stock': True}
[{'product_name': 'Bulbasaur', 'price': '£63.00', 'in_stock': True}]
{'product_name': 'Dell XPS 13', 'price': '$1281.99', 'in_stock': True}Three surprises in three rows. Prices are strings with the symbol attached. The second result came back wrapped in a list. And the laptop is in_stock: True here but None with the schema, because the page doesn't say. Without a schema the extractor fills gaps and picks shapes. With one, it doesn't. Our post on getting structured data from any website covers the single-call version.
Write field descriptions as instructions
A schema fixes names and types. It can't say what a field means on a page it hasn't seen, so put that in the descriptions. The extractor reads them as instructions:
def field(kind, description):
return {"type": kind, "description": description}
schema_v2 = {
"type": "object",
"required": ["name", "price"],
"properties": {
"name": field("string", "The product's name exactly as the page shows it"),
"price": field("number", "The price as a plain number, without the currency symbol"),
"currency": field(["string", "null"], "The currency symbol or code beside the price"),
"in_stock": field(["boolean", "null"], "true or false only if the page says, else null"),
"sku": field(["string", "null"], "The SKU or product code, or null if the page shows none"),
},
}Keep what you send simple. Spidra ignores some JSON Schema keywords, such as anyOf and $ref, and lists them in schema_warnings on the response, so write a nullable field as a type list. The structured output docs cover what a schema accepts. Anything stricter, like value ranges, belongs in your own validator, below.
Normalize in your own code
Spidra returns what the page shows: £51.77 becomes 51.77 with £ beside it. Two sites can still mean different things by the same symbol, so convert to one convention yourself. We'll store prices as whole minor units (pence, cents) with an ISO currency code. The first table maps each source to its currency:
SOURCE_CURRENCY = {"books.toscrape.com": "GBP", "scrapeme.live": "GBP", "webscraper.io": "USD"}
SYMBOLS = {
"£": {"GBP", "FKP", "GIP", "SHP", "SSP"},
"$": {"USD", "CAD", "AUD", "NZD"},
"€": {"EUR"},
}
EXPONENT = {"JPY": 0, "KRW": 0}
def currency_code(value, host):
"""An ISO code from a code or symbol. A symbol many currencies share takes the source's."""
text = (value or "").strip()
if len(text) == 3 and text.isalpha():
return text.upper()
options = SYMBOLS.get(text, set())
if len(options) == 1:
return next(iter(options))
return SOURCE_CURRENCY.get(host) if SOURCE_CURRENCY.get(host) in options else None
def minor_units(amount, code):
"""Whole pence, cents or yen as an integer, rounded half up."""
scaled = Decimal(str(amount)) * 10 ** EXPONENT.get(code, 2)
return int(scaled.quantize(Decimal(1), ROUND_HALF_UP))def is_price(value):
return isinstance(value, (int, float)) and not isinstance(value, bool) and value > 0
def clean_text(value):
return " ".join(value.split()) if isinstance(value, str) and value.strip() else Nonedef normalize_record(raw, host):
"""-> (record, problems). A value that can't be made safe becomes None and is listed."""
code, problems = currency_code(raw.get("currency"), host), []
if not is_price(raw.get("price")):
problems.append(f"price is not a positive number: {raw.get('price')!r}")
if code is None:
problems.append(f"currency {raw.get('currency')!r} is missing or ambiguous on {host}")
price = minor_units(raw["price"], code or "") if is_price(raw.get("price")) else None
in_stock = raw.get("in_stock") if isinstance(raw.get("in_stock"), bool) else None
record = {"name": clean_text(raw.get("name")), "price_minor": price, "currency": code}
return record | {"in_stock": in_stock, "sku": clean_text(raw.get("sku"))}, problemsTo follow along without a key, paste the three results from the output above:
results = [
{"name": "A Light in the Attic", "price": 51.77, "currency": "£", "in_stock": True,
"sku": "a897fe39b1053632"},
{"name": "Bulbasaur", "price": 63, "currency": "£", "in_stock": True, "sku": "4391"},
{"name": "Dell XPS 13", "price": 1281.99, "currency": "$", "in_stock": None, "sku": None},
]
hosts = ["books.toscrape.com", "scrapeme.live", "webscraper.io"]
for raw, host in zip(results, hosts):
record, problems = normalize_record(raw, host)
print(record, problems or "")Output:
{'name': 'A Light in the Attic', 'price_minor': 5177, 'currency': 'GBP', 'in_stock': True, 'sku': 'a897fe39b1053632'}
{'name': 'Bulbasaur', 'price_minor': 6300, 'currency': 'GBP', 'in_stock': True, 'sku': '4391'}
{'name': 'Dell XPS 13', 'price_minor': 128199, 'currency': 'USD', 'in_stock': None, 'sku': None} £ is shared by five currencies and $ by many more, so the symbol alone can't decide. The source's currency settles it, and a symbol that contradicts the source is reported, not repaired:
odd = {"name": "x", "price": 5, "currency": "$"}
record, problems = normalize_record(odd, "books.toscrape.com")
print(record["currency"], problems)Output:
None ["currency '$' is missing or ambiguous on books.toscrape.com"]Validate the shape, then the meaning
Check each record twice. The first check is the shape, with jsonschema. It's stricter than the schema you sent: no extra keys, a positive integer price, a three-letter currency.
SOURCE_SCHEMA = {
"type": "object",
"required": ["url", "job_id", "fetched_on", "schema_version"],
"properties": {
"url": {"type": "string"},
"job_id": {"type": ["string", "null"]},
"fetched_on": {"type": "string", "format": "date"},
"schema_version": {"type": "integer"},
},
}RECORD_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["name", "price_minor", "currency", "in_stock", "sku", "source"],
"properties": {
"name": {"type": "string", "minLength": 1},
"price_minor": {"type": "integer", "exclusiveMinimum": 0},
"currency": {"type": "string", "pattern": "^[A-Z]{3}$"},
"in_stock": {"type": ["boolean", "null"]},
"sku": {"type": ["string", "null"], "minLength": 1},
"source": SOURCE_SCHEMA,
},
}def schema_problems(record, check_formats=True):
"""Shape problems as 'path: rule'. Formats such as date only count with a format checker."""
checker = FormatChecker() if check_formats else None
validator = Draft202012Validator(RECORD_SCHEMA, format_checker=checker)
errors = sorted(validator.iter_errors(record), key=lambda e: [str(p) for p in e.path])
return [f"{'/'.join(map(str, e.path)) or 'record'}: {e.validator}" for e in errors]The second check is meaning. A record can have the right shape and still be wrong: a name that's really a paragraph, or a name that's really a price.
def semantic_problems(record):
"""Things the shape allows but a person would doubt."""
found, name = [], record.get("name") or ""
if len(name) > 200:
found.append("name is longer than 200 characters: probably a description")
if re.match(r"^[£$€]\s?\d", name):
found.append("name looks like a price")
if record.get("sku") and record.get("sku") == name:
found.append("sku repeats the name")
if (record.get("price_minor") or 0) > 100_000_000:
found.append("price is implausibly large")
return foundurl = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
source = {"url": url, "job_id": "job-1", "fetched_on": "2026-11-16", "schema_version": 1}
record = normalize_record(results[0], hosts[0])[0] | {"source": source}
print("valid:", schema_problems(record) + semantic_problems(record) or "no problems")
print("broken:", schema_problems(record | {"price_minor": 0, "currency": "gbp", "isbn": "1"}))Output:
valid: no problems
broken: ['record: additionalProperties', 'currency: pattern', 'price_minor: exclusiveMinimum']A format such as date is only checked when you ask for it. Without a format checker, "last Tuesday" passes as a date:
late = record | {"source": source | {"fetched_on": "last Tuesday"}}
print("without a format checker:", schema_problems(late, check_formats=False) or "no problems")
print("with a format checker: ", schema_problems(late))Output:
without a format checker: no problems
with a format checker: ['source/fetched_on: format']Keep the source with every record
Every record should say where it came from. Each batch item has its own url and job_id, so attach them, along with the date and the schema version. A different schema version tells you which prompt produced a record when you change it later.
def source_block(item, version, fetched_on):
return {
"url": item.url,
"job_id": item.job_id,
"fetched_on": fetched_on,
"schema_version": version,
}def build_record(item, version, fetched_on):
"""Normalize one batch item. Raw values and every problem stay next to the record."""
host = urlsplit(item.url).hostname or ""
clean, problems = normalize_record(item.result or {}, host)
record = clean | {"source": source_block(item, version, fetched_on)}
problems += schema_problems(record) + semantic_problems(record)
return {"record": record, "raw": item.result, "problems": problems}
Failed items stay visible, with their status, instead of disappearing from the dataset:
def split_batch(items, version, fetched_on):
"""Completed items become records. Failed ones are kept with their reason, not dropped."""
done = [build_record(i, version, fetched_on) for i in items if i.status == "completed"]
failed = [{"url": i.url, "status": i.status} for i in items if i.status != "completed"]
return done, failed
The job IDs below are placeholders. The batch response supplies real ones:
urls = [
"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
"https://scrapeme.live/shop/Bulbasaur/",
"https://webscraper.io/test-sites/e-commerce/allinone/product/40",
]
items = [
SimpleNamespace(url=u, result=r, status="completed", job_id=f"job-{n}")
for n, (u, r) in enumerate(zip(urls, results), 1)
]
done, failed = split_batch(items, 1, "2026-11-16")
print(json.dumps(done[2]["record"], indent=2))
print("problems:", [d["problems"] for d in done], "| failed:", failed)
Output:
{
"name": "Dell XPS 13",
"price_minor": 128199,
"currency": "USD",
"in_stock": null,
"sku": null,
"source": {
"url": "https://webscraper.io/test-sites/e-commerce/allinone/product/40",
"job_id": "job-3",
"fetched_on": "2026-11-16",
"schema_version": 1
}
}
problems: [[], [], []] | failed: []
Measure accuracy on your own sites before you trust it
Validation can't tell you whether a plausible value is the right one. A price of 6.3 instead of 63 passes every check. To know how accurate the extraction is, compare it with values a person read off the page.
Label about ten pages per site. For each field, write down what the page shows, or None if it doesn't show the field. Then count what happened to every field: the value was right, the page had nothing and so did the extractor, the value was wrong, the extractor missed a value that was there, or it invented one that wasn't.
OUTCOMES = ("correct", "correct_null", "wrong", "missed", "invented")
def same(a, b):
"""Text ignores case and spacing. Numbers allow rounding. A boolean only equals a boolean."""
if isinstance(a, bool) or isinstance(b, bool):
return isinstance(a, bool) and isinstance(b, bool) and a == b
if isinstance(a, str) and isinstance(b, str):
return " ".join(a.split()).casefold() == " ".join(b.split()).casefold()
if isinstance(a, (int, float)) and isinstance(b, (int, float)):
return abs(a - b) < 0.005
return a == b
def outcome(gold, got):
"""The page has the value or it doesn't. The extractor returned one or it didn't."""
if gold is None:
return "correct_null" if got is None else "invented"
if got is None:
return "missed"
return "correct" if same(gold, got) else "wrong"
def score(pairs, fields):
"""pairs: [(gold dict, extracted dict)] -> {field: Counter of outcomes}."""
table = {f: Counter() for f in fields}
for gold, got in pairs:
for f in fields:
table[f][outcome(gold.get(f), got.get(f))] += 1
return table
A share measured on a handful of pages is loose, so report it with an interval:
def wilson(hits, n, z=1.96):
"""A 95% interval for a share. With few pages it is wide, which is the point."""
if not n:
return (0.0, 0.0)
p, centre = hits / n, 1 + z * z / n
spread = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return ((p + z * z / (2 * n) - spread) / centre, (p + z * z / (2 * n) + spread) / centre)
def report_row(field, counts):
n, hits = sum(counts.values()), counts["correct"] + counts["correct_null"]
low, high = wilson(hits, n)
cells = " | ".join(str(counts[o]) for o in OUTCOMES)
return f"{field} | {cells} | {hits}/{n} ({low:.0%} to {high:.0%})"
def report(table):
"""A markdown table: one row per field, then the accuracy with its interval."""
header = "field | " + " | ".join(OUTCOMES) + " | accuracy"
return "\n".join([header] + [report_row(f, c) for f, c in table.items()])
These pages and extractions are made up, with three deliberate mistakes: an in_stock that was invented, a price that was wrong, and an in_stock that was missed:
gold = [
{"name": "Quiet Orchard", "price": 12.99, "in_stock": True},
{"name": "Aero Laptop 14", "price": 499.0, "in_stock": None},
{"name": "Mossling", "price": 63.0, "in_stock": True},
]
got = [
{"name": "Quiet Orchard", "price": 12.99, "in_stock": True},
{"name": "Aero Laptop 14", "price": 499.0, "in_stock": True},
{"name": "Mossling", "price": 6.3, "in_stock": None},
]
print(report(score(zip(gold, got), ["name", "price", "in_stock"])))Output:
field | correct | correct_null | wrong | missed | invented | accuracy
name | 3 | 0 | 0 | 0 | 0 | 3/3 (44% to 100%)
price | 2 | 0 | 1 | 0 | 0 | 2/3 (21% to 94%)
in_stock | 1 | 0 | 0 | 1 | 1 | 1/3 (6% to 79%)
Three pages give wide intervals, even where every field is right. That's the point of reporting them. Ten pages per site narrows them, and the per-field rows show you where to improve the descriptions.
Some pages fail and some fields come back wrong
Extraction reads text, and on a wide table it can shift a column. Here's a Wikipedia table read with a schema of language, first_appeared and designed_by:
# ... client and schema as before
result = client.scrape(
"https://en.wikipedia.org/wiki/Timeline_of_programming_languages",
prompt="Return the first 8 rows of the 1990s table",
schema=schema,
)
for row in result.content["rows"]:
print(row)Output:
{'language': 'Sather', 'first_appeared': 1990, 'designed_by': 'Steve Omohundro'}
{'language': 'AMOS BASIC', 'first_appeared': 1990, 'designed_by': 'François Lionet and Constantin Sotiropoulos'}
{'language': 'AMPL', 'first_appeared': 1990, 'designed_by': 'Robert Fourer, David Gay and Brian Kernighan at Bell Laboratories'}
{'language': 'Object Oberon', 'first_appeared': 1990, 'designed_by': 'H Mössenböck, J Templ, R Griesemer'}
{'language': 'J', 'first_appeared': 1990, 'designed_by': 'Kenneth E. Iverson, Roger Hui at Iverson Software'}
{'language': 'Haskell', 'first_appeared': 1990, 'designed_by': 'Miranda, Clean'}
{'language': 'EuLisp', 'first_appeared': 1990, 'designed_by': 'Common Lisp, Scheme'}
{'language': 'Z shell', 'first_appeared': 1990, 'designed_by': 'Paul Falstad at Princeton University'}
Haskell and EuLisp got the table's "Predecessor(s)" column ("Miranda, Clean") instead of the designers. Both values are plausible strings, so no validator above would catch them. Only a comparison with the page does. Check extracted tables against the page, or parse simple tables yourself, for example with pandas.read_html on the scraped HTML.
Cost follows the page's length, not the size of the answer. For a long page, scrape the Markdown and parse the part you need.
Some pages also fail outright. They come back as failed items, and failed pages aren't charged. Our post on scraping multiple URLs in Python covers retrying them.
Choosing an approach
Situation | Use |
|---|---|
One or two stable sites with simple HTML | A parser per site. It's free and exact. |
Several layouts, or layouts that change | One schema with a batch scrape, then normalize and validate. |
A long page or a wide table | The Markdown or HTML, and your own parser for the part you need. |
Content that JavaScript draws after the page loads | A tool that renders the page first, which Spidra does. |
What this doesn't handle
A schema fixes field names and types. It doesn't fix what a field means. If one site rates out of 5 and another out of 10, say so in the descriptions and check the result, because the extractor can't know your convention.
Pages behind a login aren't available to a plain request, and extraction can misread a page, which is why the last two sections exist. The three stores here are demo shops built for practice. Check each site's terms before you collect from it, and read our guide on whether web scraping is legal.
Adding a site should mean adding a URL
You now have an extractor where a new source is one more URL in the batch. One schema fixes the field names and types. The normalizer makes prices and currencies mean the same thing. Two layers of validation catch the records you shouldn't trust. Every record carries the page and job it came from, and the scorer tells you how accurate it is on your sites.
Here is what Spidra does for this job, and why each part matters more with every site you add:
- It removes the parser. A redesign that makes a selector fail never touches a schema, so the repair work from the first section doesn't pile up.
- It reads pages like a browser. Content that JavaScript draws after the page loads isn't missing from the result.
- It returns
nullfor what a page doesn't show. A missing value is visible, not a guess you find in production. - It keeps every URL separate. Each item in a batch has its own result and job ID, up to 50 URLs per request, so records stay traceable at any size.
- It charges nothing for a page it can't read. A failed URL costs you a retry, not money.
- It keeps working after the build. Monitors watch the pages you care about on a schedule and alert you when one changes.
If you have one stable site, keep the parser. The moment you have a second layout, or a layout that changes, the schema is the cheaper thing to maintain.
Create a free account, paste your own URLs into the batch request above, and label ten pages to score the result. Your fifth site takes no more work than your first.