Getting the first product right proves nothing. A listing page holds twenty of them, and the question is whether you got all twenty, with the right names.
Here's how to extract a product listing into JSON with a schema: one object holding a typed array of names, prices and availability. Then we check that array against the page, because the failures here are quiet. It's for backend developers who extract collections, not single records. If you only need one product page, structured output on a single URL is all you need.
Without a schema, the same prompt gives you different shapes
The obvious attempt is a prompt: "Extract the product name, price and whether it's in stock." On three single-product pages from three different stores, with output="json", that came back as:
{'product_name': 'A Light in the Attic', 'price': '£51.77', 'in_stock': True}
[{'product_name': 'Bulbasaur', 'price': '£63.00', 'in_stock': True}]
{'product_name': 'Dell XPS 13', 'price': '$1281.99', 'in_stock': True}Three results, three problems. The prices are strings with the symbol attached. The second result is wrapped in a list and the others aren't. And the laptop is in_stock: True even though that page never says so. The first two are annoying. The third is a guess, and you'd only catch it by reading the page. Code that expects one shape has to cope with all three, and a listing page makes it worse, because now you're guessing the shape of a collection.
The schema root has to be an object, so the array goes inside it
The fix is a JSON schema, and the first thing a schema asks of you is the shape of the whole answer. The natural instinct for a list of products is a schema whose root is an array. Current requests reject that:
# root_array.py
import os
from spidra import Spidra, SpidraValidationError
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
bad = {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}}}}
try:
client.scrape(URL, prompt="Extract every book in the listing", schema=bad)
except SpidraValidationError as e:
print(e.status, e.errors[0].split(". ")[0]) # the first sentence is the part to readThe answer is plain:
422 schema root "type" must be "object"The request is refused before a job is created. If you've seen a schema with "type": "array" at the root in an older tutorial, that's the reason it no longer works. The array goes one level down, as a property of a root object: {"products": [...]}.
That's a better shape anyway. The answer is always a dict you index by name, whatever the page held, and the root has room beside the list for a field like currency or a count if you ever want one. The list itself is result.content["products"].
Write the schema once: types, required fields and nulls
Two pieces. First the product, with only the fields that matter:
item = {
"type": "object",
"required": ["name", "price"],
"properties": {
"name": {"type": "string"},
"price": {"type": "number"},
"currency": {"type": ["string", "null"]},
"availability": {"type": ["string", "null"]},
},
}Then the root that holds a list of them:
schema = {
"type": "object",
"required": ["products"],
"properties": {"products": {"type": "array", "items": item}},
}Name and price are required, because a product row without them is useless. Currency and availability can be null, because a page might not show them, and a field that's allowed to be empty is how a schema avoids inventing a value. Availability is a string, not a boolean: the listing prints a phrase ("In stock"), and a boolean would force a yes or no on pages that say neither. If you'd rather not hand-write it, the JSON schema generator builds one from an example.
The smallest call on a real listing
We'll use the Travel category on books.toscrape.com, a public scraping sandbox. One page, one category, 11 books, each with a name, a price and a stock line. The page itself says "11 results".
Start a new file, and paste the item and schema from the section above where the comment says:
# product_listing.py
import json
import os
from spidra import Spidra
# ... item and schema from above go here
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
result = client.scrape(URL, prompt="Extract every book in the listing", schema=schema)
if result.ai_extraction_failed or result.extraction_empty:
raise SystemExit("No structured result. Check the page, then retry.")
print(result.stats.total_tokens, "tokens,", result.stats.duration_ms, "ms")
for product in result.content["products"]:
print(json.dumps(product, ensure_ascii=False))Output:
2055 tokens, 22271 ms
{"name": "It's Only the Himalayas", "price": 45.17, "currency": "£", "availability": "In stock"}
{"name": "Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond", "price": 49.43, "currency": "£", "availability": "In stock"}
{"name": "See America: A Celebration of Our National Parks & Treasured Sites", "price": 48.87, "currency": "£", "availability": "In stock"}
{"name": "Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel", "price": 36.94, "currency": "£", "availability": "In stock"}
{"name": "Under the Tuscan Sun", "price": 37.33, "currency": "£", "availability": "In stock"}
{"name": "A Summer In Europe", "price": 44.34, "currency": "£", "availability": "In stock"}
{"name": "The Great Railway Bazaar", "price": 30.54, "currency": "£", "availability": "In stock"}
{"name": "A Year in Provence (Provence #1)", "price": 56.88, "currency": "£", "availability": "In stock"}
{"name": "The Road to Little Dribbling: Adventures of an American in Britain (Notes From a Small Island #2)", "price": 23.21, "currency": "£", "availability": "In stock"}
{"name": "Neither Here nor There: Travels in Europe", "price": 38.95, "currency": "£", "availability": "In stock"}
{"name": "1,000 Places to See Before You Die", "price": 26.08, "currency": "£", "availability": "In stock"}That's the whole listing: 11 records, the number the page says it has. The first call took 22 seconds, and the reruns took about 10.
The guard is there for a reason. If the AI step fails, content falls back to the page's raw markdown, which is a string, and result.content["products"] would raise a TypeError. It doesn't mean the products weren't on the page.
The thing to notice in the output is the shape. result.content is a dict, result.content["products"] is a list, and each record has the same four keys. Prices are numbers, not strings, and the £ sits in its own field.
Check the array against the page
Getting an array back isn't the same as getting the right one. Three short checks. Imports for these blocks go at the top of product_listing.py. First, validate the shape with Pydantic, so a wrong type fails loudly:
from typing import Optional
from pydantic import BaseModel
class Product(BaseModel):
name: str
price: float
currency: Optional[str] = None
availability: Optional[str] = None
class Catalog(BaseModel):
products: list[Product]Second, get a ground truth for how many records the page shows and what their full names are. On a static demo site we can read that straight from the HTML. That works here because the demo's HTML is simple. Treat it as a test harness, not something to run on every store:
import html
import re
import requests
def page_facts(url):
page = requests.get(url, timeout=30).text
titles = [html.unescape(t) for t in re.findall(r'<h3><a [^>]*title="([^"]+)"', page)]
return page.count('class="product_pod"'), titlesThird, compare. Count alone isn't enough, because a count can match while a name is missing or cut off, so the check compares names:
from collections import Counter
def check(products, visible, full_titles):
names = [p.name for p in products]
print(len(products), "returned,", visible, "visible")
print("missing from the result:", [t for t in full_titles if t not in names])
print("duplicates:", [n for n, c in Counter(names).items() if c > 1])
print("truncated names:", [n for n in names if n.endswith("...")])
print("no availability:", sum(p.availability is None for p in products))Run it with check(Catalog.model_validate(result.content).products, *page_facts(URL)). On the Travel page it printed this:
11 returned, 11 visible
missing from the result: ['Full Moon over Noahâ\x80\x99s Ark: An Odyssey to Mount Ararat and Beyond']
duplicates: []
truncated names: []
no availability: 0The counts match, 11 and 11. The names don't: the check says one title is missing. But that title is right there in the result, as Full Moon over Noah’s Ark…. The check's own copy has â\x80\x99 where the apostrophe should be, so the bug is in our ground truth, not in the extraction. The page is UTF-8, but requests decoded it as Latin-1, which is what it does by default when a response doesn't name a charset. One line fixes it:
def page_facts(url):
response = requests.get(url, timeout=30)
response.encoding = "utf-8"
titles = [html.unescape(t) for t in re.findall(r'<h3><a [^>]*title="([^"]+)"', response.text)]
return response.text.count('class="product_pod"'), titlesWith that, nothing is missing. It's also a fair reminder of what this check is for. A matching count would have told us everything was fine, and the name comparison is what caught something. When it flags a record, check the check before you blame the extraction.
What the layout cost us
A complete array isn't the same as everything the page offers. Long titles were the thing I expected to go wrong. They came back whole, including the 97-character "The Road to Little Dribbling" one. A second run that asked for "the full title, never a shortened one" changed nothing, because nothing was shortened.
The star rating is the real omission. On this site it's drawn with a CSS class, not written as text, so there's nothing for the AI to read. Add a nullable rating to the item schema and ask for it:
rating_item = {**item, "properties": {**item["properties"], "rating": {"type": ["number", "null"]}}}
rating_schema = {**schema, "properties": {"products": {"type": "array", "items": rating_item}}}
prompt = "Extract every book in the listing with its star rating"
result = client.scrape(URL, prompt=prompt, schema=rating_schema)
print([p["rating"] for p in result.content["products"]])Output:
[None, None, None, None, None, None, None, None, None, None, None]All 11 come back None, which is null in the JSON. That's the nullable field doing its job: it returns null and doesn't invent a rating. If you need the rating, read the CSS class with a selector.
The pagination case is the one that bites hardest. The Mystery category says "32 results", and its first page holds 20 of them. Nothing is wrong with those 20. There are just more, and a listing page's schema returns only what's on that page, so a catalog needs every page. For a handful of pages, send them as a batch with the same schema:
listing_urls = [
"https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
"https://books.toscrape.com/catalogue/category/books/mystery_3/page-2.html",
]
batch = client.batch_scrape(listing_urls, prompt="Extract every book in the listing", schema=schema)
products = []
for entry in batch.items:
if entry.status == "completed":
products += (entry.result or {}).get("products", [])
print(batch.failed_count, "failed,", len(products), "products")Output:
0 failed, 32 productsThat matches the category's own count. The first page holds 20 cards and the second holds 12, and the batch returned 20 and 12: 32 products, 32 unique names, each page matching its own card count.
A batch comes back as completed even if some pages failed, so the loop checks each item. Failed pages cost nothing. If you'd rather click through pages on a single request, forEach with pagination can follow the next-page link for you, up to 50 items.
When you don't need AI for this
Before you reach for extraction, view the page source and search for application/ld+json. Some store listings ship their products as JSON-LD, sometimes a schema.org ItemList of Product entries, and a library like extruct reads that for free, with no model involved. If one site's HTML is stable, CSS selectors with parsel or BeautifulSoup do the same job, and they read things like CSS classes directly, such as the star rating the AI couldn't see.
AI extraction earns its credits somewhere else: when you're pulling from many stores whose HTML has nothing in common, and you want one schema to work across all of them without writing a selector set per site.
With the same root, many stores merge in one loop
That's the catalog-aggregation case, and it's where an object root pays off. Every listing, from any store, comes back as {"products": [...]}, so merging is the products += ... line in the loop above. Batch scraping takes up to 50 listing URLs per request, each with its own result. The Python SDK also has bulk_scrape, which takes a URL template, the items_path of the list (here "products") and a field to de-duplicate on, and merges the pages' lists for you.
On cost: every listing page cost 3 credits in these runs, whether it held 11 books or 20, because the cost follows the page and not how many records you keep. The two-page Mystery category cost 6. A batch of listings is that, per page.
Check the site's terms first
books.toscrape.com exists to be scraped. A real store is different: follow its terms and the law, and read our guide to the legal questions before you collect a catalog at scale.
Point it at a listing you already know the count of
The best test is a listing where you know how many products the page holds. Then the found-versus-visible check tells you something on the first run. Create a free account at app.spidra.io (300 credits, no card) and run what you just built on one.
