How to validate scraped JSON with Pydantic

The first page you test always works. You write a model, send its schema to Spidra, call model_validate, and a typed object comes out. The page that breaks your pipeline is the 400th: the one where a field is null, or the price is the word Free, or the extractor answered about the wrong page entirely. Scraped JSON that parses is not scraped JSON you can trust.

This post builds the part that comes after extraction. A Pydantic model does two jobs: it's the schema you send to Spidra and the contract you validate against. A short routine then sorts every result into an accepted object or a rejection that says why.

If you've seen model_json_schema() go into schema= and model_validate come out in the Python SDK docs, this starts where that stops, with everything that goes wrong next. In our run, that included Spidra telling us it had ignored part of our schema, and a page where the model approved a price of 0.

We ran it on real pages with Pydantic 2.10.3.

One model is both the schema and the contract

The code below comes from three small modules: models.py, schema_tools.py and validate.py. Put it in one file, in the order it appears, with these imports at the top, and it runs:

import json
from dataclasses import dataclass
from types import SimpleNamespace
from typing import Optional

from pydantic import BaseModel, ConfigDict, Field, ValidationError

class Book(BaseModel):
    title: str
    price: float = Field(description="The price as a number, without the currency symbol")
    in_stock: bool
    upc: str = Field(description="The UPC from the Product Information table")
    rating: Optional[int] = Field(None, description="The star rating from 1 to 5, if it is shown")
    isbn: Optional[str] = None

The Field descriptions travel with the schema. The two optional fields have defaults, so a result that leaves them out is still valid. Here's what Pydantic writes for the optional isbn, and which fields it marks as required:

def demo_schema():
    schema = Book.model_json_schema()
    print(json.dumps(schema["properties"]["isbn"], indent=2))
    print("required:", schema["required"])

Output:

{
  "anyOf": [
    {
      "type": "string"
    },
    {
      "type": "null"
    }
  ],
  "default": null,
  "title": "Isbn"
}
required: ['title', 'price', 'in_stock', 'upc']

Pydantic writes a nullable field as anyOf with a null type, plus a default and a title. The Spidra docs show nullable fields as "type": ["string", "null"]. That difference mattered in the first live job. We sent the schema exactly as Pydantic wrote it, for a book page, and Spidra's schema_warnings came back with this:

"anyOf" at schema.properties.rating is not supported and will be ignored
"anyOf" at schema.properties.isbn is not supported and will be ignored

The job ran, and the content that came back validated against Book. But Spidra had ignored anyOf on both optional fields, and anyOf is where Pydantic writes the type of an optional field. schema_warnings is where to look: it lists the keywords Spidra dropped.

Nested models make Pydantic's output stranger. A model that contains another model comes out as $defs and $ref:

class Offer(BaseModel):
    price: float
    currency: Optional[str] = None


class NestedBook(BaseModel):
    title: str
    offer: Offer

We sent the nested model three ways, for the same page:

Every variant came back with the nested object, {"price": 51.77, "currency": "£"}, and every one validated. That's less reassuring than it sounds. In the first row Spidra ignored the $ref, so the schema never said what an offer contains. The prompt, which described the offer in words, is the likely reason it came back right anyway. A warning isn't a failure, and a clean result isn't proof that your schema was followed. Validation is what tells you.

Two things are clear from the table. Inlining the references removes the $defs and $ref warnings, and title and default don't seem to matter, because dropping them didn't change the warnings. The anyOf is what's left. This helper flattens the references, and refuses a schema that nests deeper than Spidra's limit of 5 levels or refers to itself:

def inline_refs(schema):
    """Replace every $ref with the definition it points to, and drop $defs."""
    return _walk(schema, schema.get("$defs", {}), 0)

def _walk(node, defs, depth):
    if isinstance(node, list):
        return [_walk(item, defs, depth) for item in node]
    if not isinstance(node, dict):
        return node
    if "$ref" in node:
        if depth >= 5:
            raise ValueError("the schema nests deeper than 5 levels, or refers to itself")
        return _walk(defs[node["$ref"].split("/")[-1]], defs, depth + 1)
    return {key: _walk(value, defs, depth) for key, value in node.items() if key != "$defs"}

And this one rewrites Pydantic's nullable form into the type list the Spidra docs show:

def nullable_types(schema):
    """Rewrite anyOf-with-null as the type list Spidra documents, ["string", "null"]."""
    if isinstance(schema, list):
        return [nullable_types(item) for item in schema]
    if not isinstance(schema, dict):
        return schema
    options = schema.get("anyOf", [])
    others = [o for o in options if o != {"type": "null"}]
    if len(options) == 2 and len(others) == 1 and isinstance(others[0].get("type"), str):
        rest = {key: value for key, value in schema.items() if key != "anyOf"}
        return nullable_types({**rest, **others[0], "type": [others[0]["type"], "null"]})
    return {key: nullable_types(value) for key, value in schema.items()}

for_spidra runs both, in the order that works, because an anyOf around a $ref can only be rewritten once the reference is inlined:

def for_spidra(schema):
    """Flatten $ref first, then rewrite nullable fields, so Spidra has nothing to ignore."""
    return nullable_types(inline_refs(schema))

def demo_for_spidra():
    schema = for_spidra(Book.model_json_schema())
    print(json.dumps(schema["properties"]["isbn"]))
    print("anyOf anywhere:", "anyOf" in json.dumps(schema))

Output:

{"default": null, "title": "Isbn", "type": ["string", "null"]}
anyOf anywhere: False

The nullable field is now ["string", "null"], and no anyOf is left anywhere in the schema. Send for_spidra(Book.model_json_schema()) instead of the raw schema. We checked this rewrite against the docs, not against a live job, so read schema_warnings the first time you use it.

Check Spidra's own signals before you validate

A scrape result can be unusable before Pydantic sees it. ai_extraction_failed means the AI step failed and content fell back to raw markdown, which is a string, not an object. extraction_empty is the other flag on the result. Either would reach model_validate as the wrong kind of input, so check them first, along with the type of the content:

def check_signals(result):
    """The reason a scrape result is not worth validating, or None."""
    if result.ai_extraction_failed:
        return "ai_extraction_failed: content fell back to raw markdown"
    if result.extraction_empty:
        return "extraction_empty: nothing was extracted"
    if not isinstance(result.content, dict):
        return f"content is a {type(result.content).__name__}, not an object"
    return None

These signals catch an extraction that failed. They don't catch one that went wrong, as the last live result below shows. In our run, neither flag was set on any of the eight jobs that completed. A ninth, for a book page that doesn't exist, never returned a result at all. The job failed with this message:

scrape failed: Failed to scrape https://books.toscrape.com/catalogue/this-book-does-not-exist_0/index.html: The page was not delivered: the page answered HTTP 404. Nothing was charged. Check the URL.

The routine below records that as a rejection at the scrape stage. The job cost nothing.

A valid result, a missing optional field, and a wrong type

Here are the three cases, on hand-written dicts shaped like Spidra's results, so you can run them without an account:

BASE = {
    "title": "A Light in the Attic",
    "price": 51.77,
    "in_stock": True,
    "upc": "a897fe39b1053632",
}

def demo_trio():
    cases = {
        "valid": {**BASE, "rating": 3},
        "missing optional": BASE,
        "wrong type": {**BASE, "price": "£51.77"},
    }
    for name, data in cases.items():
        try:
            print(f"{name}: {Book.model_validate(data)}")
        except ValidationError as error:
            print(f"{name}: {error.error_count()} error")
            for e in error.errors(include_url=False):
                print("  ", e["type"], e["loc"], e["msg"], repr(e["input"]))

Output:

valid: title='A Light in the Attic' price=51.77 in_stock=True upc='a897fe39b1053632' rating=3 isbn=None
missing optional: title='A Light in the Attic' price=51.77 in_stock=True upc='a897fe39b1053632' rating=None isbn=None
wrong type: 1 error
   float_parsing ('price',) Input should be a valid number, unable to parse string as a number '£51.77'

The valid result passes. The one with isbn and rating left out passes too, because both default to None. And the string "£51.77" where a float was expected fails with float_parsing, naming the field and quoting the input.

A ValidationError can be read three ways. str(error) is for people, errors() gives you dicts you can store, and json() gives you a string. All three on the same failure:

def demo_error_views():
    try:
        Book.model_validate({**BASE, "price": "£51.77"})
    except ValidationError as error:
        print(error)
        print(error.errors(include_url=False))
        print(error.json(include_url=False))

Output:

1 validation error for Book
price
  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='£51.77', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/float_parsing
[{'type': 'float_parsing', 'loc': ('price',), 'msg': 'Input should be a valid number, unable to parse string as a number', 'input': '£51.77'}]
[{"type":"float_parsing","loc":["price"],"msg":"Input should be a valid number, unable to parse string as a number","input":"£51.77"}]

The text ends with a link that contains the Pydantic version (2.10 here), so it changes when you upgrade. Pass include_url=False when you store errors, and your logs stay the same.

Other wrong types fail in the same way, each with its own error type:

def demo_wrong_types():
    bad = {"price": "Free", "in_stock": "In stock", "title": None, "upc": 12345}
    for field, value in bad.items():
        try:
            Book.model_validate({**BASE, field: value})
        except ValidationError as error:
            e = error.errors(include_url=False)[0]
            print(f"{field:9s} {value!r:11} {e['type']}: {e['msg']}")

Output:

price     'Free'      float_parsing: Input should be a valid number, unable to parse string as a number
in_stock  'In stock'  bool_parsing: Input should be a valid boolean, unable to interpret input
title     None        string_type: Input should be a valid string
upc       12345       string_type: Input should be a valid string

What the real pages did to the model

Hand-written dicts show the mechanics. These are real results. We sent Book.model_json_schema() to four book pages. The first, A Light in the Attic, came back like this:

{
  "title": "A Light in the Attic",
  "price": 51.77,
  "in_stock": true,
  "upc": "a897fe39b1053632",
  "rating": null,
  "isbn": null
}

And it validated:

title='A Light in the Attic' price=51.77 in_stock=True upc='a897fe39b1053632' rating=None isbn=None

Two things stand out. The price is a number. The page shows £51.77, but the schema said number, so Spidra converted it, and the float_parsing case from the demo didn't happen. If you need the original text, including the currency symbol, ask for a string and normalize it yourself. Our post on normalizing prices, currencies and units in scraped data covers that.

The other thing is that isbn and rating are both null, on all four book pages. The pages have no ISBN, and the star rating is a CSS class, not text, so the extractor can't see it. Both fields are optional, so both validated.

The live wrong type came from the rating. We made it required with a model that changes one line:

class RatedBook(Book):
    """The same book with the rating required, so a null rating is an error."""

    rating: int

All five results that came back failed the same way. This is the first:

{"url": "a-light-in-the-attic_1000", "stage": "validation", "problems": [{"field": "rating", "type": "int_type", "msg": "Input should be a valid integer", "input": "None"}]}

That's an int_type error from a real result: a required integer, and the input was None.

A valid answer to the wrong question

The fifth page we tried was example.com, which has no book on it. We sent the same book schema. This is what came back:

{
  "title": "Example Domain",
  "price": 0,
  "in_stock": true,
  "upc": "N/A",
  "rating": null,
  "isbn": null
}

Both flags were False. There's no price, no UPC and no stock status on that page, and the extractor filled every required field anyway: a price of 0, a UPC of "N/A", and in_stock as true. The Book model accepted all of it, because those values have the right types.

Types can't catch this. Limits can. The UPCs on our four book pages were each exactly 16 hexadecimal characters, and a real price is above zero. A second model adds both checks, plus a limit on the rating:

class CheckedBook(Book):
    """The same fields plus checks that catch a plausible but wrong answer. Validate with this."""

    price: float = Field(gt=0)
    upc: str = Field(pattern=r"^[0-9a-f]{16}$")
    rating: Optional[int] = Field(None, ge=1, le=5)

We keep Book as the schema we send, and use CheckedBook to validate. We didn't test whether Spidra honours keywords like pattern, so the stricter rules stay on our side. We saved the real example.com result as a constant, so the check runs without an account:

EXAMPLE_COM = {
    "title": "Example Domain",
    "price": 0,
    "in_stock": True,
    "upc": "N/A",
    "rating": None,
    "isbn": None,
}

def demo_checked():
    print("Book:       ", Book.model_validate(EXAMPLE_COM))
    try:
        CheckedBook.model_validate(EXAMPLE_COM)
    except ValidationError as error:
        for e in error.errors(include_url=False):
            print(
                "CheckedBook:", e["loc"][0], e["type"], "|", e["msg"], "| input", repr(e["input"])
            )

Output:

Book:        title='Example Domain' price=0.0 in_stock=True upc='N/A' rating=None isbn=None
CheckedBook: price greater_than | Input should be greater than 0 | input 0
CheckedBook: upc string_pattern_mismatch | String should match pattern '^[0-9a-f]{16}$' | input 'N/A'

Book accepts it, and quietly turns the 0 into 0.0. CheckedBook rejects both invented values and says which. Here's the batch routine, from the next section, run over all six saved results:

Book: accepted 5, rejected 1
CheckedBook: accepted 4, rejected 2
{"url": "example.com", "stage": "validation", "problems": [{"field": "price", "type": "greater_than", "msg": "Input should be greater than 0", "input": "0"}, {"field": "upc", "type": "string_pattern_mismatch", "msg": "String should match pattern '^[0-9a-f]{16}$'", "input": "'N/A'"}]}
{"url": "this-book-does-not-exist_0", "stage": "scrape", "problems": [{"field": "(scrape)", "type": "scrape_failed", "msg": "RuntimeError: scrape failed: Failed to scrape https://books.toscrape.com/catalogue/this-book-does-not-exist_0/index.html: The page was not delivered: the page answered HTTP 404. Nothing was charged. Check the URL.", "input": ""}]}
null rates: {'title': '0%', 'price': '0%', 'in_stock': '0%', 'upc': '0%', 'rating': '100%', 'isbn': '100%'}

Under Book, five of six passed, including the wrong page. Under CheckedBook, four pass. The two rejections are the wrong page, at the validation stage, and the page that doesn't exist, at the scrape stage.

Three ways to say "might be missing"

Pydantic gives you three ways to model a field that may not be there, and they aren't the same. Two of them differ only by a = None:

class Tolerant(BaseModel):
    isbn: Optional[str] = None


class PresentOrNull(BaseModel):
    isbn: Optional[str]

def demo_nullable():
    for model in (Tolerant, PresentOrNull):
        print(model.__name__, "required:", model.model_json_schema().get("required"))
        for label, data in (("absent", {}), ("null", {"isbn": None})):
            try:
                print(f"  {label}: {model.model_validate(data)}")
            except ValidationError as e:
                print(f"  {label}: {e.errors(include_url=False)[0]['type']}")

Output:

Tolerant required: None
  absent: isbn=None
  null: isbn=None
PresentOrNull required: ['isbn']
  absent: missing
  null: isbn=None

Spidra's schema docs say nullable fields come back as null when the page doesn't have them, and our run agrees: both optional fields came back as explicit null, on every page.

So a field written as Optional[str] with no default says "the key must be there, and its value may be null". That catches a result that silently dropped the key. With = None, a dropped key looks the same as a missing value. Use the strict form for fields you depend on, and the tolerant one for fields that are truly optional.

What lax mode lets through

By default, Pydantic coerces. That's convenient for most data and risky for scraped data, because an extractor can hand back a value of the wrong type, and Pydantic will quietly fix it:

def demo_lax():
    cases = (
        ("price", "51.77"),
        ("in_stock", "no"),
        ("in_stock", "yes"),
        ("rating", 3.0),
        ("rating", "3"),
        ("rating", 3.5),
    )
    for field, value in cases:
        try:
            book = Book.model_validate({**BASE, field: value})
            print(f"{field}={value!r:8} -> {getattr(book, field)!r}")
        except ValidationError as error:
            print(f"{field}={value!r:8} -> {error.errors(include_url=False)[0]['type']}")

Output:

price='51.77'  -> 51.77
in_stock='no'     -> False
in_stock='yes'    -> True
rating=3.0      -> 3
rating='3'      -> 3
rating=3.5      -> int_from_float

The numeric string, "yes" and "no", 3.0 and "3" are all accepted and converted. Only a fractional rating fails. The 0 that Spidra returned for example.com was converted the same way, into 0.0. If a silent conversion would hide a real problem, ask for strictness, and also for no extra keys, since an extractor can invent them:

def demo_strict_and_extra():
    try:
        Book.model_validate({**BASE, "price": "51.77"}, strict=True)
    except ValidationError as error:
        print("strict:", error.errors(include_url=False)[0]["type"])

    class NoExtras(Book):
        model_config = ConfigDict(extra="forbid")

    try:
        NoExtras.model_validate({**BASE, "translations": {"fr": "x"}})
    except ValidationError as error:
        print("extra=forbid:", error.errors(include_url=False)[0]["type"])

Output:

strict: float_type
extra=forbid: extra_forbidden

Strict mode rejects the numeric string (float_type), and extra="forbid" rejects a key the model never asked for. You can turn strict on per call, per model, or for a single field with Field(strict=True).

A routine that sorts results into accepted and rejected

Validation raises an exception. A pipeline needs a decision it can act on: keep this, set that aside, and remember why. The routine returns one of two small objects:

@dataclass
class Accepted:
    item: object  # a validated model instance
    url: str
    job_id: "str | None"


@dataclass
class Rejected:
    url: str
    job_id: "str | None"
    stage: str  # "scrape", "signals" or "validation"
    problems: list  # each: field, type, msg, input

An error converts to plain dicts that name the field, the kind of problem and the input, shortened, because a missing error quotes the whole dict it was checking:

def summarize(error, limit=80):
    """A ValidationError as plain dicts: the field, the kind of problem, the input shortened."""
    return [
        {
            "field": ".".join(map(str, e["loc"])) or "(whole result)",
            "type": e["type"],
            "msg": e["msg"],
            "input": repr(e["input"])[:limit],
        }
        for e in error.errors(include_url=False)
    ]

Then one function runs the signal check, then the model, and returns a result or a rejection with a stage that says where it stopped:

def validate_result(result, model, url, job_id=None):
    reason = check_signals(result)
    if reason:
        problem = {"field": "(whole result)", "type": "signal", "msg": reason, "input": ""}
        return Rejected(url, job_id, "signals", [problem])
    try:
        return Accepted(model.model_validate(result.content), url, job_id)
    except ValidationError as error:
        return Rejected(url, job_id, "validation", summarize(error))

A batch is that function in a loop, with scrape failures turned into rejections too:

def scrape_failure(item):
    problem = {"field": "(scrape)", "type": "scrape_failed", "msg": item["error"], "input": ""}
    return Rejected(item["url"], item.get("job_id"), "scrape", [problem])


def validate_batch(items, model):
    """items: dicts with url, job_id, and either result or error. -> (accepted, rejected)"""
    accepted, rejected = [], []
    for item in items:
        if item.get("error"):
            rejected.append(scrape_failure(item))
            continue
        outcome = validate_result(item["result"], model, item["url"], item.get("job_id"))
        (accepted if isinstance(outcome, Accepted) else rejected).append(outcome)
    return accepted, rejected

Here are the four kinds of outcome, with CheckedBook as the model, on results made up to match each one. fake builds a stand-in for a Spidra result:

def fake(content, failed=False, empty=False):
    return SimpleNamespace(content=content, ai_extraction_failed=failed, extraction_empty=empty)

def demo_routine():
    items = [
        {"url": "u/valid", "job_id": "j1", "result": fake({**BASE, "rating": 3})},
        {"url": "u/wrong-type", "job_id": "j2", "result": fake({**BASE, "price": "£51.77"})},
        {"url": "u/markdown", "job_id": "j3", "result": fake("# Example Domain", failed=True)},
        {"url": "u/missing", "job_id": None, "error": "scrape failed: the page returned 404"},
    ]
    accepted, rejected = validate_batch(items, CheckedBook)
    print(f"accepted {len(accepted)}, rejected {len(rejected)}")
    for r in rejected:
        print(
            json.dumps({"url": r.url, "stage": r.stage, "problems": r.problems}, ensure_ascii=False)
        )

Output:

accepted 1, rejected 3
{"url": "u/wrong-type", "stage": "validation", "problems": [{"field": "price", "type": "float_parsing", "msg": "Input should be a valid number, unable to parse string as a number", "input": "'£51.77'"}]}
{"url": "u/markdown", "stage": "signals", "problems": [{"field": "(whole result)", "type": "signal", "msg": "ai_extraction_failed: content fell back to raw markdown", "input": ""}]}
{"url": "u/missing", "stage": "scrape", "problems": [{"field": "(scrape)", "type": "scrape_failed", "msg": "scrape failed: the page returned 404", "input": ""}]}

Validation can't tell you what the extractor couldn't see

Validation can't tell a field that's sometimes empty from one that's always empty. Optional[int] accepts None on the first page and on the thousandth, so if the extractor can't see a field, every result still passes. The fix is to measure it:

def null_rates(accepted, fields):
    """Validation passes a null in an optional field forever. This shows how often it happens."""
    total = len(accepted) or 1
    return {name: sum(getattr(a.item, name) is None for a in accepted) / total for name in fields}

In our run, rating and isbn were None in 100% of the accepted results, and the other four fields in 0% (the last line of the output above). For isbn, that's right: no page has one. For rating it's the problem. The pages do show a rating, as stars, and the extractor can't read it, so every result is missing a field the page has. To validation, the two look identical.

A field that's null on every page is a mismatch between your model and what the extractor can see, not a fact about the data. Pick a threshold for the fields you expect, and fail the run when a rate crosses it.

From validated objects to your pipeline

The accepted objects are ordinary typed values. In an ETL job, write item.model_dump() to your store and the rejections to a JSONL file for review. In a typed API, return them from a function whose signature says list[Book]. In an analytics job, pd.DataFrame([a.item.model_dump() for a in accepted]) gives you a table.

Review the rejections grouped by stage and error type: a run of float_parsing on price means the page changed, a run of signals means the extraction failed, and a run of greater_than or string_pattern_mismatch means the extractor is guessing.