How to normalize prices, currencies, and units in scraped data

Here are two prices: 1.234 and 1,234. In Germany, the first one is one thousand two hundred and thirty-four. In the US, it's a little more than one. A scraper sees the same characters either way, and so does the library you hand them to.

That's the problem behind every attempt to normalize scraped prices, currencies and units. A catalog built from a dozen countries holds $, £ and ¥ that each stand for several currencies, decimals written as commas and as points, thousands separated by dots, spaces, apostrophes and invisible characters, and sizes like 1 gal that mean different amounts in different places. Most pipelines guess. A wrong guess doesn't fail. It quietly puts a price that's off by a factor of a thousand into your database.

This post builds a pipeline that doesn't guess. Spidra extracts the text exactly as the page shows it. Your code turns that text into an amount, an ISO currency and a unit, using the locale and currency you know about the source. Anything that could mean two things is rejected, with a reason. We tested it on a live scrape of 20 products, a 54-row multi-country catalog, 154 test cases and 100,000 random strings, each tried five ways. It does not convert currencies. Exchange rates are a different job, and the last section says what that job needs.

Keep the text, take the locale from the source, reject the rest

Three rules shape everything that follows. Keep the raw text next to every normalized value, so you can always check the work. Take the locale, currency and unit system from the source you configured, never from the string itself, because the string can't tell you. And if a string can mean two things and the source can't settle it, reject it.

The output is one JSON record per product, or one rejection with a reason. Here's a fixture row from a French grocery source, before and after. (It's a fixture we wrote, not a scraped page, and its provenance says so.)

{"title": "Eau minérale", "price_text": "4,50 €", "size_text": "6 x 1,5 L"}

{
  "title": "Eau minérale",
  "url": null,
  "price": {
    "raw": "4,50 €",
    "amount": "4.50",
    "currency": "EUR",
    "minor_units": 450
  },
  "quantity": {
    "raw": "6 x 1,5 L",
    "dimension": "volume",
    "value": "9000",
    "unit": "ml",
    "count": 6
  },
  "unit_price": {
    "amount": "0.5000",
    "currency": "EUR",
    "per": "1 l"
  },
  "provenance": {
    "source": "fixture-fr",
    "page_url": "fixture://fr/epicerie",
    "fetched_at": null,
    "extractor": "fixture",
    "job_id": null,
    "locale": "fr_FR",
    "source_currency": "EUR",
    "rules": {
      "price": "symbol € belongs to one currency; locale fr_FR: decimal ',', group ' '",
      "quantity": "l = 1000 ml; locale fr_FR: decimal ',', group ' '",
      "unit_price": "4.50 EUR / 9000 ml x 1000, rounded half up"
    }
  }
}

raw keeps what the page said. amount is a string holding a Decimal, never a float, and minor_units is the same amount in the currency's smallest unit as an integer. The rules explain which rule produced each value, in words a person can check.

Extract the text exactly as the page shows it

Spidra's scrape can return JSON that matches a schema. The temptation is to ask for numbers: price: number. Don't. When the extractor converts, you lose the original text, and you can't tell a good conversion from a bad one. Ask for the text (our post on nested JSON from product listings goes deeper on schemas):

BOOK = {
    "type": "object",
    "required": ["title", "price_text"],
    "properties": {
        "title": {"type": "string"},
        "price_text": {
            "type": "string",
            "description": "The price exactly as shown, with its currency symbol",
        },
        "availability_text": {"type": ["string", "null"]},
        "url": {"type": ["string", "null"]},
    },
}

A schema's root has to be an object, so the books go in an array property:

LISTING_SCHEMA = {
    "type": "object",
    "required": ["books"],
    "properties": {"books": {"type": "array", "items": BOOK}},
}

LISTING_PROMPT = (
    "Extract every book on the page: its title, its price text, its availability text and "
    "the link to its page. Copy each value exactly as it appears on the page. "
    "Do not convert, round, translate or reformat anything."
)

The prompt tells it not to convert, round or reformat. The scrape then runs as a job, and we poll it, because that gives us the job ID to put on every record:

def extract(client, url, prompt, schema, timeout=180):
    """Run one scrape and wait for it. Returns the job id too, so records can name their job."""
    queued = client.start_scrape(
        ScrapeParams(urls=[ScrapeUrl(url=url)], prompt=prompt, schema=schema)
    )
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        job = client.get_scrape(queued.job_id)
        if job.status == "completed":
            return queued.job_id, job.result
        if job.status == "failed":
            raise RuntimeError(f"scrape failed: {job.error}")
        time.sleep(3)
    raise TimeoutError(f"scrape {queued.job_id} did not finish in {timeout} s")

In the Python SDK, a scrape that takes a schema takes ScrapeParams and ScrapeUrl objects. On 30 October 2026 we ran it on the first page of books.toscrape.com, a practice site built for exactly this. One scrape job returned all 20 books in 13.9 seconds of Spidra time, using 3,746 tokens, with no warnings and no failed extraction. These are the first three rows, exactly as they came back:

{"title": "A Light in the Attic", "price_text": "£51.77", "availability_text": "In stock", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}
{"title": "Tipping the Velvet", "price_text": "£53.74", "availability_text": "In stock", "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"}
{"title": "Soumission", "price_text": "£50.10", "availability_text": "In stock", "url": "https://books.toscrape.com/catalogue/soumission_998/index.html"}

Every one of the 20 price_text values had the same shape: a £ and two decimals, as text. Nothing came back converted into a number. That's a check on shape, not proof that every digit matches the page, so spot-check a sample against the live page the first time you point this at a new source.

If a site shows different prices or currencies depending on where you visit from, Spidra's proxyCountry option fetches the page from a chosen country (it takes a two-letter code for 50 countries). We didn't need it here. If you use it, record the country next to the locale.

The same string means different things

Before writing any parser, it's worth seeing how much ambiguity is out there. The Unicode CLDR data that Babel ships (we used Babel 2.17.0) answers it. A symbol can belong to many currencies: $ is used by 27 currencies, £ by 5, kr by 4 and ¥ by 2. In that data, 14 of 278 symbols are shared like that.

Numbers are ambiguous in a narrower way. We wrote 8 amounts the way 31 locales write them, which gave 24 distinct strings. Exactly 4 of them mean more than one value depending on the locale:

They share one shape: a single separator followed by exactly three digits. Every other string in the set has only one reading. That's a precise thing to reject, and the pipeline does.

Two more facts shape the code. The thousands separator isn't always a space or a comma: French writes it as U+202F (a narrow no-break space), Swedish, Russian and Polish as U+00A0, and Swiss German as a curly apostrophe. And currencies differ in how many decimals they have: 43 have none (JPY, KRW), 256 have two, and 6 have three (BHD, JOD, KWD, LYD, OMR, TND). The number of decimals turns out to be a useful cross-check, as the last section shows.

Settle the currency first, or reject the price

The currency comes from one of four places, in this order: an ISO code in the text (CHF, USD), a symbol that belongs to only one currency (€, ₹, zł), the source's configured currency when the symbol is shared, or an explicit symbol policy you wrote down for the source. If none of them settles it, the price is rejected. The code below is excerpted from our normalize.py. It leans on that module's imports and helpers, so it won't run pasted on its own.

def resolve_currency(token, hint, policy=None):
    symbols, codes = currency_tables()
    if token in codes:
        return token, f"ISO code {token} in the text"
    found = symbols.get(token, set())
    if not found:
        raise Rejected("UNRECOGNISED_TEXT", f"{token!r} is not a currency")
    if len(found) == 1:
        return next(iter(found)), f"symbol {token} belongs to one currency"
    return _shared_symbol(token, found, hint, policy)

def _shared_symbol(token, found, hint, policy):
    """Several currencies use this symbol: only the source can say which one."""
    shared = f"symbol {token} is shared by {len(found)} currencies"
    if hint in found:
        return hint, f"{shared}; the source says {hint}"
    chosen = (policy or {}).get(token)
    if chosen in found:
        return chosen, f"{shared}; the source policy maps it to {chosen}"
    if hint or chosen:
        raise Rejected("CURRENCY_CONFLICT", f"{token} is not used for {hint or chosen}")
    raise Rejected("AMBIGUOUS_SYMBOL", f"{shared} and the source gave no currency")

Here's what that does with a few strings. The source's currency and locale come from the source configuration, not from the text:

A marketplace that sells in several currencies can't name one. It can write a policy for the symbols it's willing to settle, such as {"$": "USD", "£": "GBP"}, and leave the rest ambiguous on purpose. In our catalog that policy resolves $999 to 999.00 USD and £199 to 199.00 GBP, and still rejects ¥54,000, because a yen sign could be JPY or CNY and the policy didn't say. The policy lives in the source's configuration, where someone can review it, instead of inside a function.

Parse the number with the locale you know

Text first needs cleaning. Scraped prices carry no-break spaces, narrow no-break spaces, curly apostrophes, and invisible marks: the right-to-left mark that Arabic pages add, zero-width spaces, a byte-order mark at the start. All of them have to go before anything is parsed:

def clean(text):
    """Plain spaces and apostrophes, and no invisible marks (bidi, zero-width, BOM) in the way."""
    text = unicodedata.normalize("NFKC", text)
    text = "".join(ch for ch in text if unicodedata.category(ch) != "Cf")
    text = text.replace("\u2019", "'").replace("\u02bc", "'").replace("\u2018", "'")
    return re.sub(r"\s+", " ", text).strip()

With a locale, the number has to fit that locale's grammar exactly. The decimal and group characters come from CLDR, and anything that doesn't fit is rejected:

def parse_number(text, locale=None, exponent=None):
    """-> (value, decimals, rule). Strict with a locale; unambiguous text only without one."""
    if not locale:
        return _infer(text, exponent)
    decimal, group, indian = separators(locale)
    if not any(re.fullmatch(form, text) for form in _forms(decimal, group, indian)):
        detail = f"{text!r} is not valid in {locale} (decimal {decimal!r}, group {group!r})"
        raise Rejected("BAD_NUMBER_FORMAT", detail)
    value, decimals = _digits(text, decimal, group)
    return value, decimals, f"locale {locale}: decimal {decimal!r}, group {group!r}"

So 1.234,56 € is 1234.56 EUR under de_DE, while 1.234,56 under en_US is a rejected BAD_NUMBER_FORMAT. And a currency's decimals check the result: 1.234 under en_US for USD parses as 1.234, which has three decimals, and a dollar doesn't, so it's rejected too.

Without a locale, only text with a single possible reading gets through. This function decides what a lone separator means:

def _lone_mark(text, mark, exponent):
    """One '.' or ',' and nothing else: a decimal or a thousands group? Few shapes can tell."""
    whole, frac = text.split(mark)
    if len(frac) != 3 or len(whole) > 3 or whole == "0":  # cannot be a thousands group
        why = f"no locale: one {mark!r} followed by {len(frac)} digits is a decimal"
        return (*_digits(text, mark), why)
    if exponent == 0:  # a currency with no decimals can only mean grouping
        why = f"no locale: the currency has no decimals, so {mark!r} groups thousands"
        return _digits(text, None, mark)[0], 0, why
    detail = f"{text!r}: one {mark!r} and three digits could be a decimal or a thousands group"
    raise Rejected("AMBIGUOUS_SEPARATOR", detail)

A lone mark with three digits after it is ambiguous, and rejected, unless something settles it. A leading 0 settles it (0,999 can't be a thousands group, because a group doesn't start with zero), and so does a currency with no decimals (¥1,234 in yen can only be a group).

Some strings aren't a number at all. A catalog is full of them:

Free is rejected on purpose. A free product is a decision for you to make explicitly, not something for a parser to infer from a missing number.

What price-parser and pint do with the same strings

You may already be using price-parser, which is built for "parsing prices extracted from web pages", or pint for units. Both are good at what they do, and both are designed to produce an answer. That's the difference. We ran the hard strings through both (price-parser 0.5.1 and pint 0.26.2, as of October 2026):

price-parser's hints are real: given the exact separator characters, it reads CHF 1’299.00 and 0,999 € correctly. (It has to be the curly apostrophe; the ASCII one still gives 1.) What no hint can do is say "this is a range" or "this is two prices", because it has no way to refuse. For units:

1 fl oz becomes a femtoliter times an ounce, a unit that doesn't exist. 1,5 L becomes 15 litres. And ton and gal quietly take their US meanings. That's a defensible default for a general-purpose library. It's a bad one for a catalog.

Reject units that can mean two things

The same words mean different amounts in different countries. From the unit table the pipeline uses:

So gallon, pint, quart, fl oz and ton need the source's unit system (us, uk or metric), and a bare oz needs to be told whether it's a weight or a fluid ounce. Without that, they're rejected:

def _by_system(word, system):
    """Words like gallon, pint and ton mean different amounts in different countries."""
    sizes = BY_SYSTEM[word]
    if system not in sizes:
        raise Rejected(
            "AMBIGUOUS_UNIT", f"{word} means different amounts in {sorted(sizes)}; give the system"
        )
    dimension = BY_SYSTEM_DIMENSION.get(word, "volume")
    return (dimension, "g" if word == "ton" else "ml", sizes[system]), f"{word} ({system})"

Units are parsed into grams, millilitres or a count, as exact Decimal values. A pack like 6 x 330 ml becomes 1,980 ml in 6 items, and pack of 12 is a count of 12.

Unit prices follow from that, and they stay inside a dimension. A price per litre needs a volume, and a price per 100 g needs a mass. We never convert one into the other, because that would need a density we don't have:

def unit_price(money, quantity, per):
    """Price per 100 g, 1 kg, 1 l, 100 ml or 1 item. Mass and volume never convert to each other."""
    dimension, base, size = PER[per]
    if quantity.dimension != dimension:
        raise Rejected("DIMENSION_MISMATCH", f"cannot price a {quantity.dimension} per {per}")
    places = Decimal(1).scaleb(-(get_currency_precision(money.currency) + 2))
    amount = (money.amount * size / quantity.value).quantize(places, rounding=ROUND_HALF_UP)
    return UnitPrice(
        amount,
        money.currency,
        per,
        f"{money.amount} {money.currency} / {quantity.value} {base} x {size}, rounded half up",
    )

The amount is rounded half up to two places more than the currency has, and the rule that produced it is recorded.

Every record keeps its raw text and its source, or says why it was rejected

The source's configuration is a small dataclass. Everything the text can't tell you lives here:

@dataclass(frozen=True)
class Source:
    name: str
    url: str
    locale: "str | None"  # how this source writes numbers, from you, never guessed from the text
    currency: "str | None"  # the currency this source sells in
    system: "str | None" = None  # "us", "uk" or "metric": which gallon, pint or ton
    dimension: "str | None" = None  # "mass" or "volume", if the size field is always one of them
    max_decimals: "int | None" = (
        None  # allow more decimals than the currency has (fuel prices), on purpose
    )
    symbols: "dict | None" = (
        None  # a policy for shared symbols, e.g. {"$": "USD"}, written down and reviewable
    )

A row either becomes a record or a rejection. No valid price means no record, and a size that can't be read only loses the size:

def normalize_row(row, source, meta):
    """-> (record or None, rejections). No valid price, no record. A bad size loses the size."""
    try:
        price = parse_price(
            row["price_text"], source.locale, source.currency, source.max_decimals, source.symbols
        )
    except Rejected as error:
        return None, [rejection(row, source, "price_text", error, meta)]
    quantity, per, rejects = parse_size(row, source, price, meta)
    return build_record(row, source, meta, price, quantity, per), rejects

Here's the real thing. This is the first of the 20 rows Spidra returned, and the record that came out the other side:

{"title": "A Light in the Attic", "price_text": "£51.77", "size_text": null, "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}

{
  "title": "A Light in the Attic",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "price": {
    "raw": "£51.77",
    "amount": "51.77",
    "currency": "GBP",
    "minor_units": 5177
  },
  "quantity": null,
  "unit_price": null,
  "provenance": {
    "source": "books.toscrape.com",
    "page_url": "https://books.toscrape.com/",
    "fetched_at": "2026-10-30T13:51:22+00:00",
    "extractor": "spidra scrape",
    "job_id": "40ff81ff-25cf-490b-82e0-8ca4269f11ce",
    "locale": "en_GB",
    "source_currency": "GBP",
    "rules": {
      "price": "symbol £ is shared by 5 currencies; the source says GBP; locale en_GB: decimal '.', group ','",
      "quantity": null,
      "unit_price": null
    }
  }
}

All 20 became records, and none was rejected. That's the easy case: one currency, a dot for the decimal and no sizes, which is why the hard cases come from the fixture catalog below. The provenance names the job, the page, the time and the rule that settled the currency, so any of the 20 records can be traced back to the scrape that produced it.

The detail pages show a price three ways, and they have to add up. We check that the price excluding tax plus the tax equals the price including it, and reject the product if it doesn't:

We checked three. This is what the scrape returned for each, and then the check:

{"title": "A Light in the Attic", "price_excl_tax_text": "£51.77", "tax_text": "£0.00", "price_incl_tax_text": "£51.77", "availability_text": "In stock (22 available)"}
{"title": "Tipping the Velvet", "price_excl_tax_text": "£53.74", "tax_text": "£0.00", "price_incl_tax_text": "£53.74", "availability_text": "In stock (20 available)"}
{"title": "Soumission", "price_excl_tax_text": "£50.10", "tax_text": "£0.00", "price_incl_tax_text": "£50.10", "availability_text": "In stock (20 available)"}

tax check passed: £51.77 + £0.00 = £51.77 -> 51.77 GBP
tax check passed: £53.74 + £0.00 = £53.74 -> 53.74 GBP
tax check passed: £50.10 + £0.00 = £50.10 -> 50.10 GBP

All three added up. The tax was £0.00 on each, so it's an easy test, but the check is there for the day it isn't.

When a row is rejected, the reason travels with it:

{
  "field": "price_text",
  "raw": "£10 - £20",
  "code": "RANGE",
  "detail": "['10', '20'] in '£10 - £20'",
  "title": "Tea bags",
  "source": "fixture-uk",
  "page_url": "fixture://uk/groceries",
  "job_id": null
}

The tests that earn the trust

There are 154 tests, and almost all of them are tables of text with the expected answer written down first:

The round trip is the strongest one. For 31 locales, it writes amounts the way CLDR says that locale writes them (in the locale's own currency, using Babel), then parses them back and demands the same amount. That includes Arabic text wrapped in invisible right-to-left marks.

The multi-country catalog is the other half. It's 54 rows from 13 sources: German, French, Swiss, British, American, Indian, Japanese, Swedish, Brazilian, Polish and Kuwaiti retailers, a fuel station and a marketplace. 40 became records, and 14 rows were rejected on price. Three more lost only their size. Here is every rejection:

Then we attacked it. The fuzz test builds 100,000 random strings from digits, separators, symbols and words, and parses each under five different configurations, 500,000 attempts in all. 56,192 were accepted, 443,808 were rejected, none came out inconsistent (negative, in exponent notation, or with minor units that disagreed with the amount), and nothing raised an exception except the rejection we designed.

That's the finished version. The first version had bugs, and the tests found these six before any real page did:

• Decimal.normalize() turned a 500 g weight into 5E+2, which is what would have landed in the JSON.
• Right-to-left marks and zero-width spaces, invisible to a person, blocked parsing entirely.
• Arabic decimal and thousands separators were rejected with a misleading reason.
• The fuzz test found a crash on 54 324,3 2, which passed the whole-number check and never checked the fraction.
• 4 pints was rejected as an unknown unit, because the table had pint and not its plural.
• 1.299,– €, with the dash before the currency sign, wasn't recognized as 1.299,00.

What the checks can't catch: a wrong locale

Everything above depends on the source's locale being right. So what happens if it isn't? We wrote amounts the way one locale does, read them as another, and counted, for every pair of 30 locales:

In our test, a wrong locale was never silently wrong for dollars, euros or yen, because the strict grammar and the currency's decimals catch it between them. For the dinar it was wrong 8% of the time, by a factor of 1,000: KWD12.500, written for en_US and read as de_DE, comes out as 12,500.000 instead of 12.500. The cross-check disappears when the currency has the same number of decimals as the three digits after the separator, and six currencies do (BHD, JOD, KWD, LYD, OMR, TND).

If you scrape in one of those currencies, keep a known price in each source's configuration as a canary, and check it on every run. And always keep the locale in the provenance, so when something looks wrong you can see what the parser was told.

What this pipeline leaves alone: exchange rates

A normalized record says 51.77 GBP. It says nothing about euros, and it shouldn't. Converting needs a rate, the date the rate applies to, and where it came from, and different sources give different rates for the same day. Treat conversion as a separate step. It reads normalized records and writes a new field beside the original, with the rate, its date and its source, and it never overwrites amount or currency. That keeps the clean record clean, and it keeps a bad rate from corrupting data you can't re-derive.

Try it on a page of your own

Create a free account at app.spidra.io and point extract() at a product listing you care about. Start with one page, and read the first twenty rows by eye. If you track prices over time, this is what makes the history trustworthy: every record names the scrape, the page and the time it came from, so a changed price can be traced to its source before anyone acts on it. Our whole run, one listing page and three detail pages, was four scrape jobs. It took 56 seconds and charged 11 credits. The JSON schema generator writes the schema for you, how credits work has the pricing rule, and the Python SDK docs cover the options we skipped.