Getting product data from different online stores is only the first part of building a catalog. The difficult part comes when you try to combine the results. One store identifies a product by its ASIN, another uses a Walmart item ID, and a third gives you an eBay listing number. Their titles, prices, sellers, and stock information may also differ.
If you group those records by title alone, it's easy to treat a bundle or a different color as the same item. You might also compare a refurbished offer with the price of a new product.
In this tutorial, we'll use Spidra and Python to collect product details from multiple websites, normalize them into one schema, and match the records we can verify. Listings without enough evidence will remain separate for review.
Start with products, variants, and offers
Before collecting anything, decide what one record in your catalog represents.
A product is the broader item, such as the Sony WH-1000XM5 headphones. A variant is a specific version, such as the black model. An offer is a retailer or seller listing for that variant at a particular price and condition.
Those distinctions matter because two stores may sell the same variant, while one store may have several sellers offering it. New and refurbished listings can belong to the same product, but they should not be combined when calculating a new-product price range.
Here is a simplified catalog record based on the Sony listings used throughout this guide:
{
"product": "Sony WH-1000XM5",
"variants": [
{
"gtin": "00027242923232",
"color": "Black",
"offers": [
{ "source": "walmart", "price": 198, "currency": "USD", "condition": "New" },
{ "source": "ebay", "price": 181.15, "currency": "USD", "condition": "New" }
]
}
]
}This is a normalized view, not a raw API response. Notice that the prices stay on the offers. If the Walmart price changes tomorrow, the product itself hasn't become a different product.
For larger catalogs, you'll also want a source-specific listing ID, seller, product URL, availability, capture time, and the original values you used for normalization. Don't replace a source's data with your cleaned version. Store both so you can investigate a questionable match later.
Discover listings, then open the product pages
Search pages help you find products, but they don't always contain the identifiers needed to match them reliably.
For example, the November 11 collection in the source research used the query noise cancelling headphones on Amazon, Walmart, and eBay. It returned 123 search listings in total, but none of those rows contained a GTIN or UPC. Product pages gave us more to work with.
The Walmart page for the black Sony WH-1000XM5 included these fields:
{
"usItemId": "386006068",
"title": "Sony WH-1000XM5 The Best Wireless Noise Canceling Headphones, Black",
"model": "WH1000XM5/B",
"upc": "027242923232",
"gtin": "00027242923232",
"price": 198,
"currency": "USD"
}The eBay item details for listing 800822312459 reported the same GTIN. That gives us a stronger reason to match the two offers than their similar titles.
Spidra has Templates for finding products through marketplace searches. Once you've selected the relevant results, read the individual product pages to retrieve identifiers and seller details. If you only need to compare a pair of known listings, our Amazon and Walmart price-comparison tutorial covers that smaller workflow.
Fetch product details with the Spidra Python SDK
You'll need a Spidra account and API key. Install the SDK, then set your key in an environment variable:
pip install spidra
export SPIDRA_API_KEY="spd_YOUR_API_KEY"For the three known Sony listings, use Batch scrape with JSON output:
import os
from spidra import Spidra
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
urls = [
"https://www.walmart.com/ip/386006068",
"https://www.ebay.com/itm/800822312459",
"https://www.amazon.com/dp/B09XS7JWHH",
]
batch = client.batch_scrape(urls, output="json")Spidra returns structured fields for these popular-site product URLs without a custom prompt. The completed batch contains an item for each URL. Check item.status before using its result, because a batch may finish even when individual URLs fail.
records = []
for item in batch.items:
if item.status != "completed" or not isinstance(item.result, dict):
print("Skipped:", item.url, item.error)
continue
records.append({"url": item.url, "data": item.result})This gives us the source records, not the unified catalog yet. Each site's result still uses its own field names. See the Batch scrape documentation for larger URL lists and retrying failed items.
Normalize the records without losing their meaning
The next step is to map the source fields into your own format. For example, Walmart uses usItemId, eBay uses itemId, and Amazon uses asin. All three are useful, but they are not interchangeable product identifiers.
The same applies to seller information, availability, and condition. Amazon's buy box can describe a particular seller's offer rather than the only available offer. eBay explicitly labels used and refurbished listings. Walmart may return multiple sellers for one product.
Keep source IDs and offer details together, and leave fields null when a page doesn't provide them. In the Sony collection, Amazon's buy box seller was listed as Amazon Resale. That seller name alone doesn't establish that the offer is new, so the normalized record should not set condition: "New" without supporting data.
Price and currency also need to stay together. A Nike Air Force 1 Mid '07 product page in the source research returned £109.99 and style code CW2289-111. Converting 109.99 into a plain number without retaining GBP would make later comparisons misleading.
For retailers without a ready-made structured result, Spidra lets you supply a JSON schema to request the fields your catalog needs. Use nullable fields for identifiers, stock status, and price. A discontinued Crutchfield listing in the research returned a product model but no price. That should remain null, not 0 and not a guessed current price.
Match by verified identifiers, not titles
GTINs are useful because the same identifier can follow a specific product variant across retailers. UPC-A codes can be normalized to the 14-digit representation by adding leading zeros. The Sony UPC 027242923232, for example, corresponds to GTIN 00027242923232.
Validate the number before treating it as evidence. The following helper accepts GTIN-8, GTIN-12, GTIN-13, and GTIN-14, then checks its final digit:
def valid_gtin(value):
digits = str(value or "").replace(" ", "").replace("-", "")
if not digits.isdigit() or len(digits) not in (8, 12, 13, 14):
return None
digits = digits.zfill(14)
total = sum(int(d) * (3 if i % 2 == 0 else 1)
for i, d in enumerate(reversed(digits[:-1])))
return digits if (10 - total % 10) % 10 == int(digits[-1]) else Noneprint(valid_gtin("027242923232"))
# 00027242923232A valid check digit only confirms the identifier's format. It doesn't prove that a retailer assigned it to the right product. You should still check for conflicting brands, variants, or bundle contents. Google also requires separate identifiers for color and size variants, which is a useful reminder not to collapse every model family into one saleable item.
If a GTIN isn't available, brand plus a manufacturer part number or model can help locate possible matches. But an mpn field is not automatically trustworthy. In another product-page example from the source research, a Costco item number appeared in a field labeled as an MPN. An ordinary retailer inventory number isn't necessarily a manufacturer part number.
A practical matching policy is:
- Confirm offers with the same valid GTIN when the product and variant details don't contradict one another.
- Review offers that share a brand and model or MPN but lack a verified GTIN.
- Reject or isolate conflicting variants, multipacks, and bundles until you can establish their identity.
You can still group related variants under one product family for navigation. Just don't use the family relationship as proof that their prices describe an identical item.
Build the catalog and keep a review queue
The accompanying spidra-unified-catalog.py example implements a conservative version of that policy. It normalizes the retailer results, creates one catalog group per verified GTIN, and writes unverified offers to a separate review file instead of silently merging them.
The example includes a compact fixture reconstructed from the November 11 research notes, so you can check the matching logic without making a live API request. Run it locally:
python spidra-unified-catalog.pyThe fixture produces:
Verified cross-source products: 1
Offers on verified cross-source products: 2
Review queue: 1The Walmart and eBay records share the GTIN, so they're grouped. The Amazon record has an ASIN but no GTIN in this reduced fixture, so it stays in the review queue. A human may be able to confirm it against the same model and color, but the script won't pretend that a similar title is enough.
The script writes catalog-output.json and catalog-review.json. To fetch the product pages instead of using the fixture, run it with --live after setting your API key. The live results can differ from the research snapshot as prices, stock, and available fields change.
This is deliberately a small catalog builder. The example reads one primary offer per product page and uses a title-based bundle flag as a warning, not a complete bundle detector. A production system will also need to expand seller-level offer arrays, use stable database IDs, group related variants, and provide a review workflow for decisions that can't be made safely from identifiers alone.
Compare offers only after matching the right variant
The original Sony collection demonstrates why this matters. The verified black-model listings were $181.15 on eBay and $198.00 on Walmart on November 11, 2026. Other eBay offers for the same product were listed as refurbished at $159.99. Walmart search results also contained bundles with accessories at substantially higher prices.
A comparison that selects the lowest number across those listings would mix different conditions and package contents. Instead, filter the matched variant's offers by condition, currency, seller, and fulfillment assumptions before calculating a price range. Keep sponsored search results marked as sponsored, but don't treat the advertising label itself as proof that the price or product identity is wrong.
One other detail matters when you start collecting at scale: location. Product prices and availability can depend on the shopper's region or delivery postcode. A price without its collection context may look comparable even when it isn't.
Handle missing data and refresh the catalog
Not every source will return a usable page. In the source research, the eBay search template failed twice, while a batch request using a listings schema returned search results. Best Buy and Target didn't return usable details for the selected Sony listings. These outcomes should be recorded as failed or incomplete sources, not as products that don't exist.
You should also validate successful responses. A job can finish while returning content that doesn't include the product fields you expected. Check for a useful title, a price when one is published, and the identifiers you require before accepting a result into the catalog.
Once your catalog is populated, schedule refreshes for offers that matter. Spidra Monitors can check pages or saved jobs for changes. When an offer changes, update its timestamp and values, then run the same matching and validation rules again. Keep historical prices separately if your application needs price trends or alerts.
Remember that every source has its own terms and applicable legal requirements. Review those before collecting data from retailers at scale. Our web scraping legality guide covers the broader considerations.
Build your catalog around records you can verify
A unified catalog needs more than product names and prices in a common JSON format. It needs a reliable way to identify variants, preserve each seller's offer, and explain why two records were merged.
Start with a handful of product URLs you know. Use Spidra to retrieve their details, run them through the same normalizer, and inspect the review queue before expanding to more stores. That will show you which identifiers you can trust and which sources need more work.