Skip to main content
TutorialsNovember 19, 2026 · 11 min read

How to remove duplicate records from multiple websites

Learn how to deduplicate scraped data with Python and Spidra, normalize URLs, match records, and export unique CSV rows with their original sources.

How to remove duplicate records from multiple websites

Scraping multiple websites often leaves you with duplicate data. The same company might appear in several directories, a product might have listings across different stores, or two webpages might contain identical records.

These duplicates can inflate your counts, create conflicting entries, and make downstream processing harder.

You can deduplicate scraped data with Python by normalizing URLs, matching records using consistent keys, and keeping track of where each record came from.

This tutorial uses Spidra Search and Batch Scrape to collect data, then builds a Python pipeline that removes duplicate records and exports a clean CSV with source information.

We'll demonstrate the matching logic using overlapping pages from Quotes to Scrape, then explain what changes when matching companies or products across different websites.

Separate duplicate URLs from duplicate records

Before removing duplicates, decide what you're actually matching.

There are two different problems.

URL deduplication removes repeated webpage addresses before scraping.

For example:

https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/page/1/?utm_source=newsletter

If the second URL differs only by a tracking parameter, you can treat both as the same retrieval target.

Record deduplication happens after scraping.

Two different webpages might contain the same quote, product, or company. Even if their URLs are different, the extracted records may represent the same thing.

For example:

https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/js/

These pages display the same first-page quotes through different rendering approaches.

Removing duplicate URLs won't detect that overlap. You need to compare the extracted records.

For real-world companies and products, this becomes entity resolution. Two records can have different names or descriptions but still refer to the same entity.

We'll handle URL cleanup first, then record matching.

Collect records using Spidra Search and Scrape

If you already have a list of URLs, you can start scraping directly.

Otherwise, use Spidra Search to discover relevant pages.

Install the Python SDK:

python -m pip install spidra

Create a Spidra API key and set it in the SPIDRA_API_KEY environment variable.

Then search for relevant pages:

# discover.py
import os
from spidra import Spidra

client = Spidra(
    api_key=os.environ["SPIDRA_API_KEY"]
)

results = client.search(
    "quotes to scrape author tags",
    sources=["web"],
    limit=10,
)

urls = [
    result.url
    for result in (results.data.web or [])
]

Search returns result information that includes URLs you can inspect before scraping.

For this example, we'll use three known pages from Quotes to Scrape, a website designed for scraping practice.

urls = [
    "https://quotes.toscrape.com/page/1/",
    "https://quotes.toscrape.com/js/",
    "https://quotes.toscrape.com/tag/love/",
]

We'll request the same fields from every page.

Extract records using a shared schema

A consistent schema makes it easier to compare results from different webpages.

For our quote dataset, each record needs an author and quote text.

schema = {
    "type": "object",
    "properties": {
        "quotes": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "author": {"type": "string"},
                    "text": {"type": "string"},
                },
            },
        },
    },
}

Use Batch Scrape to process the selected pages:

batch = client.batch_scrape(
    urls,
    prompt="Extract every quote on the page",
    schema=schema,
)

Spidra returns an individual result for each URL, so you can preserve its source and check whether the extraction succeeded.

records = []

for item in batch.items:
    if item.status != "completed":
        print("FAILED", item.url, item.error)
        continue

    for quote in item.result["quotes"]:
        records.append({
            **quote,
            "source_url": item.url,
        })

Keep the original source_url alongside each extracted record.

A batch can finish while individual pages fail, so check the item statuses rather than assuming the entire collection succeeded. Failed pages aren't charged.

The Batch Scrape documentation covers the available options. For larger URL collections, see how to scrape multiple URLs in Python and handle failures.

The combined Search and Batch Scrape example follows the documented SDK interface. Its new end-to-end execution still needs a live capture before being presented as tested output.

Normalize URLs without removing meaningful information

The first deduplication step happens before scraping.

A URL may contain tracking parameters that don't change the page's content.

For example:

https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/page/1/?utm_source=weekly

The companion Python script includes a normalize_url() function that removes known tracking parameters, normalizes the scheme and hostname, and preserves query parameters that may identify different resources.

You can use it like this:

# check_urls.py
from deduplicate import normalize_url

urls = [
    "https://quotes.toscrape.com/page/1/?utm_source=weekly",
    "https://quotes.toscrape.com/page/1/",
    "https://quotes.toscrape.com/js/",
]

for url in urls:
    print(normalize_url(url))

Output:

https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/js/

The first two URLs now share the same normalized value.

You can remove repeated values while preserving their order:

unique_urls = list(dict.fromkeys(
    normalize_url(url) for url in urls
))

However, avoid deleting query parameters indiscriminately.

A product website might use variant=blue and variant=red to identify different product variants. Removing those parameters could merge distinct pages.

Likewise, HTTP and HTTPS, www and non-www, or trailing-slash variations shouldn't automatically be treated as equivalent on every website.

Use redirects or other reliable page evidence before merging URL aliases.

URL normalization identifies possible repeated retrieval targets. It doesn't establish that the entities described on those pages are identical.

Match duplicate records using normalized fields

Once you've collected the records, you need to decide which ones represent the same item.

For our quote dataset, we'll use two fields:

  • Author name

  • Quote text

Both values need normalization before comparison.

For example, these two strings should be equivalent for this matching rule:

“André Gide”
"André Gide"

Python can decode HTML entities, normalize Unicode, remove extra whitespace, and standardize letter case.

The companion script defines:

def normalized_text(value):
    value = unicodedata.normalize(
        "NFKC",
        html.unescape(value or "")
    ).casefold()

    value = " ".join(value.strip().split())
    return value.strip("\u201c\u201d\"' ")

The function uses Python's standard unicodedata and html modules. Their imports are included in the complete script.

Next, build a matching key from the author and quote text.

def record_key(author, text):
    return json.dumps(
        [
            normalized_text(author),
            normalized_text(text),
        ],
        ensure_ascii=False,
    )

Now two records with the same normalized author and quote receive the same identity key, even if their text formatting differs.

This is exact matching after normalization, not fuzzy semantic matching.

It works for this quote dataset because the selected fields identify the record we're trying to preserve.

Why matching by author alone fails

Consider two records:

[
  {
    "author": "Albert Einstein",
    "text": "Try not to become a man of success. Rather become a man of value."
  },
  {
    "author": "Albert Einstein",
    "text": "There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle."
  }
]

Both quotes have the same author but contain different text.

If you remove duplicates using only the author field, you'll lose a valid quote.

The sample dataset contains 10 unique quotes attributed to eight distinct authors. Matching only by author would reduce the dataset incorrectly.

Comparing normalized author and quote text preserves the distinct records.

The same rule applies to other datasets: the fields you use for matching must actually identify the type of record you're deduplicating.

Merge records while preserving their original sources

After identifying a duplicate, don't simply discard its source.

Suppose the same quote appears on two pages:

https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/js/

You want one record in the final dataset, but you still need to know that both pages contained it.

The resulting record can retain a source list.

Here is a shortened view of one record from the local output:

{
  "author": "Albert Einstein",
  "source_urls": [
    "https://quotes.toscrape.com/js/",
    "https://quotes.toscrape.com/page/1/"
  ],
  "source_count": 2
}

The complete CSV also retains the quote text and a stable record identifier.

The Python script calculates an identity key for each quote and uses it to group matching observations.

When another record has the same key, the script marks it as a duplicate while retaining its source-page URL.

The script also records the original URL separately from its normalized form.

This gives you three useful pieces of information: the unique record, the pages where it appeared, and the reason another observation was merged with it.

That becomes useful when source data changes or you need to investigate a questionable match.

Export deduplicated records to CSV

Let's turn the matching logic into a usable dataset.

The companion project includes a CSV constructed from previously captured quote records. It contains 20 observations across two page views: the static first page and its JavaScript-rendered equivalent.

The matching logic uses the normalized author and quote text.

Run the local checks first:

python test_deduplicate.py

Output:

13 local assertions passed (no API calls)

Then run the deduplication script:

python deduplicate.py input_records.csv --out-dir output

Output:

20 input observations -> 10 unique quote records
20 distinct record-to-source links; 10 duplicate observations

The script creates three files.

unique_records.csv contains one row per unique quote. Each row includes the record ID, author, text, representative source URL, and the list of associated source URLs.

provenance.csv connects each unique record to its original and normalized source-page URLs.

merge_audit.csv records which observations created new records, which matched an existing identity, and which require review because essential fields are missing.

The local output contains 10 unique records, while preserving all 20 distinct record-to-source links.

These figures describe the locally processed sample constructed from earlier quote data, not a fresh live Spidra scraping run.

A larger recorded Spidra example

The Spidra product recipe also contains a separate scraping example using overlapping quote pages.

Its output is:

4 URLs -> 3 after normalizing
30 records -> 19 unique
Albert Einstein | The world as we have created it is a process of ou | found on 2 pages

That example demonstrates why URL deduplication isn't enough.

Three distinct page URLs produced 30 extracted quote observations, but only 19 unique quote identities.

The JavaScript page repeated the first page's quotes, and the tag page shared another quote with the collection.

The original recipe uses more aggressive URL alias normalization than the conservative approach in our companion code. Adopt those broader equivalences only when they're valid for the target website.

Avoid false-positive matches between companies and products

The quote example uses normalized text for exact record matching.

Company and product records need more careful identity rules.

For example, two directories may list a business under slightly different names:

Example Technologies Ltd
Example Technologies

The similarity suggests a possible match, but doesn't establish that both listings refer to the same legal organization.

For company records, stronger matching evidence can include a verified company registration identifier, an independently confirmed official website, and consistent organization details.

Even when two records share a brand name, they might represent subsidiaries, branches, or separate companies.

Product matching has a similar problem.

Two retailers may list products with nearly identical names, but one might be a different model, color, condition, or bundle.

Use verified identifiers such as GTINs when available. Otherwise, compare corroborated manufacturer, model, and variant attributes.

A useful matching policy has three outcomes:

Evidence

Decision

Reliable identity match with no conflicting attributes

Merge

Conflicting identifiers or important variant details

Keep separate

Similar names but insufficient proof

Flag for review

Fuzzy matching can help you find possible duplicates, but it shouldn't automatically override conflicting identifiers.

If you're working specifically with retail product data, our guide to building a unified product catalog from multiple websites covers product identity, seller offers, and variants in more detail.

Store unique records and source observations separately

CSV works well for a small export.

For recurring enrichment pipelines, a database makes it easier to retain history and apply updated matching rules.

Instead of deleting duplicate input rows permanently, separate resolved records from individual source observations.

The companion project includes a SQLite schema with three tables:

  • unique_records stores resolved identities and their matching rules.

  • source_observations stores the original source URL, raw payload, and collection metadata.

  • match_review stores uncertain matches that require a decision.

This allows several observations to reference the same resolved entity.

It also lets you preserve information such as the original search query, collection timestamp, and scraping job identifier.

Keep a representative source URL separate from the canonical URL of the underlying entity.

The first page where you found a company isn't necessarily its official website. A normalized webpage URL also doesn't prove entity identity.

If you haven't verified a canonical entity URL, leave that value empty.

Handle duplicates when new data arrives

The same matching rules can be used in a recurring enrichment pipeline.

When a new batch arrives, compare its records against the identities already stored in your database.

If a new observation matches an existing identity, attach it to that record while preserving the new source and collection time.

If the identity is new, create a new record.

If the match is uncertain, keep the observation and send it for review instead of forcing a merge.

A failed scrape should also remain separate from an empty or missing record. If a page couldn't be retrieved, that doesn't prove the entity no longer exists.

Spidra handles the collection and extraction steps. Your application decides which observations represent the same entity and which values should update existing records.

This separation keeps the data pipeline easier to audit and correct.

Build a clean dataset without losing source information

Removing duplicates from scraped data requires two separate steps: normalize repeated URLs before retrieval, then match the records extracted from those pages.

For the quote example, the Python pipeline reduces 20 overlapping observations to 10 unique records while retaining their source associations and an audit of the merge decisions.

The same processing structure can accept data from different websites. For companies and products, use reliable identity evidence and review uncertain matches before merging them.

Create a Spidra account, collect records from webpages you're permitted to scrape, and use the companion Python script to export deduplicated CSV data with its sources preserved.

Frequently asked questions

Normalize the fields that identify each record, build a consistent identity key, and group observations with matching keys. Keep their source URLs and matching decisions. For quotes, author plus quote text can work; for companies and products, stronger identity attributes are usually necessary.

Share this article

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.