Skip to main content
Tutorials · 13 min read

How to search the web and extract JSON with Python

Learn how to search the web, filter relevant pages, and extract structured JSON using Python and Spidra Search with Batch Scrape.

How to search the web and extract JSON with Python

Extracting structured data from a webpage is easier when you already know its URL. But many applications start with a search query rather than a list of websites.

A company intelligence tool might need to discover businesses in a particular industry. A research assistant might need to find relevant sources before extracting facts. In both cases, the application has to decide which pages are worth processing before it can collect useful data.

This tutorial shows how to search the web and extract JSON using Python and Spidra Search. We'll build a pipeline that finds webpages, selects relevant candidates, extracts structured records with Batch Scrape, validates the results, removes basic duplicates, and preserves source URLs.

The example searches for open-source vector databases. The final output is a JSON dataset accompanied by an audit showing which pages were selected, excluded, or failed.

Search the web to discover relevant URLs

Suppose your application receives this query:

open source vector database

You want to collect the names of vector databases, their licenses, and the programming languages they're written in.

You don't have a list of source URLs yet.

Spidra Search handles this discovery step. It returns webpage titles, URLs, descriptions, and search positions that your application can inspect before scraping.

Install the Python SDK and Pydantic:

pip install spidra "pydantic>=2,<3"

Create an API key in your Spidra dashboard and store it in the SPIDRA_API_KEY environment variable. The Python SDK documentation covers authentication and setup.

Start with a search request:

# pipeline.py
import os
from spidra import Spidra

client = Spidra(
    api_key=os.environ["SPIDRA_API_KEY"]
)

query = "open source vector database"

found = client.search(
    query,
    sources=["web"],
    limit=5,
)

The SDK handles waiting for the search result.

Now collect the fields needed for source selection:

hits = [
    {
        "position": result.position,
        "title": result.title,
        "url": result.url,
        "description": result.description,
    }
    for result in (found.data.web or [])
]

These are candidate webpages. Finding them doesn't establish that their content is relevant, accurate, or suitable for extraction.

We'll make those decisions separately.

Filter and rank the search results

Searching for more pages than you intend to scrape gives your application room to exclude unsuitable results.

An earlier Spidra Search execution for our query returned these five results:

Rank

Search result

Decision

1

Best Open-Source Vector Database in 2026

Selected

2

8 Best Open Source Vector Database Tools

Selected

3

Comparing Open Source Vector Databases

Selected

4

Vector Database Benchmark

Excluded: page budget

5

Open Source Vector Database Tools

Excluded: page budget

For this example, we'll process at most three pages.

The first three candidates were selected according to their search positions. The remaining two weren't processed because of the page budget.

That's an important distinction: a page excluded by the processing budget isn't necessarily irrelevant.

The companion script contains a select_results() function that:

  • Rejects non-HTTP URLs, obvious local addresses, and repeated URLs.

  • Checks whether enough query terms appear in each result's title.

  • Keeps the search provider's ranking among eligible candidates.

  • Selects up to the configured page limit.

  • Records an exclusion reason for every result that isn't selected.

Call it after collecting the search results:

# Inside pipeline.py
selected, decisions = select_results(
    hits,
    query,
    max_pages=3,
)

urls = [item["url"] for item in selected]

The title-matching rule is intentionally simple. It helps screen candidates but isn't a reliable measure of search relevance.

A page can mention the right words without answering the query. Another relevant page might use different terminology.

For production research, inspect source descriptions, consider trusted domains, and evaluate the selected page content before treating it as evidence. URLs from search results are also untrusted input, so server-side URL and redirect safeguards still matter.

Keep the selection decisions in your audit rather than discarding the excluded results.

Define the JSON schema for extraction

Once we have the selected URLs, we need a consistent structure for the data returned from each page.

For our vector database example, every product record should contain:

  • name: The database's name.

  • license: The stated license or null if unavailable.

  • written_in: The stated implementation language or null if unavailable.

A JSON Schema defines these fields and the expected data types.

# pipeline.py, excerpt
SCHEMA = {
    "type": "object",
    "properties": {
        "products": {"type": "array", "items": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "license": {"type": ["string", "null"]},
                "written_in": {"type": ["string", "null"]},
            },
            "required": ["name", "license", "written_in"],
        }},
    },
    "required": ["products"],
}

The products field is an array because a comparison article may mention several databases.

License and implementation language can be null, but their keys are still required. This lets downstream code distinguish a known value from a missing one without guessing.

Now write a prompt that limits extraction to information on the source page:

PROMPT = (
    "List the open-source vector databases this page "
    "actually names. Copy names from the source. "
    "Give license and implementation language only "
    "if explicitly stated on the page; otherwise use null. "
    "Return an empty list if the page contains none."
)

The schema and prompt have different jobs.

The schema constrains the result's structure. The prompt specifies what information to collect and when to leave fields unresolved.

Neither proves that the extracted facts are correct. We'll validate the structure locally and keep the sources for factual review.

See the structured output documentation for supported schemas and extraction behavior.

Extract structured JSON with Batch Scrape

We now have three selected URLs and a schema defining the information we want.

Spidra Scrape handles individual URLs. For several pages requiring the same extraction, Batch Scrape is the better fit.

It processes each URL independently and provides a separate status, result, and error for each page.

Use the same prompt and schema for all selected URLs:

# pipeline.py, excerpt
batch = client.batch_scrape(
    urls,
    prompt=PROMPT,
    schema=SCHEMA,
)

The Python SDK waits for the batch result.

A batch can finish even when one or more pages fail. Always inspect individual items rather than relying only on the batch status.

for item in batch.items:
    if item.status != "completed":
        # Record item.url and item.error
        continue

    # Validate item.result before accepting records

What happened during extraction?

The Spidra execution used for this example attempted to scrape the first three search results.

Source

Status

Extraction outcome

Designveloper

Failed

No usable result

DataStackHub

Completed

Product names returned

Redis

Completed

Empty products array

The DataStackHub result included these entries in the retained output:

{
  "products": [
    {
      "name": "Qdrant",
      "written_in": null
    },
    {
      "name": "Milvus",
      "written_in": null
    },
    {
      "name": "Weaviate",
      "written_in": null
    }
  ]
}

The example above contains three visible records from the earlier execution. The full original response wasn't retained, so it shouldn't be interpreted as the complete list of extracted products.

The Redis page completed but returned no records. Notes from the execution identified nearly empty content associated with a cookie wall.

The Designveloper page failed, although the retained evidence doesn't include its exact error message.

These outcomes demonstrate why a completed request shouldn't automatically be counted as a successful extraction.

A failed scrape, a completed scrape with no matching records, and a completed scrape containing valid records are three different outcomes.

Your pipeline needs to distinguish them. If a page is nearly empty because of a cookie wall or another access problem, it may need a different retrieval approach rather than an empty database entry.

Validate extracted JSON with Pydantic

A schema tells Spidra what to return. Local validation checks that the returned object satisfies the application's rules before it enters a dataset.

Use Pydantic to define the accepted record structure:

# pipeline.py, excerpt
from pydantic import BaseModel, ConfigDict, Field

class Product(BaseModel):
    model_config = ConfigDict(
        extra="forbid", str_strip_whitespace=True
    )
    name: str = Field(min_length=1)
    license: str | None
    written_in: str | None

class Extraction(BaseModel):
    model_config = ConfigDict(extra="forbid")
    products: list[Product]

Both nullable fields must be present, but their values can be None.

This avoids silently accepting a response that omits required keys. It also rejects unexpected fields and empty product names.

Validate each completed result:

from pydantic import ValidationError

try:
    validated = Extraction.model_validate(
        item.result
    )
except ValidationError:
    # Record an invalid_json_shape outcome
    pass

The companion script implements this check across every batch item and records the outcome in its audit.

The older retained response excerpt doesn't contain the license field. For the local replay, those missing values are explicitly represented as null. That allows the example to demonstrate the validation and export steps without inventing license information.

A fresh extraction using the complete schema should provide every required key, even when its value is null.

Validation doesn't prove factual accuracy

Suppose a page produces:

{
  "name": "ExampleDB",
  "license": "MIT",
  "written_in": "Python"
}

That object could pass the schema even if the page actually describes a different license or programming language.

Pydantic checks the types, presence, and format of fields. Confirming that a license or language is correct requires examining evidence from the source, ideally an authoritative project page when the distinction matters.

For a research or company-intelligence system, structural validation should happen before factual acceptance, not replace it.

Deduplicate records and retain their sources

The same product may appear on several result pages.

Without deduplication, your application could store multiple rows for Qdrant, Milvus, or another database simply because several comparison articles mentioned it.

For this tutorial, we'll begin with conservative name normalization.

# pipeline.py, excerpt
import re

def unique_name(name):
    return " ".join(
        re.findall(r"[a-z0-9]+", name.casefold())
    )

This maps simple formatting variants such as Qdrant and qdrant to the same key.

The companion script groups records using that key and retains every source URL associated with a match.

When fields conflict, it records the disagreement for review rather than blindly overwriting the existing value.

This approach isn't full entity resolution. Two different names can refer to the same company, while similar names may refer to unrelated products.

If your workflow needs company-level matching, canonical identifiers, or more sophisticated duplicate detection, our guide to removing duplicate records from multiple websites covers those problems separately.

For the query-to-JSON pipeline, the immediate goal is to avoid obvious duplicates without losing where each observation came from.

Export the validated records to JSON

The companion script writes two output files.

records.json contains accepted, deduplicated records. audit.json contains the search results, selection decisions, extraction outcomes, and available timing and credit information.

The retained Spidra output includes three visible product names. Processing those observations locally produces:

[
  {
    "name": "Qdrant",
    "license": null,
    "written_in": null,
    "source_urls": [
      "https://www.datastackhub.com/top-tools/open-source-vector-database-tools/"
    ]
  },
  {
    "name": "Milvus",
    "license": null,
    "written_in": null,
    "source_urls": [
      "https://www.datastackhub.com/top-tools/open-source-vector-database-tools/"
    ]
  },
  {
    "name": "Weaviate",
    "license": null,
    "written_in": null,
    "source_urls": [
      "https://www.datastackhub.com/top-tools/open-source-vector-database-tools/"
    ]
  }
]

This JSON is the locally validated representation of the three visible observations from the earlier Spidra execution. It isn't presented as the entire dataset originally returned by Batch Scrape.

Every record includes a source URL, making it possible to investigate the extracted information later.

That URL identifies where the product was mentioned. It does not establish the product's official website or verify its license.

Run the complete pipeline

The companion code includes a local replay so you can inspect the pipeline without making API requests.

Run:

python pipeline.py --replay

The local replay reports:

Unique observed records: 3
Selected pages: 3
Page outcomes: ['failed_scrape', 'accepted', 'no_usable_records']

The replay uses the historical response excerpt. It doesn't perform a new search or scrape.

To run the complete pipeline against the live API, use your own API key:

python pipeline.py \
  --query "open source vector database" \
  --limit 5 \
  --max-pages 3

The script searches, selects pages, submits the batch, validates returned objects, deduplicates accepted records, and saves the results and audit information.

The new search results, page statuses, and extracted records may differ from the example because webpages and search rankings change. The complete pipeline has passed local validation tests, while its current end-to-end API execution remains unverified.

Measure search relevance and extraction quality separately

A successful API request doesn't necessarily mean your data pipeline worked.

There are two quality questions to answer.

Did Search find the right pages?

Search relevance evaluates whether the selected webpages are suitable sources for the original query.

A relevant page should contain information that can help answer the question or supply the requested records.

Our example selected three of five candidates using a title-based screen and a page budget. That is a selection count, not a verified relevance score.

To evaluate relevance, review a sample of selected and excluded pages, label whether each page is useful for the task, and inspect false positives and false negatives.

A page with a convincing title can still contain unrelated, outdated, or shallow information.

Did extraction recover useful records?

Extraction quality evaluates the content returned from the selected pages.

For the earlier three-page batch:

Outcome

Pages

Failed to scrape

1

Completed with usable records in the retained excerpt

1

Completed with no usable records

1

The page with usable records still needs field-level verification.

A production pipeline should separately track how many pages returned valid JSON, how many returned empty or invalid results, which fields are missing, and how many sampled facts can be confirmed against their sources.

Keeping these measurements separate helps identify where the pipeline needs improvement.

If Search repeatedly selects irrelevant pages, changing the extraction prompt won't solve the discovery problem.

If Search identifies relevant pages but extraction returns empty or inaccurate records, improving ranking alone won't fix the result.

Both stages affect the final dataset, but they fail for different reasons.

Track processing time, credits, and failures

The Search-to-Batch execution used in this example recorded its total processing time and credit consumption.

Measurement

Recorded result

Search results returned

5

Pages submitted for scraping

3

Failed scrapes

1

Completed pages with empty extraction

1

Total elapsed time

54 seconds

Total credits

7

The recorded total includes one credit for Search and the remaining credits for page processing. Failed scrape items weren't charged.

These figures describe the original execution. They aren't a benchmark of the updated companion script.

The retained evidence doesn't include the complete per-page credit breakdown, exact failure message, or full extraction response. Those details are therefore excluded from the reported results rather than estimated.

For new executions, the companion script measures Search and Batch Scrape wall-clock durations separately and retains credit values reported for individual batch items.

The Search charge can be confirmed from the account's logs when preparing a complete cost report.

The Batch Scrape documentation covers item statuses, credit reporting, and retrying failed pages.

When should you use Search, Scrape, or Batch Scrape?

Choose the Spidra operation based on what your application already knows.

Starting point

Use

A question or search query, with no URLs

Search

One known webpage requiring structured extraction

Scrape

Several selected URLs requiring the same extraction

Batch Scrape

Search results that only need page Markdown

Search with page-content fetching

Search can fetch Markdown from returned webpages directly. That's useful when your application wants the page text without a custom extraction schema.

For this tutorial, Batch Scrape is appropriate because each selected page needs the same JSON structure, its own status, and an independently validated result.

If you already know the relevant URLs, skip Search and begin with Scrape or Batch Scrape.

Build a search-to-JSON pipeline you can audit

A reliable search-to-JSON workflow needs more than a successful search response and a list of extracted objects.

It should show why sources were selected, what each page returned, which records passed validation, and where the accepted information came from.

Spidra Search discovers candidate pages. Batch Scrape extracts selected pages into a shared schema. Python handles filtering, validation, deduplication, and provenance.

Together, these steps form a reusable foundation for research assistants, company intelligence tools, and other applications that need structured information without knowing the source URLs in advance.

Create a Spidra account and run the companion Python pipeline with your own query. Review the selection audit and extracted JSON before adding the records to your application.

Frequently asked questions

Use a search API to find candidate webpages, filter the results, and submit selected URLs to a scraping API with a JSON Schema. Validate the returned objects in Python, deduplicate records using reliable identifiers, and preserve source URLs so the extracted information remains traceable.

Share this article

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.