Extracting structured data from a webpage is easier when you already know its URL. But many applications start with a search query rather than a list of websites.
A company intelligence tool might need to discover businesses in a particular industry. A research assistant might need to find relevant sources before extracting facts. In both cases, the application has to decide which pages are worth processing before it can collect useful data.
This tutorial shows how to search the web and extract JSON using Python and Spidra Search. We'll build a pipeline that finds webpages, selects relevant candidates, extracts structured records with Batch Scrape, validates the results, removes basic duplicates, and preserves source URLs.
The example searches for open-source vector databases. The final output is a JSON dataset accompanied by an audit showing which pages were selected, excluded, or failed.
Search the web to discover relevant URLs
Suppose your application receives this query:
open source vector databaseYou want to collect the names of vector databases, their licenses, and the programming languages they're written in.
You don't have a list of source URLs yet.
Spidra Search handles this discovery step. It returns webpage titles, URLs, descriptions, and search positions that your application can inspect before scraping.
Install the Python SDK and Pydantic:
pip install spidra "pydantic>=2,<3"Create an API key in your Spidra dashboard and store it in the SPIDRA_API_KEY environment variable. The Python SDK documentation covers authentication and setup.
Start with a search request:
# pipeline.py
import os
from spidra import Spidra
client = Spidra(
api_key=os.environ["SPIDRA_API_KEY"]
)
query = "open source vector database"
found = client.search(
query,
sources=["web"],
limit=5,
)The SDK handles waiting for the search result.
Now collect the fields needed for source selection:
hits = [
{
"position": result.position,
"title": result.title,
"url": result.url,
"description": result.description,
}
for result in (found.data.web or [])
]These are candidate webpages. Finding them doesn't establish that their content is relevant, accurate, or suitable for extraction.
We'll make those decisions separately.
Filter and rank the search results
Searching for more pages than you intend to scrape gives your application room to exclude unsuitable results.
An earlier Spidra Search execution for our query returned these five results:
Rank | Search result | Decision |
|---|---|---|
1 | Selected | |
2 | Selected | |
3 | Selected | |
4 | Excluded: page budget | |
5 | Excluded: page budget |
For this example, we'll process at most three pages.
The first three candidates were selected according to their search positions. The remaining two weren't processed because of the page budget.
That's an important distinction: a page excluded by the processing budget isn't necessarily irrelevant.
The companion script contains a select_results() function that:
Rejects non-HTTP URLs, obvious local addresses, and repeated URLs.
Checks whether enough query terms appear in each result's title.
Keeps the search provider's ranking among eligible candidates.
Selects up to the configured page limit.
Records an exclusion reason for every result that isn't selected.
Call it after collecting the search results:
# Inside pipeline.py
selected, decisions = select_results(
hits,
query,
max_pages=3,
)
urls = [item["url"] for item in selected]The title-matching rule is intentionally simple. It helps screen candidates but isn't a reliable measure of search relevance.
A page can mention the right words without answering the query. Another relevant page might use different terminology.
For production research, inspect source descriptions, consider trusted domains, and evaluate the selected page content before treating it as evidence. URLs from search results are also untrusted input, so server-side URL and redirect safeguards still matter.
Keep the selection decisions in your audit rather than discarding the excluded results.
Define the JSON schema for extraction
Once we have the selected URLs, we need a consistent structure for the data returned from each page.
For our vector database example, every product record should contain:
name: The database's name.license: The stated license ornullif unavailable.written_in: The stated implementation language ornullif unavailable.
A JSON Schema defines these fields and the expected data types.
# pipeline.py, excerpt
SCHEMA = {
"type": "object",
"properties": {
"products": {"type": "array", "items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"license": {"type": ["string", "null"]},
"written_in": {"type": ["string", "null"]},
},
"required": ["name", "license", "written_in"],
}},
},
"required": ["products"],
}The products field is an array because a comparison article may mention several databases.
License and implementation language can be null, but their keys are still required. This lets downstream code distinguish a known value from a missing one without guessing.
Now write a prompt that limits extraction to information on the source page:
PROMPT = (
"List the open-source vector databases this page "
"actually names. Copy names from the source. "
"Give license and implementation language only "
"if explicitly stated on the page; otherwise use null. "
"Return an empty list if the page contains none."
)The schema and prompt have different jobs.
The schema constrains the result's structure. The prompt specifies what information to collect and when to leave fields unresolved.
Neither proves that the extracted facts are correct. We'll validate the structure locally and keep the sources for factual review.
See the structured output documentation for supported schemas and extraction behavior.
Extract structured JSON with Batch Scrape
We now have three selected URLs and a schema defining the information we want.
Spidra Scrape handles individual URLs. For several pages requiring the same extraction, Batch Scrape is the better fit.
It processes each URL independently and provides a separate status, result, and error for each page.
Use the same prompt and schema for all selected URLs:
# pipeline.py, excerpt
batch = client.batch_scrape(
urls,
prompt=PROMPT,
schema=SCHEMA,
)The Python SDK waits for the batch result.
A batch can finish even when one or more pages fail. Always inspect individual items rather than relying only on the batch status.
for item in batch.items:
if item.status != "completed":
# Record item.url and item.error
continue
# Validate item.result before accepting recordsWhat happened during extraction?
The Spidra execution used for this example attempted to scrape the first three search results.
Source | Status | Extraction outcome |
|---|---|---|
Designveloper | Failed | No usable result |
DataStackHub | Completed | Product names returned |
Redis | Completed | Empty |
The DataStackHub result included these entries in the retained output:
{
"products": [
{
"name": "Qdrant",
"written_in": null
},
{
"name": "Milvus",
"written_in": null
},
{
"name": "Weaviate",
"written_in": null
}
]
}The example above contains three visible records from the earlier execution. The full original response wasn't retained, so it shouldn't be interpreted as the complete list of extracted products.
The Redis page completed but returned no records. Notes from the execution identified nearly empty content associated with a cookie wall.
The Designveloper page failed, although the retained evidence doesn't include its exact error message.
These outcomes demonstrate why a completed request shouldn't automatically be counted as a successful extraction.
A failed scrape, a completed scrape with no matching records, and a completed scrape containing valid records are three different outcomes.
Your pipeline needs to distinguish them. If a page is nearly empty because of a cookie wall or another access problem, it may need a different retrieval approach rather than an empty database entry.
Validate extracted JSON with Pydantic
A schema tells Spidra what to return. Local validation checks that the returned object satisfies the application's rules before it enters a dataset.
Use Pydantic to define the accepted record structure:
# pipeline.py, excerpt
from pydantic import BaseModel, ConfigDict, Field
class Product(BaseModel):
model_config = ConfigDict(
extra="forbid", str_strip_whitespace=True
)
name: str = Field(min_length=1)
license: str | None
written_in: str | None
class Extraction(BaseModel):
model_config = ConfigDict(extra="forbid")
products: list[Product]Both nullable fields must be present, but their values can be None.
This avoids silently accepting a response that omits required keys. It also rejects unexpected fields and empty product names.
Validate each completed result:
from pydantic import ValidationError
try:
validated = Extraction.model_validate(
item.result
)
except ValidationError:
# Record an invalid_json_shape outcome
passThe companion script implements this check across every batch item and records the outcome in its audit.
The older retained response excerpt doesn't contain the license field. For the local replay, those missing values are explicitly represented as null. That allows the example to demonstrate the validation and export steps without inventing license information.
A fresh extraction using the complete schema should provide every required key, even when its value is null.
Validation doesn't prove factual accuracy
Suppose a page produces:
{
"name": "ExampleDB",
"license": "MIT",
"written_in": "Python"
}That object could pass the schema even if the page actually describes a different license or programming language.
Pydantic checks the types, presence, and format of fields. Confirming that a license or language is correct requires examining evidence from the source, ideally an authoritative project page when the distinction matters.
For a research or company-intelligence system, structural validation should happen before factual acceptance, not replace it.
Deduplicate records and retain their sources
The same product may appear on several result pages.
Without deduplication, your application could store multiple rows for Qdrant, Milvus, or another database simply because several comparison articles mentioned it.
For this tutorial, we'll begin with conservative name normalization.
# pipeline.py, excerpt
import re
def unique_name(name):
return " ".join(
re.findall(r"[a-z0-9]+", name.casefold())
)This maps simple formatting variants such as Qdrant and qdrant to the same key.
The companion script groups records using that key and retains every source URL associated with a match.
When fields conflict, it records the disagreement for review rather than blindly overwriting the existing value.
This approach isn't full entity resolution. Two different names can refer to the same company, while similar names may refer to unrelated products.
If your workflow needs company-level matching, canonical identifiers, or more sophisticated duplicate detection, our guide to removing duplicate records from multiple websites covers those problems separately.
For the query-to-JSON pipeline, the immediate goal is to avoid obvious duplicates without losing where each observation came from.
Export the validated records to JSON
The companion script writes two output files.
records.json contains accepted, deduplicated records. audit.json contains the search results, selection decisions, extraction outcomes, and available timing and credit information.
The retained Spidra output includes three visible product names. Processing those observations locally produces:
[
{
"name": "Qdrant",
"license": null,
"written_in": null,
"source_urls": [
"https://www.datastackhub.com/top-tools/open-source-vector-database-tools/"
]
},
{
"name": "Milvus",
"license": null,
"written_in": null,
"source_urls": [
"https://www.datastackhub.com/top-tools/open-source-vector-database-tools/"
]
},
{
"name": "Weaviate",
"license": null,
"written_in": null,
"source_urls": [
"https://www.datastackhub.com/top-tools/open-source-vector-database-tools/"
]
}
]This JSON is the locally validated representation of the three visible observations from the earlier Spidra execution. It isn't presented as the entire dataset originally returned by Batch Scrape.
Every record includes a source URL, making it possible to investigate the extracted information later.
That URL identifies where the product was mentioned. It does not establish the product's official website or verify its license.
Run the complete pipeline
The companion code includes a local replay so you can inspect the pipeline without making API requests.
Run:
python pipeline.py --replayThe local replay reports:
Unique observed records: 3
Selected pages: 3
Page outcomes: ['failed_scrape', 'accepted', 'no_usable_records']The replay uses the historical response excerpt. It doesn't perform a new search or scrape.
To run the complete pipeline against the live API, use your own API key:
python pipeline.py \
--query "open source vector database" \
--limit 5 \
--max-pages 3The script searches, selects pages, submits the batch, validates returned objects, deduplicates accepted records, and saves the results and audit information.
The new search results, page statuses, and extracted records may differ from the example because webpages and search rankings change. The complete pipeline has passed local validation tests, while its current end-to-end API execution remains unverified.
Measure search relevance and extraction quality separately
A successful API request doesn't necessarily mean your data pipeline worked.
There are two quality questions to answer.
Did Search find the right pages?
Search relevance evaluates whether the selected webpages are suitable sources for the original query.
A relevant page should contain information that can help answer the question or supply the requested records.
Our example selected three of five candidates using a title-based screen and a page budget. That is a selection count, not a verified relevance score.
To evaluate relevance, review a sample of selected and excluded pages, label whether each page is useful for the task, and inspect false positives and false negatives.
A page with a convincing title can still contain unrelated, outdated, or shallow information.
Did extraction recover useful records?
Extraction quality evaluates the content returned from the selected pages.
For the earlier three-page batch:
Outcome | Pages |
|---|---|
Failed to scrape | 1 |
Completed with usable records in the retained excerpt | 1 |
Completed with no usable records | 1 |
The page with usable records still needs field-level verification.
A production pipeline should separately track how many pages returned valid JSON, how many returned empty or invalid results, which fields are missing, and how many sampled facts can be confirmed against their sources.
Keeping these measurements separate helps identify where the pipeline needs improvement.
If Search repeatedly selects irrelevant pages, changing the extraction prompt won't solve the discovery problem.
If Search identifies relevant pages but extraction returns empty or inaccurate records, improving ranking alone won't fix the result.
Both stages affect the final dataset, but they fail for different reasons.
Track processing time, credits, and failures
The Search-to-Batch execution used in this example recorded its total processing time and credit consumption.
Measurement | Recorded result |
|---|---|
Search results returned | 5 |
Pages submitted for scraping | 3 |
Failed scrapes | 1 |
Completed pages with empty extraction | 1 |
Total elapsed time | 54 seconds |
Total credits | 7 |
The recorded total includes one credit for Search and the remaining credits for page processing. Failed scrape items weren't charged.
These figures describe the original execution. They aren't a benchmark of the updated companion script.
The retained evidence doesn't include the complete per-page credit breakdown, exact failure message, or full extraction response. Those details are therefore excluded from the reported results rather than estimated.
For new executions, the companion script measures Search and Batch Scrape wall-clock durations separately and retains credit values reported for individual batch items.
The Search charge can be confirmed from the account's logs when preparing a complete cost report.
The Batch Scrape documentation covers item statuses, credit reporting, and retrying failed pages.
When should you use Search, Scrape, or Batch Scrape?
Choose the Spidra operation based on what your application already knows.
Starting point | Use |
|---|---|
A question or search query, with no URLs | Search |
One known webpage requiring structured extraction | Scrape |
Several selected URLs requiring the same extraction | Batch Scrape |
Search results that only need page Markdown | Search with page-content fetching |
Search can fetch Markdown from returned webpages directly. That's useful when your application wants the page text without a custom extraction schema.
For this tutorial, Batch Scrape is appropriate because each selected page needs the same JSON structure, its own status, and an independently validated result.
If you already know the relevant URLs, skip Search and begin with Scrape or Batch Scrape.
Build a search-to-JSON pipeline you can audit
A reliable search-to-JSON workflow needs more than a successful search response and a list of extracted objects.
It should show why sources were selected, what each page returned, which records passed validation, and where the accepted information came from.
Spidra Search discovers candidate pages. Batch Scrape extracts selected pages into a shared schema. Python handles filtering, validation, deduplication, and provenance.
Together, these steps form a reusable foundation for research assistants, company intelligence tools, and other applications that need structured information without knowing the source URLs in advance.
Create a Spidra account and run the companion Python pipeline with your own query. Review the selection audit and extracted JSON before adding the records to your application.