How to build a company enrichment API with Python

A company name is rarely enough for a sales research tool or B2B database. You need to find its website, collect public details, and know where each value came from. This tutorial builds a company enrichment API with Python and FastAPI that accepts a company name or domain and returns a structured profile with field-level source URLs.

We'll use Spidra for website search and structured page extraction. FastAPI will handle requests, candidate matching, missing fields, and conflicts. The result is a useful public-web enrichment service, not a verified corporate registry.

Set up FastAPI and Spidra

You'll need Python 3.10 or later and a Spidra API key from the dashboard. Start with a small project:

mkdir company-enrichment-api && cd company-enrichment-api
python -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn spidra

On Windows, activate the virtual environment with .venv\Scripts\activate instead. Set the API key as an environment variable:

export SPIDRA_API_KEY="spd_YOUR_API_KEY"

On PowerShell, use $env:SPIDRA_API_KEY="spd_YOUR_API_KEY". Keep your real key out of source control.

The Python SDK has an asynchronous client, AsyncSpidra, which fits FastAPI's async routes. Its search() method returns search results; scrape() accepts URLs and can extract data against a JSON schema. The current SDK documentation is the reference for both methods.

Accept either a name or a domain

A domain lets you go straight to a website. A name needs another step because more than one company may use it. The request model should allow one input at a time.

Create main.py:

# main.py
from fastapi import FastAPI
from pydantic import BaseModel, Field, model_validator

app = FastAPI(title="Company enrichment API")

class EnrichmentRequest(BaseModel):
    name: str | None = Field(default=None, max_length=120)
    domain: str | None = Field(default=None, max_length=253)

    @model_validator(mode="after")
    def exactly_one(self):
        if sum(bool(x and x.strip()) for x in (self.name, self.domain)) != 1:
            raise ValueError("Provide exactly one nonempty name or domain")
        return self

The endpoint will accept {"name":"Stripe"} or {"domain":"stripe.com"}. A request with both fields, or neither, returns a validation error.

Before passing a supplied domain to a scraping service, also check its syntax and reject URLs, IP addresses, and obvious internal-use hostnames. The companion main.py supplied with this tutorial includes the hostname filter and the remaining route functions. That filter is a starting point, not complete protection against malicious DNS or redirects.

Find company websites with Spidra Search

For name-based requests, search for candidate company sites rather than guessing a domain from the name. A guess such as adding .com can point to a different business.

# Search for candidate company websites
from spidra import AsyncSpidra

async def search_company(client: AsyncSpidra, name: str):
    result = await client.search(
        f'"{name}" official company website',
        sources=["web"], limit=8,
    )
    return result.data.web or []

Search returns metadata such as page titles, URLs, and descriptions. Use it to build a candidate list. Remove directories and social profiles when you're looking for the company's own domain, and collapse duplicate results from the same host. The Spidra Search guide explains the search response and available filters.

A simple ranking can use three signals: resemblance between the requested name and domain, a matching page title, and a matching description. Keep the scoring visible to the caller so it's possible to understand the selection. Our example requires stronger evidence than a domain resemblance alone.

Don't make search ranking sound like identity verification. The best result is a candidate website. If two candidates score too closely, the service returns HTTP 409 with the alternatives instead of silently selecting one. If no result clears the minimum score, it returns 404.

The response also includes a selection record with the search query, scored candidates, and chosen domain. This makes company-name resolution inspectable when someone reviews an enriched record later.

Search for an About page before scraping

The homepage often explains a business, but its About page may be a better source for a founding date or headquarters. Don't assume every company uses /about.

Start with the homepage. Then search for an About or Contact page on the selected domain, and only keep URLs whose hostname matches it:

# Find one likely company information page
async def select_pages(client, domain):
    pages = [f"https://{domain}/"]
    job = await client.search(
        f"site:{domain} about company", sources=["web"], limit=10,
    )
    for item in job.data.web or []:
        url = cleaned_result_url(str(item.url))
        if url and urlsplit(url).hostname == domain:
            if urlsplit(url).path.strip("/").lower() in {"about", "about-us", "company", "contact", "contact-us"}:
                pages.append(url)
                break
    return list(dict.fromkeys(pages))

cleaned_result_url() is a hostname and URL-format helper in the downloadable implementation. It drops unexpected schemes, credentials, ports, and URL fragments. For production, network-level URL safety still needs a separate review.

Limiting the example to two pages keeps the extraction and its sources easy to follow. If your benchmark shows that too many fields are missing, you can widen page discovery later.

Extract company inforation with a JSON schema

We'll collect the details that businesses sometimes publish on their own websites: name, description, industry, headquarters, founding year, and LinkedIn URL. If the page doesn't state something, the field should be null.

Define the extraction schema:

# Structured fields returned from each page
FIELDS = (
    "company_name", "description", "industry",
    "headquarters", "founded_year", "linkedin_url",
)
SCHEMA = {
    "type": "object",
    "properties": {key: {"type": ["string", "null"]} for key in FIELDS},
    "required": list(FIELDS),
}

Pass the schema to Spidra Scrape, along with an instruction to extract only explicit page content:

# Extract fields from one selected page
from spidra import ScrapeParams, ScrapeUrl

job = await client.scrape(ScrapeParams(
    urls=[ScrapeUrl(url=page_url)],
    prompt=("Extract only company facts explicitly stated on this page. "
            "Return null for fields not stated. Do not infer missing facts."),
    schema=SCHEMA, output="json",
), timeout=120.0)
page_data = job.content

Spidra supports multiple URLs in one scrape request. Here, we submit each selected page separately so that fields can be attributed to the particular page that produced them. The SDK's scrape() call waits for the job to finish; you don't need to write a polling loop. See the Spidra documentation for the available scraping options.

A schema controls the shape of extracted data. It does not independently verify a founding date, address, or company identity.

Normalize fields and retain their sources

Company sites don't use a shared vocabulary. One page may say “Fintech” while another says “Financial technology.” Some leave the founding year out entirely. Normalize formatting without pretending those descriptions mean the same thing.

# Normalize a single extracted value
import re

def normalize_field(field: str, value):
    if not isinstance(value, str):
        return None
    value = " ".join(value.strip().split())
    if not value or value.casefold() in {"n/a", "unknown", "null"}:
        return None
    if field == "founded_year":
        return int(value) if re.fullmatch(r"(?:18|19|20)\d{2}", value) else None
    return value

The complete implementation also validates LinkedIn hostnames. For every populated field, it keeps the page URL under sources with a company_website_statement basis. When two selected pages return the same normalized value, both URLs can be retained.

When pages disagree, the example doesn't simply take the first value. It moves the field into conflicts, leaves the profile value as null, and lets a downstream reviewer decide. A single source URL is attribution, not proof that a statement is true; even two pages on the same company website are not independent confirmation.

Return a company profile from /enrich

The FastAPI route joins the steps: validate the request, choose a company domain, select pages, scrape each page, normalize fields, and assemble a response.

The response contains domain, profile, and per-field sources along with selection, pages_requested, missing_fields, conflicts, and failed_pages. It also includes independently_verified: false. A matched status means the pipeline selected a candidate and processed pages without detected conflicts or page failures. It does not mean the company is legally verified or that all fields are complete.

The full, runnable main.py is included in the companion code package. Once you've added it to the project, start the server:

uvicorn main:app --reload

Open http://127.0.0.1:8000/docs to use FastAPI's interactive documentation, or send a request with curl:

curl -s -X POST 'http://127.0.0.1:8000/enrich' \
  -H 'Content-Type: application/json' \
  -d '{"domain":"stripe.com"}' | python -m json.tool

That command prints the JSON returned by your actual request. The precise fields depend on the pages Spidra can access and what those pages publish. Don't substitute a hand-written company profile for output from a real call.

Check ambiguous names and missing-data rates

A useful company enrichment test should include more than an easy domain lookup. The code package includes benchmark.py, which sends these 12 inputs and saves the real HTTP responses to a JSONL file:

Run the benchmark against a local server configured with your Spidra key:

python benchmark.py --output live-enrichment-results.jsonl

The script records the input, HTTP status, selected website, search candidates, extracted fields, and source URLs. It also calculates the missing-data rate for each field among requests that returned a profile. Invalid inputs, ambiguous matches, and failed enrichment requests are counted separately.

A short company name may resolve cleanly on some searches and ambiguously on others. Don't pre-label Mercury or Delta as guaranteed failures. Inspect the candidate list and manually decide whether the selected website is actually the intended company. That review is needed before you can report a correct-selection rate.

The benchmark involves live API calls and may incur usage charges. It is separate from the included unit tests, which use fixtures to check application behavior rather than real company data.

Know when a company website is not enough

Public company websites work well for a description, homepage, or explicitly published social link. They are a weaker foundation for legal entity identity, employee counts, revenue, or email deliverability.

Before taking this beyond a prototype, add authentication, request limits, caching, retry budgets, and audit logs. The hostname filter in the example is not a complete SSRF defense: DNS resolution, redirects, and network egress controls all matter when a server makes requests based on user input. Avoid storing unneeded personal information. Review sites' terms and robots rules before collecting data; our guide to web scraping laws and practices covers the main considerations.

Build an enrichment API that can explain its answers

You now have the building blocks of a company enrichment API in Python: resolve a website when given a name, scrape selected pages against a schema, normalize the result, and return source URLs and missing fields. The response remains useful even when parts of the company profile are unavailable.

If you're building a sales research tool or B2B product, get a Spidra API key and run the sample service against a small set of companies you know. Keep the source links. They're the first thing you'll need when a record looks wrong.