Skip to main content
GuidesNovember 18, 2026 · 13 min read

How to build a website-to-JSON API with FastAPI

Build a FastAPI endpoint that takes a URL and the fields you want, returns validated JSON, and answers failures with a clear error.

How to build a website-to-JSON API with FastAPI

Your product needs a data endpoint: send a URL, get the same JSON back every time, whatever the page looks like. This tutorial builds that website-to-JSON API with FastAPI. At the end you'll have a POST /extract endpoint with a server-side API key, a fixed schema, a Pydantic response model, and errors a client can act on.

We'll skip setup details. If you haven't used the Python SDK before, the Spidra Python tutorial covers the install and your API key. The structured output guide explains schemas in depth. Here we only wire them into an API.

Decide the contract before writing code

The caller sends a URL and the fields they want. They get back those fields, validated, or one error shape that always looks the same.

{ "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "fields": ["name", "price"] }

Every failure uses {"error": {"code", "message", "url", "detail"}} and one of these statuses:

Status

code

When

422

invalid_request

The request body is wrong (bad URL, unknown field)

422

url_not_allowed

The URL is private or internal

422

nothing_extracted

The page loaded but none of the fields were on it

502

retrieval_failed

The page could not be retrieved

503

busy, service_unavailable

Too many requests, or a problem on our side

504

timeout

The page took too long

Callers can branch on code and never need to read the message.

Define the request, the schema and the response

Install what you need and set your key:

pip install spidra fastapi uvicorn
export SPIDRA_API_KEY=spd_YOUR_API_KEY

The schema is fixed on the server. Callers don't send field types, they pick names from a list. That keeps the output stable and stops anyone sending a schema you didn't plan for.

# app.py
ProductField = Literal["name", "price", "currency", "in_stock", "sku"]

class ExtractRequest(BaseModel):
    url: HttpUrl
    fields: list[ProductField] = Field(default=ALL_FIELDS, min_length=1)

class Product(BaseModel):
    name: Optional[str] = None
    price: Optional[float] = None
    currency: Optional[str] = None
    in_stock: Optional[bool] = None
    sku: Optional[str] = None

class ExtractResponse(BaseModel):
    url: str
    data: Product

Every Product field is optional because pages don't always show everything. Product also writes the Spidra schema, so the model and the schema can't drift apart:

def build_schema(fields: list[str]) -> dict:
    schema = Product.model_json_schema()
    schema["properties"] = {k: v for k, v in schema["properties"].items() if k in fields}
    return schema

Only the requested fields go into the schema, so Spidra only extracts those.

Call Spidra from the endpoint

The client is created once at startup and reads the key from the environment. The key never appears in a request or a response.

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.spidra = AsyncSpidra(api_key=os.environ["SPIDRA_API_KEY"])
    yield

app = FastAPI(lifespan=lifespan)

The endpoint is async def with AsyncSpidra, so waiting on a page doesn't block the server. scrape() submits the job and polls until it finishes.

@app.post("/extract", response_model=ExtractResponse, response_model_exclude_unset=True)
async def extract(request: ExtractRequest):
    url = str(request.url)
    try:
        result = await app.state.spidra.scrape(
            url, prompt=PROMPT, schema=build_schema(request.fields), timeout=120
        )
    except SpidraError as error:
        raise translate(error, url)
    return to_response(url, request.fields, result)

timeout=120 bounds the wait. str(request.url) is needed because Pydantic v2 HttpUrl is not a string. response_model_exclude_unset=True returns only the fields the caller asked for. We'll write translate and to_response in a moment.

Start the server:

uvicorn app:app

Run it on two different page layouts

The first page is a book on books.toscrape.com.

curl -X POST localhost:8000/extract -H "Content-Type: application/json" \
  -d '{"url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"}'

Response:

{"url":"https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html","data":{"name":"Tipping the Velvet","price":53.74,"currency":"£","in_stock":true,"sku":"90fa61229261140a"}}

The second is a laptop on a different store with a different layout.

curl -X POST localhost:8000/extract -H "Content-Type: application/json" \
  -d '{"url": "https://webscraper.io/test-sites/e-commerce/allinone/product/40"}'

Response:

{"url":"https://webscraper.io/test-sites/e-commerce/allinone/product/40","data":{"name":"Dell XPS 13","price":1281.99,"currency":"$","in_stock":null,"sku":null}}

Same shape, and price is a number both times. The laptop page doesn't show stock or a SKU, so those come back null instead of a guess. That's why the model fields are optional.

To get fewer fields, ask for fewer:

curl -X POST localhost:8000/extract -H "Content-Type: application/json" \
  -d '{"url": "https://webscraper.io/test-sites/e-commerce/allinone/product/40", "fields": ["name", "price"]}'

Response:

{"url":"https://webscraper.io/test-sites/e-commerce/allinone/product/40","data":{"name":"Dell XPS 13","price":1281.99}}

Turn bad requests into one clear error

FastAPI rejects a bad body before your code runs, and before Spidra is called. Its default 422 body is long and doesn't match our other errors, so we reshape it:

@app.exception_handler(RequestValidationError)
async def invalid_request_handler(request: Request, exc: RequestValidationError):
    details = [{"field": ".".join(str(p) for p in e["loc"][1:]), "problem": e["msg"]} for e in exc.errors()]
    body = {"error": {"code": "invalid_request", "message": "The request body is not valid.", "url": None, "detail": details}}
    return JSONResponse(body, status_code=422)

A malformed URL:

curl -X POST localhost:8000/extract -H "Content-Type: application/json" -d '{"url": "not-a-url"}'

Response:

{"error":{"code":"invalid_request","message":"The request body is not valid.","url":null,"detail":[{"field":"url","problem":"Input should be a valid URL, relative URL without a base"}]}}

A field that isn't in the allowlist ("fields": ["name", "rating"]):

{"error":{"code":"invalid_request","message":"The request body is not valid.","url":null,"detail":[{"field":"fields.1","problem":"Input should be 'name', 'price', 'currency', 'in_stock' or 'sku'"}]}}

A scheme that isn't http or https (ftp://example.com/file):

{"error":{"code":"invalid_request","message":"The request body is not valid.","url":null,"detail":[{"field":"url","problem":"URL scheme should be 'http' or 'https'"}]}}

Private addresses pass FastAPI but Spidra refuses them, and the SDK raises SpidraValidationError. We map it to its own code. Sending http://192.168.1.1/:

{"error":{"code":"url_not_allowed","message":"This URL can't be read.","url":"http://192.168.1.1/","detail":"URL 1: private and internal URLs are not allowed"}}

Turn retrieval failures into useful errors

Pages fail: they 404, the domain doesn't exist, or the site blocks the visit. The SDK raises SpidraJobFailedError, and translate turns every Spidra exception into our error shape:

def translate(error: SpidraError, url: str) -> ApiError:
    if isinstance(error, SpidraJobFailedError):
        return ApiError(502, "retrieval_failed", "The page could not be retrieved.", url, str(error))
    if isinstance(error, SpidraTimeoutError):
        return ApiError(504, "timeout", "The page took too long to process.", url)
    if isinstance(error, SpidraValidationError):
        return ApiError(422, "url_not_allowed", "This URL can't be read.", url, "; ".join(error.errors or []))
    # ... rate limits, server errors, account problems: see the full file below

A page that doesn't exist:

curl -X POST localhost:8000/extract -H "Content-Type: application/json" \
  -d '{"url": "https://books.toscrape.com/this-page-does-not-exist"}'

Response:

{"error":{"code":"retrieval_failed","message":"The page could not be retrieved.","url":"https://books.toscrape.com/this-page-does-not-exist","detail":"Failed to scrape https://books.toscrape.com/this-page-does-not-exist: The page was not delivered: the page answered HTTP 404. Nothing was charged. Check the URL."}}

A domain that doesn't exist:

{"error":{"code":"retrieval_failed","message":"The page could not be retrieved.","url":"https://this-domain-does-not-exist-spidra-example.com/","detail":"Failed to scrape https://this-domain-does-not-exist-spidra-example.com/: the page could not be reached (ERR_TUNNEL_CONNECTION_FAILED)"}}

Both are a 502, with Spidra's own explanation in detail. Failed pages aren't charged. We don't turn a 404 into a 404 of our own: the SDK only gives us the message text, and matching on text breaks when the wording changes.

One failure doesn't raise at all. A page can load fine and still hold no product. Spidra returns every field in the schema, all null. Sending that back as a 200 would make the caller's code fail later, so to_response catches it:

def to_response(url: str, fields: list[str], result) -> ExtractResponse:
    content = result.content
    values = {k: content.get(k) for k in fields} if isinstance(content, dict) else {}
    if result.ai_extraction_failed or not any(v is not None for v in values.values()):
        raise ApiError(422, "nothing_extracted", "The page was read, but none of the requested fields were found.", url)
    return ExtractResponse(url=url, data=Product.model_validate(values))

Pointing the endpoint at example.com:

{"error":{"code":"nothing_extracted","message":"The page was read, but none of the requested fields were found.","url":"https://example.com/","detail":null}}

Handle slow pages and busy periods

async def keeps your server responsive while a job runs, and timeout=120 stops a caller waiting forever. On a timeout the SDK raises SpidraTimeoutError, we answer 504, and the job keeps running on Spidra's side.

Some callers shouldn't hold a connection open at all. For them, split the endpoint in two: submit returns a job ID right away, and a second endpoint reports the result.

@app.post("/extract/jobs", status_code=202)
async def start_extract(request: ExtractRequest):
    url = str(request.url)
    try:
        job = await app.state.spidra.start_scrape(url, prompt=PROMPT, schema=build_schema(request.fields))
    except SpidraError as error:
        raise translate(error, url)
    jobs[job.job_id] = (url, request.fields)
    return {"job_id": job.job_id, "status": "processing"}

jobs is a dict in memory here. Use Redis or a database in production, because a restart empties it. The polling endpoint calls get_scrape, returns 202 while the job runs, and 200 with the same body as /extract when it's done (the code is in the full file below).

curl -X POST localhost:8000/extract/jobs -H "Content-Type: application/json" \
  -d '{"url": "https://webscraper.io/test-sites/e-commerce/allinone/product/40", "fields": ["name", "price", "in_stock"]}'

Response:

{"job_id":"d41088f8-57b2-4e9a-b5b4-796a2ec7a72f","status":"processing"}

Polling GET /extract/jobs/d41088f8-57b2-4e9a-b5b4-796a2ec7a72f answers 202 {"job_id":"d41088f8-57b2-4e9a-b5b4-796a2ec7a72f","status":"processing"} while it runs. Once the job is done it answers 200:

{"url":"https://webscraper.io/test-sites/e-commerce/allinone/product/40","data":{"name":"Dell XPS 13","price":1281.99,"in_stock":null}}

Finished jobs stay readable for about an hour. After that Spidra answers not found and the endpoint returns job_not_found.

Spidra limits you to 60 scrape submissions a minute and 10 unfinished scrapes at once. The SDK retries 502, 503 and 504 itself. A 429 on submit is yours to handle, so translate returns 503 busy with a Retry-After header and your callers know when to try again. If your Spidra credits run out or your key is rejected, the caller shouldn't see that. translate logs it for you and returns a plain 503 service_unavailable.

The full app.py

Everything above in one file. Save it and run uvicorn app:app.

# app.py
import logging
import os
from contextlib import asynccontextmanager
from typing import Literal, Optional

from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field, HttpUrl
from spidra import (
    AsyncSpidra,
    SpidraAuthenticationError,
    SpidraError,
    SpidraForbiddenError,
    SpidraJobFailedError,
    SpidraNotFoundError,
    SpidraPaymentRequiredError,
    SpidraRateLimitError,
    SpidraServerError,
    SpidraTimeoutError,
    SpidraValidationError,
)

log = logging.getLogger("extract-api")
ProductField = Literal["name", "price", "currency", "in_stock", "sku"]
ALL_FIELDS = list(ProductField.__args__)


# --- Models: the request, the fixed schema, the response -------------------
class ExtractRequest(BaseModel):
    url: HttpUrl
    fields: list[ProductField] = Field(default=ALL_FIELDS, min_length=1)


class Product(BaseModel):
    name: Optional[str] = None
    price: Optional[float] = None
    currency: Optional[str] = None
    in_stock: Optional[bool] = None
    sku: Optional[str] = None


class ExtractResponse(BaseModel):
    url: str
    data: Product


def build_schema(fields: list[str]) -> dict:
    schema = Product.model_json_schema()
    schema["properties"] = {k: v for k, v in schema["properties"].items() if k in fields}
    return schema


PROMPT = "Extract the product on this page. Use null for anything the page does not show."


# --- One error shape for every failure -------------------------------------
class ApiError(Exception):
    def __init__(self, status, code, message, url=None, detail=None, headers=None):
        self.status, self.headers = status, headers
        self.body = {"error": {"code": code, "message": message, "url": url, "detail": detail}}


def translate(error: SpidraError, url: str) -> ApiError:
    if isinstance(error, SpidraJobFailedError):
        return ApiError(502, "retrieval_failed", "The page could not be retrieved.", url, str(error))
    if isinstance(error, SpidraTimeoutError):
        return ApiError(504, "timeout", "The page took too long to process.", url)
    if isinstance(error, SpidraValidationError):
        return ApiError(422, "url_not_allowed", "This URL can't be read.", url, "; ".join(error.errors or []))
    if isinstance(error, SpidraRateLimitError):
        wait = int(error.retry_after or 30)
        return ApiError(503, "busy", "Too many requests right now. Try again shortly.", url, headers={"Retry-After": str(wait)})
    if isinstance(error, SpidraNotFoundError):
        return ApiError(404, "job_not_found", "No such job. Finished jobs expire after about an hour.", url)
    if isinstance(error, (SpidraAuthenticationError, SpidraForbiddenError, SpidraPaymentRequiredError)):
        log.error("Spidra account problem: %s", error)  # ours to fix, not the caller's
        return ApiError(503, "service_unavailable", "The extraction service is unavailable.", url)
    if isinstance(error, SpidraServerError):
        return ApiError(503, "service_unavailable", "The extraction service is busy. Try again shortly.", url)
    return ApiError(502, "upstream_error", "Extraction failed.", url)


def to_response(url: str, fields: list[str], result) -> ExtractResponse:
    content = result.content
    values = {k: content.get(k) for k in fields} if isinstance(content, dict) else {}
    if result.ai_extraction_failed or not any(v is not None for v in values.values()):
        raise ApiError(422, "nothing_extracted", "The page was read, but none of the requested fields were found.", url)
    return ExtractResponse(url=url, data=Product.model_validate(values))


# --- App ---------------------------------------------------------------------
@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.spidra = AsyncSpidra(api_key=os.environ["SPIDRA_API_KEY"])  # server-side only
    yield


app = FastAPI(lifespan=lifespan)


@app.exception_handler(ApiError)
async def api_error_handler(request: Request, exc: ApiError):
    return JSONResponse(exc.body, status_code=exc.status, headers=exc.headers)


@app.exception_handler(RequestValidationError)
async def invalid_request_handler(request: Request, exc: RequestValidationError):
    details = [{"field": ".".join(str(p) for p in e["loc"][1:]), "problem": e["msg"]} for e in exc.errors()]
    body = {"error": {"code": "invalid_request", "message": "The request body is not valid.", "url": None, "detail": details}}
    return JSONResponse(body, status_code=422)


@app.post("/extract", response_model=ExtractResponse, response_model_exclude_unset=True)
async def extract(request: ExtractRequest):
    url = str(request.url)
    try:
        result = await app.state.spidra.scrape(
            url, prompt=PROMPT, schema=build_schema(request.fields), timeout=120
        )
    except SpidraError as error:
        raise translate(error, url)
    return to_response(url, request.fields, result)


# --- For slow pages: submit now, poll later ---------------------------------
jobs: dict[str, tuple[str, list[str]]] = {}  # job_id -> (url, fields). Use Redis or a database in production.


@app.post("/extract/jobs", status_code=202)
async def start_extract(request: ExtractRequest):
    url = str(request.url)
    try:
        job = await app.state.spidra.start_scrape(url, prompt=PROMPT, schema=build_schema(request.fields))
    except SpidraError as error:
        raise translate(error, url)
    jobs[job.job_id] = (url, request.fields)
    return {"job_id": job.job_id, "status": "processing"}


@app.get("/extract/jobs/{job_id}", response_model=None)
async def get_extract(job_id: str):
    if job_id not in jobs:
        raise ApiError(404, "job_not_found", "No such job.")
    url, fields = jobs[job_id]
    try:
        job = await app.state.spidra.get_scrape(job_id)
    except SpidraError as error:
        raise translate(error, url)
    if job.status == "failed":
        raise translate(SpidraJobFailedError(job.error, job_id=job_id), url)
    if job.status != "completed":
        return JSONResponse({"job_id": job_id, "status": "processing"}, status_code=202)
    return to_response(url, fields, job.result).model_dump(exclude_unset=True)

FastAPI also serves interactive docs for the endpoint at /docs, generated from the same models.

What this doesn't handle

  • Blocked pages. Some sites block automated visits even with a real browser. Those come back as retrieval_failed. Stealth Mode helps with many of them, not all.
  • Pages behind a login. Spidra can use your own session cookies. You'd add that parameter to the endpoint.
  • More than one kind of page. One fixed schema fits one kind of page. Articles or job posts each need their own model and their own route.
  • Hidden data. The AI reads what the page shows. Values buried in HTML attributes, like a star rating drawn with a CSS class, come back null.
  • Your own endpoint's protection. There is no authentication, caching or rate limiting on /extract. Add them before you expose it, and remember that callers choose the URL. You're responsible for the terms of the sites they point at, and our legal guide is a good place to start.
  • When you don't need any of this. If you only ever read one site with stable HTML, a requests call and a CSS selector is cheaper and enough. This approach earns its place when the pages vary or the site is one you don't control.

Conclusion

You built POST /extract: a URL and a field list in, validated JSON out, one error shape for everything that can go wrong. Two stores with different layouts returned the same fields, and pages that failed told your callers why.

The endpoint is short because of what it doesn't contain. There is no browser to launch, no proxy pool to rent, no CAPTCHA solver to wire in and no selector to rewrite when a store changes its markup. Spidra opens real browsers, routes through proxies, solves CAPTCHAs and runs the AI extraction, and you describe the result as a Pydantic model. When a page can't be delivered, the job fails and costs nothing, so a bad URL from a caller doesn't cost you anything.

That is what turns a one-off extraction into a feature you can ship. The same API key covers the rest of the product. Batch scrape reads up to 50 URLs in one request. Map and Crawl find the pages for you. Parse reads PDFs and Office files. Over 90 ready-made templates return a table for common jobs, and monitors alert you when a price or a page changes. Every plan has every feature. Plans differ in credits, proxy bandwidth, pages per crawl and monitors, so the tool you prototype with is the one you run in production. Official SDKs in ten languages and an MCP server cover the rest of your stack, whatever it's written in.

Create a free account at app.spidra.io, paste your key into SPIDRA_API_KEY, and run app.py against your own product URLs. The Scrape page's Code button in the dashboard writes the matching request for any page you try. Read more about the AI scraper, and the Python SDK docs list every option.

Frequently asked questions

Use async def with AsyncSpidra. A scrape spends most of its time waiting on the page, and an async endpoint lets the server handle other requests during that wait. A plain def endpoint with the sync Spidra client also works, because FastAPI runs it in a thread pool, but you'll run out of threads sooner under load.

Share this article

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.