Structured Output

Here is a situation a lot of people run into. You set up a scrape, the AI does its job, and you get JSON back... but the field names are not what you expected. Or a field you needed is missing because the AI was not confident enough to include it. Or you run the same scrape twice and the shape is slightly different between runs.

The moment you try to push that data into a database or process it downstream, you start writing all sorts of defensive handling just to deal with the inconsistency.

We built structured output to fix that at the source. You tell Spidra exactly what shape you want, and it returns data in that shape, every time.

You do it by passing a schema alongside your prompt. The schema is standard JSON Schema, so if you have used it before it will feel familiar. Here is a real example:

curl --request POST \
  --url https://api.spidra.io/api/scrape \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: YOUR_API_KEY' \
  --data '{
    "urls": [{ "url": "https://jobs.example.com/senior-engineer" }],
    "prompt": "Extract the job details",
    "schema": {
      "type": "object",
      "required": ["title", "company", "remote", "employment_type"],
      "properties": {
        "title":           { "type": "string" },
        "company":         { "type": "string" },
        "location":        { "type": ["string", "null"] },
        "remote":          { "type": ["boolean", "null"] },
        "salary_min":      { "type": ["number", "null"] },
        "salary_max":      { "type": ["number", "null"] },
        "employment_type": {
          "type": ["string", "null"],
          "enum": ["full_time", "part_time", "contract", null]
        },
        "skills": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    }
  }'

You do not need to set "output": "json" manually. When you pass a schema, Spidra sets that automatically.

The result comes back clean, with the exact fields you defined:

{
  "title": "Senior Software Engineer",
  "company": "Acme Corp",
  "location": "Austin, TX",
  "remote": true,
  "salary_min": 140000,
  "salary_max": 180000,
  "employment_type": "full_time",
  "skills": ["Python", "React", "PostgreSQL", "Docker"]
}

The required rule is the most important thing to understand here. Fields you put in required will always be present in the response, even when the AI cannot find them on the page. In that case it writes null rather than leaving the field out.

Fields you leave out of required may be omitted entirely if there is nothing to extract. So if you need a field to always show up in your output, even as null, put it in required. If you are fine with it just being absent when there is no data, leave it out of required and the AI will skip it rather than guess.

Making fields nullable. To let a field be either a value or null, pass the type as an array:

{ "type": ["string", "null"] }

This works for any type: string, number, boolean. It tells the AI it can write null if the page does not have that information.

Spidra validates your schema before running anything. If there is a structural problem, like the root not being type object, or the schema being nested too deeply, you get a 422 back before the job is ever queued (no credits are used).

Structured output also works with batch scraping. Pass a schema in a batch request and every URL in the batch returns data in the same shape.

Read the full docs here.