How to process a batch of PDFs with Python

A folder of PDFs is easy to parse once. The trouble starts when it's a folder that fills up every week, and one file is password-protected, another is a scan, and a third turns out not to be a PDF at all.

Here's how to process a batch of PDFs with Python: estimate what every file will cost before you spend anything, send the readable ones as one batch, deal with each file's result or failure on its own, notice when a file never comes back, and write a manifest so next week's run knows what already happened. It's for Python engineers with a recurring pile of reports, invoices or research papers. If you have a single PDF, our single-document guide is the shorter path, and the Parse docs list every option.

Our test folder has ten files, and three are broken on purpose

Real folders are messy, so ours is too. We generated ten files with a short script (so there's no private data in them): five one-page text invoices, a 12-page quarterly report, one invoice that's only a picture, and three files that should fail. Then we added a public link, the 15-page "Attention Is All You Need" paper, to show that links and files can share a batch.

The files that can be read add up to 33 pages with the paper (5 + 12 + 1 + 15), so we expected about 33 credits for plain Markdown before any fields.

A loop of single parses works until the first bad file

You'll need pip install spidra pydantic and a folder called inbox with some PDFs in it (ours holds the ten files above). The first attempt is the one everyone writes: loop over the folder and call parse() on each file.

# loop.py
import os
from pathlib import Path
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

for path in sorted(Path("inbox").glob("*.pdf")):
    doc = client.parse(str(path))
    print(path.name, doc.total_pages, "pages,", doc.credits_used, "credits")

parse() waits for one document and raises if it can't be parsed. Sorted by name, our folder starts with the damaged invoice, so the loop never reaches the other nine. Here's what each of the three bad files raises:

# ... client as before
from spidra import SpidraError

for name in ("protected-invoice.pdf", "damaged-invoice.pdf", "not-a-pdf.pdf"):
    try:
        client.parse(f"inbox/{name}")
    except SpidraError as e:
        print(name, "->", type(e).__name__, "| status:", e.status, "|", e)

Output:

protected-invoice.pdf -> SpidraValidationError | status: 422 | This PDF is password protected. Remove the password and try again.
damaged-invoice.pdf -> SpidraServerError | status: 500 | The document couldn't be prepared for parsing.
not-a-pdf.pdf -> SpidraValidationError | status: 422 | This PDF couldn't be read. It may be damaged or not really a PDF.

Three files, two different exception classes, and the damaged one is a 500, not a 422. Catch the base class, SpidraError, and don't assume a 5xx means "try again later". Nothing was charged for any of them.

A try/except keeps the loop alive, but it still waits for one file at a time: the five text invoices took 4.3 seconds that way. That's the real difference from single-document parsing. One call means one document and one exception. A batch is many documents, and as we found out, you have to count them.

Estimate every file before you spend anything

Parse can tell you what a file will cost without parsing it. estimate_parse() returns the page count and the credits, charges nothing, and keeps the file for 24 hours, so you can parse it later by its file_id without uploading it again. Start a new file:

# batch_parse.py
import csv
import hashlib
import json
import os
import re
import time
from collections import Counter
from pathlib import Path
from typing import Optional

from pydantic import BaseModel, ValidationError
from spidra import Spidra, SpidraError

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
INBOX, OUT = Path("inbox"), Path("out")
LINKS = ["https://arxiv.org/pdf/1706.03762"]
BUDGET = 100      # credits you're willing to spend on one run
BATCH_SIZE = 50   # documents per batch request
FIELDS = ["file", "status", "pages", "credits", "seconds", "ocr_pages", "check", "error", "markdown", "json", "sha256"]

Then the estimate. A file the estimate can't read is turned away (the manifest calls it rejected) and kept apart, so it never reaches the batch:

def label(item):
    return item.name if isinstance(item, Path) else item


def estimate(items):
    """Free pre-flight. Returns the estimates and the files that were turned away."""
    ready, rejected = {}, {}
    for item in items:
        try:
            ready[item] = client.estimate_parse(str(item))
        except SpidraError as e:
            rejected[item] = f"{e.status}: {e}"
    return ready, rejected

Here's what the estimates said for our folder:

That's 33 credits, exactly the page count we predicted, and the link got a file_id like any uploaded file. All three broken files were stopped here, each with its own message:

damaged-invoice.pdf      500: The document couldn't be prepared for parsing.
not-a-pdf.pdf            422: This PDF couldn't be read. It may be damaged or not really a PDF.
protected-invoice.pdf    422: This PDF is password protected. Remove the password and try again.

Add it up before you commit. The budget is a plain constant, and if the estimate is over it, the script stops before parsing anything:

def is_invoice(item):
    return "invoice" in label(item).lower()


def gate(items):
    """Estimate everything first. Stop before spending if the run is over budget."""
    ready, rejected = estimate(items)
    pages = sum(e.credits for e in ready.values())
    fields = sum(e.credits for item, e in ready.items() if is_invoice(item))  # the fields pass reads these pages again
    print(len(items), "to do |", len(ready), "estimated |", len(rejected), "turned away |", pages + fields, "credits, before AI tokens")
    if pages + fields > BUDGET:
        raise SystemExit(f"Over the budget of {BUDGET} credits. Nothing was parsed.")
    return ready, rejected

Output:

11 to do | 8 estimated | 3 turned away | 39 credits, before AI tokens

That's the 33 pages plus the 6 invoice pages that the fields pass will read again. The estimate counts pages, and pulling fields out of the invoices adds AI tokens on top.

Send what's left as one batch, then count what comes back

Now the batch. We pass only file_ids, so nothing is uploaded twice:

def parse_all(file_ids, **options):
    """One batch per 50 documents. A failed document comes back as a result, not an exception."""
    docs, started = [], time.perf_counter()
    for i in range(0, len(file_ids), BATCH_SIZE):
        batch = client.parse_batch([], file_ids=file_ids[i:i + BATCH_SIZE], **options)
        print(" ", batch.counts, "|", batch.credits_used, "credits")
        docs += batch.documents
    print(f"  {len(docs)} documents in {time.perf_counter() - started:.1f} s")
    return docs

parse_batch() waits for the whole batch, polling for you, then hands back the counts, the credits and one result per document. A batch holds up to 50 documents, so the loop sends a bigger folder in chunks. Any options you pass apply to every document in it, which is why the invoices get their own pass later. This is what the Markdown pass printed:

Markdown pass
  {'queued': 0, 'running': 0, 'completed': 7, 'failed': 0} | 21 credits
  7 documents in 5.7 s

Eight file IDs went in. Seven results came back. The missing one was the 12-page quarterly report, failed was 0, and there was no error and no warning. The report wasn't charged either: 21 credits is the 33 we estimated minus its 12.

We saw the same thing in a smaller test, the five text invoices sent as one batch:

# ... client as before
five = [f"inbox/invoice-00{i}.pdf" for i in range(1, 6)]
started = time.perf_counter()
batch = client.parse_batch(five)
print(f"{time.perf_counter() - started:.1f} s | counts {batch.counts}")
print([d.file_name for d in batch.documents])

Output:

1.2 s | counts {'queued': 0, 'running': 0, 'completed': 4, 'failed': 0}
['invoice-001.pdf', 'invoice-004.pdf', 'invoice-002.pdf', 'invoice-003.pdf']

Five files in, four documents out, 4 credits, and not in the order we sent them. So don't read 1.2 seconds against the loop's 4.3 as a win: the batch did less work. Two lessons. Match results to files by file_id, never by position. And count what you get back.

A batch can come back short, so check it against what you sent

Our first version of this script didn't count. It printed 7 documents and carried on, and the quarterly report was simply never in the manifest. It happened in every batch that held more than one document: 4 of 5, 7 of 8 and, as we'll see next, 4 of 6. We couldn't find a pattern. It wasn't always the last file, and it wasn't always the biggest.

Every result becomes a row in the manifest, matched to its file by file_id:

def new_row(name, doc=None, sha="", status="", error=""):
    meta = (doc.metadata or {}) if doc else {}
    return {
        "file": name, "status": doc.status if doc else status,
        "pages": doc.pages_parsed if doc else "", "credits": doc.credits_used if doc else 0,
        "seconds": round((doc.duration_ms or 0) / 1000, 1) if doc else 0,
        "ocr_pages": meta.get("ocrPages", ""), "check": "", "error": (doc.error if doc else error) or "",
        "markdown": "", "json": "", "sha256": sha,
    }


def safe(name, suffix):
    return OUT / (re.sub(r"[^A-Za-z0-9._-]", "_", name) + suffix)

Every file we sent has to be accounted for. If a file ID has no result, the script says so and records the file as missing, which the next run treats as unfinished:

def find_missing(file_ids, docs, names, rows):
    """A batch can return fewer results than you sent. Say so, and make the next run retry."""
    answered = {doc.file_id for doc in docs}
    for fid in file_ids:
        if fid not in answered:
            name = label(names[fid])
            print("  no result for", name)
            row = rows.get(name) or new_row(name)
            row["status"], row["error"] = "missing", "the batch returned no result for this file"
            rows[name] = row

The Markdown pass ties it together:

def markdown_pass(names, rows):
    print("Markdown pass")
    docs = parse_all(list(names))
    find_missing(list(names), docs, names, rows)
    for doc in docs:
        item = names.get(doc.file_id)
        name = label(item) if item else (doc.file_name or doc.job_id)
        rows[name] = new_row(name, doc, sha256(item) if isinstance(item, Path) else "")
        if doc.status == "completed":
            path = safe(name, ".md")
            path.write_text(doc.markdown or "", encoding="utf-8")
            rows[name]["markdown"] = str(path)

A document that does reach the batch and then fails comes back as its own result, with status set to failed and an error, and the other documents carry on. That's how the Parse docs describe it. None of our files got that far, because the free estimate had already turned the bad ones away, so we can't show you one. new_row handles it anyway.

Markdown to read, typed fields to compute with

Markdown is for people and search. This is what invoice-001.pdf came back as:

# INVOICE INV-1001

From: Northwind Supplies Date: 2026-09-04

|Consulting, 4 hours|330.00|
|---|---|
|Hosting, monthly|49.00|
|Setup fee|75.50|
|Subtotal|454.50|
|Tax (20%)|90.90|

## Total due 545.40

For invoices you want fields a program can add up. Describe them with a schema, in two pieces:

LINE_ITEM = {
    "type": "object",
    "required": ["description", "amount"],
    "properties": {"description": {"type": "string"}, "amount": {"type": "number"}},
}

INVOICE_SCHEMA = {
    "type": "object",
    "required": ["invoice_number", "total"],
    "properties": {
        "invoice_number": {"type": "string"},
        "invoice_date": {"type": ["string", "null"]},
        "vendor": {"type": ["string", "null"]},
        "line_items": {"type": "array", "items": LINE_ITEM},
        "tax": {"type": ["number", "null"]},
        "total": {"type": "number"},
    },
}

Then validate what comes back, because "valid JSON" isn't the same as "right":

class LineItem(BaseModel):
    description: str
    amount: float


class Invoice(BaseModel):
    invoice_number: str
    invoice_date: Optional[str] = None
    vendor: Optional[str] = None
    line_items: list[LineItem] = []
    tax: Optional[float] = None
    total: float

def check_invoice(data):
    try:
        invoice = Invoice.model_validate(data)
    except ValidationError:
        return "invalid fields"
    computed = sum(i.amount for i in invoice.line_items) + (invoice.tax or 0)
    return "ok" if abs(computed - invoice.total) < 0.01 else f"totals differ: {computed:.2f} vs {invoice.total:.2f}"

The check adds up the line items and the tax and compares that with the total. A mismatch means an amount was misread, and you'd rather learn that from a flag than from a customer.

The fields pass sends only the invoices, and only the ones whose Markdown pass completed. The IDs from the first pass still work for 24 hours, so there's no second upload:

def store_fields(row, name, doc):
    row["credits"] += doc.credits_used
    if doc.status == "completed" and isinstance(doc.json, dict):
        path = safe(name, ".json")
        path.write_text(json.dumps(doc.json, indent=2, ensure_ascii=False), encoding="utf-8")
        row["json"], row["check"] = str(path), check_invoice(doc.json)
    else:
        row["check"] = doc.error or "no fields returned"

def fields_pass(names, rows):
    ids = [fid for fid, item in names.items() if is_invoice(item) and rows.get(label(item), {}).get("status") == "completed"]
    print("Invoice fields pass")
    docs = parse_all(ids, prompt="Extract the invoice fields", schema=INVOICE_SCHEMA)
    find_missing(ids, docs, names, rows)
    for doc in docs:
        name = label(names[doc.file_id]) if doc.file_id in names else doc.file_name
        if name in rows:
            store_fields(rows[name], name, doc)
        else:
            print("  result for an unknown file:", doc.file_id, doc.file_name)

We sent six invoices (the five text ones and the scan). This is what the first version printed:

Invoice fields pass
  {'queued': 0, 'running': 0, 'completed': 4, 'failed': 0} | 8 credits
  4 documents in 3.6 s

Four came back, at 2 credits each: the page plus the AI tokens. Invoices 004 and 005 were the two that didn't. The scan did come back, and it's the interesting one, because it has no text layer and AI had to read the page:

{
  "invoice_number": "INV-2001",
  "invoice_date": "2026-09-18",
  "vendor": "Redwood Logistics",
  "line_items": [
    {
      "description": "Consulting, 6 hours",
      "amount": 540
    },
    {
      "description": "Hosting, monthly",
      "amount": 49
    }
  ],
  "tax": 117.8,
  "total": 706.8
}

Every field matches the page, and the totals check passes. The manifest records the scan with ocr_pages set to 1, so you can see which files needed AI. (Dates come back as written on the invoice. Ours are already ISO, but yours might not be, so convert them yourself.)

The manifest tells next week's run what already happened

A recurring job needs memory. The manifest is one CSV row per file: status, pages, credits, seconds, the totals check, the error, the output paths and a SHA-256 of the file. That last column is how the next run knows what to skip:

def sha256(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            digest.update(chunk)
    return digest.hexdigest()


def read_manifest():
    if not Path("manifest.csv").exists():
        return {}
    with open("manifest.csv", newline="", encoding="utf-8") as f:
        return {row["file"]: row for row in csv.DictReader(f)}

def todo(previous):
    """New or changed files, and links not done yet. A link has no file to hash, so delete its row to refresh it."""
    items = []
    for path in sorted(INBOX.glob("*.pdf")):
        old = previous.get(path.name)
        if old and old["status"] == "completed" and old["sha256"] == sha256(path):
            continue
        items.append(path)
    return items + [u for u in LINKS if previous.get(u, {}).get("status") != "completed"]

A file that completed and hasn't changed is skipped without being uploaded. A file that failed, went missing or was turned away is tried again, and the ones turned away cost nothing. A link has no file to hash, so delete its row to refresh it. The rest of the script writes the manifest and ties the steps together:

def write_manifest(rows):
    with open("manifest.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(sorted(rows.values(), key=lambda r: r["file"]))
    print("Wrote manifest.csv:", dict(Counter(r["status"] for r in rows.values())))

def main():
    OUT.mkdir(exist_ok=True)
    rows = read_manifest()
    ready, rejected = gate(todo(rows))
    for item, why in rejected.items():
        rows[label(item)] = new_row(label(item), status="rejected", error=why)
    names = {e.file_id: item for item, e in ready.items()}
    markdown_pass(names, rows)
    fields_pass(names, rows)
    write_manifest(rows)

This is the manifest our first version left after three runs (the error, output path and SHA-256 columns are left out here):

file,status,pages,credits,seconds,ocr_pages,check
damaged-invoice.pdf,rejected,,0,0,,
https://arxiv.org/pdf/1706.03762,completed,15,15,1.4,0,
invoice-001.pdf,completed,1,3,0.1,0,ok
invoice-002.pdf,completed,1,3,0.2,0,ok
invoice-003.pdf,completed,1,3,0.1,0,ok
invoice-004.pdf,completed,1,1,0.2,0,
invoice-005.pdf,completed,1,1,0.1,0,
invoice-006.pdf,completed,1,3,0.1,0,ok
not-a-pdf.pdf,rejected,,0,0,,
protected-invoice.pdf,rejected,,0,0,,
quarterly-report.pdf,completed,12,12,0.1,0,
scanned-invoice.pdf,completed,1,3,2.5,1,ok

Look at invoice-004.pdf and invoice-005.pdf. They're completed, but the check column is empty and they cost 1 credit, not 3, because their fields never came back and the first version marked them done anyway. They're unchanged, so no later run will ever touch them. That's the hole the missing status closes: with it, those two rows would read missing and the next run would retry them.

The quarterly report is only in this manifest because of the second run:

4 to do | 1 estimated | 3 turned away | 12 credits, before AI tokens
Markdown pass
  {'queued': 0, 'running': 0, 'completed': 1, 'failed': 0} | 12 credits
  1 documents in 1.1 s
Invoice fields pass
  0 documents in 0.0 s
Wrote manifest.csv: {'rejected': 3, 'completed': 8}

Nothing had changed on disk, and it still parsed one file: the report the first run never got a result for. It had no row, so it was still on the to-do list. The three broken files were estimated and turned away again, which costs nothing. The run took 3.5 seconds and 12 credits.

For the third run we added one new invoice, and only that file was parsed:

4 to do | 1 estimated | 3 turned away | 2 credits, before AI tokens
Markdown pass
  {'queued': 0, 'running': 0, 'completed': 1, 'failed': 0} | 1 credits
  1 documents in 1.0 s
Invoice fields pass
  {'queued': 0, 'running': 0, 'completed': 1, 'failed': 0} | 2 credits
  1 documents in 3.6 s
Wrote manifest.csv: {'rejected': 3, 'completed': 9}

That's 3 credits (1 for the page, 2 for the fields) and 7.2 seconds.

How close the estimate got

The estimate said 33 credits for Markdown. The first run charged 21, because the report didn't come back, and the second run charged 12 for it. 21 + 12 = 33, to the credit.

The text invoices took 0.1 to 0.2 seconds each to parse, the 15-page paper 1.4, and the scan 2.5, because AI had to read it. The scan cost the same single page credit as the text pages. Everything we ran for this article, experiments included, cost 53 credits. How credits work has the rule.

When a plain Python loop is enough

If every file is a text PDF and you only need the text, you don't need any of this. (Licences below come from each package's PyPI page, checked in October 2026.) pypdf and pdfplumber read text layers for free, under BSD and MIT licences. PyMuPDF describes itself as high performance, but its PyPI listing says it's dual-licensed, AGPL or a commercial licence, which matters for commercial work. Docling parses PDFs, Word and HTML files into one document format, and its convert_all() has a raises_on_error=False option for carrying on past files that fail. Scans need OCR on top, which means installing Tesseract.

Parse earns its credits when the folder is mixed: scans next to text PDFs, Word, Excel and PowerPoint files, uploads that aren't on the web, and a cost you want to see first. It has limits too. Complex tables, multi-column pages and handwriting can come out imperfect, and a scan is only as good as the scan. parse_batch() blocks until the batch finishes, so for a huge folder you want to submit and check on later, call the REST API instead. There's no Node method for Parse either.

Uploaded files stay for 24 hours, so delete the sensitive ones

Parse deletes uploaded files after 24 hours. If the documents are sensitive, delete them yourself once you're done:

# ... client as before
est = client.estimate_parse("inbox/invoice-001.pdf")
print(client.delete_parse_file(est.file_id))
client.parse(file_id=est.file_id)

Output:

{'status': 'deleted', 'fileId': '3ad224ed-…'},

The last line raises a 404: "No file with that fileId. Files can be parsed again for 24 hours after upload."

Run it on a folder you already know

The best test is a folder where you know which files are bad. Create a free account at app.spidra.io (300 credits, no card) and run batch_parse.py on a folder of your own, with BUDGET set low the first time. The script stops before parsing if the estimate is over it. Then read the manifest: every rejected or missing row is a file you'd otherwise never have noticed. The Python SDK docs cover the options we skipped, and the JSON schema generator helps with the schema.