Skip to main content
TutorialsNovember 17, 2026 · 9 min read

How to handle asynchronous scraping jobs in production

Build a Python runner for async scraping jobs with persistent state, safe polling, backoff, timeouts, and recovery after worker restarts.

How to handle asynchronous scraping jobs in production

Async scraping API jobs can outlive the workers that submitted them. A deployment can restart a worker before a result is ready, a status request can fail, or a job can run past your application's waiting deadline. Without a saved job ID and a recovery path, your backend can lose track of work or submit the same scrape again.

This tutorial builds a Python runner that stores Spidra job IDs in SQLite, checks their status with bounded backoff, saves terminal responses, and resumes accepted jobs after a worker restart. The example uses one worker so the recovery behavior is easy to follow. We'll cover what changes for a multi-worker backend at the end.

Separate Spidra's status from your application's status

The Spidra Scrape API accepts a URL request and returns HTTP 202 with a jobId. You then call the Scrape status endpoint until the remote job reaches a terminal state. If you only need to submit a scrape and poll it in one process, the Spidra API tutorial covers that simpler flow.

For a customer-facing backend, also maintain a local state:

Local state

Meaning

queued

The request is recorded locally, or the remote job has not started processing.

running

Spidra reports that the job is active.

completed

A terminal response has been saved locally.

failed

Submission was rejected, or Spidra reported a terminal failure.

timeout

Your application's waiting deadline elapsed.

A Scrape status such as waiting maps to local queued; active maps to local running. Keep the raw provider status separately. A local timeout does not cancel the Spidra job. It only stops your application's wait.

Two exceptional states handle uncertain submissions. submitting means the worker has committed its intention to POST, but has not yet saved a remote ID. If it crashes at that point, the outcome is unknown. submission_unknown means the request may have reached Spidra without a usable job ID returning. Neither state should trigger an automatic replacement POST.

Submit the job and persist its ID

Use Python Requests so the HTTP behavior stays visible. Install the dependency and put your Spidra API key in an environment variable:

python -m pip install requests
export SPIDRA_API_KEY="spd_YOUR_API_KEY"

In Windows PowerShell, use $env:SPIDRA_API_KEY="spd_YOUR_API_KEY" instead of export.

A bare submission is short:

# submit_job.py
import os
import requests

base = "https://api.spidra.io/api"
headers = {"x-api-key": os.environ["SPIDRA_API_KEY"]}
response = requests.post(
    f"{base}/scrape", headers=headers,
    json={"urls": [{"url": "https://books.toscrape.com/"}]},
    timeout=(5, 25),
)
response.raise_for_status()
job_id = response.json()["jobId"]

timeout=(5, 25) bounds connection and read waits for this HTTP request. It does not impose a deadline on the remote scrape. Spidra may still be processing the job after the request returns.

The bare snippet also has a reliability gap: the job ID lives only in memory. In the companion spidra_async_evidence.py, the submit command first creates a local SQLite row, marks it submitting before the POST, then commits the returned jobId and changes its local state to queued before polling begins. A row left in submitting after a crash needs reconciliation.

The relevant table is:

CREATE TABLE jobs (
  id TEXT PRIMARY KEY,
  url TEXT NOT NULL,
  spidra_id TEXT,
  state TEXT NOT NULL,
  remote_status TEXT,
  created_at TEXT NOT NULL,
  deadline_at TEXT NOT NULL,
  updated_at TEXT NOT NULL,
  poll_count INTEGER NOT NULL DEFAULT 0,
  last_error TEXT
);

The local UUID is the identifier your own application can expose to a customer. Keep the provider ID and API key on the server. A production API can return your local ID with HTTP 202 while a separate worker handles the remote request.

Persisting before the POST is necessary, but it doesn't make submission exactly-once. If the worker loses its connection after Spidra accepts the POST but before the remote ID is committed, it cannot safely assume the request failed. The runner keeps submitting after an abrupt crash or sets submission_unknown after an ambiguous network failure or server error. Neither state is resubmitted automatically. Spidra's documented five-second deduplication window is not a durable idempotency guarantee.

Poll the existing job, not the original URL

Once you have committed the remote job ID, status requests are reads. They must use that ID rather than repeat the original submission:

# Check an accepted job
response = requests.get(
    f"{base}/scrape/{job_id}",
    headers=headers,
    timeout=(5, 20),
)
response.raise_for_status()
payload = response.json()
remote_status = payload["status"]

An HTTP 200 can carry a status of failed, so check the response body rather than treating a successful HTTP request as a successful extraction. Continue checking while the job is waiting or active. Stop when it reports completion or failure, and keep the original error for diagnosis.

When a completed response arrives, save it before changing the local row to completed. If the process crashes between those actions, a restarted worker can fetch and save the response again. If you reverse them, a crash can leave a completed record whose result was never stored.

The companion script writes the terminal JSON to a temporary file, flushes it, atomically replaces the output file, and then commits the local terminal state. That gives this single-worker example a clear recovery order. For a product backend, save the result and update the state with an appropriately durable storage design.

Finally, distinguish transport completion from useful data. A remote scrape can finish while its extraction contains missing fields or an ai_extraction_failed fallback. Validate the values your product needs before treating the extracted record as usable.

Back off from transient errors and set a waiting deadline

Polling every fraction of a second wastes requests. A sensible worker checks periodically while work is pending, then backs off further after temporary failures such as HTTP 429, 502, or 503. The Spidra rate-limit documentation explains 429 responses and the available limit headers.

The companion runner uses this backoff function for temporary polling failures:

# Backoff for a temporary polling failure
import random

def retry_delay(failures, retry_after=None):
    delay = min(30, 2 ** min(failures, 4)) + random.uniform(0, 1)
    return max(delay, retry_after) if retry_after is not None else delay

The runner applies that delay after transient failures and introduces a small jitter between ordinary pending checks. It respects a numeric Retry-After value when one is returned. Each HTTP call has its own timeout, and each local job has a persisted UTC deadline that is checked before polling. An HTTP request already in flight can finish after that deadline, so this is a bounded polling policy, not an exact wall-clock cancellation mechanism.

If the deadline has passed when the runner next checks it, the runner marks the local record timeout and stops waiting, while retaining the remote ID. That lets an operator investigate later. It does not mean Spidra has stopped processing the job. For repeated temporary errors, the example pauses a worker session rather than looping forever; another worker session can resume the same record.

Treat errors differently during submission. Retrying a GET for an existing ID is not equivalent to repeating a POST that creates remote work. If submission times out ambiguously, investigate the original request before creating a replacement. If a polling GET returns 404, don't label it a failed scrape automatically: the ID may be wrong or the result may no longer be available. Record the uncertainty and consult the available job logs.

Recover after deliberately interrupting the worker

A worker that remembers jobs only in Python variables has nothing to resume after a restart. Our runner reads its unfinished rows from SQLite and uses their saved remote IDs:

# Find accepted jobs that still need polling
rows = db.execute("""
    SELECT id, spidra_id FROM jobs
    WHERE state IN ('queued', 'running')
      AND spidra_id IS NOT NULL
    ORDER BY created_at
""").fetchall()
for row in rows:
    wait_for_job(db, row["id"])

This deliberately resumes accepted jobs with saved IDs. Rows in submitting or submission_unknown need reconciliation, not automated resubmission. This teaching runner does not dispatch locally queued requests that never reached the remote API; a complete production queue also needs a submission dispatcher that can distinguish safe unsent work from uncertain submissions.

You can reproduce an interrupted worker with the companion script. Run this in a fresh folder so there are no earlier unfinished jobs:

python spidra_async_evidence.py submit "https://books.toscrape.com/"
python spidra_async_evidence.py work --crash-before-poll
python spidra_async_evidence.py work
python spidra_async_evidence.py show

The second command intentionally terminates the process after the job ID has already been committed to disk. The third starts a fresh worker and resumes polling the same remote ID. The worker does not need to observe every intermediate status; the job may already have completed before the first resumed poll.

Distinguish a failed scrape from a local timeout

A remote failure should preserve Spidra's returned error and status response. A local timeout should preserve the last observed status and remote ID. Neither should automatically launch a replacement scrape.

The companion script provides two additional checks. Run these in a fresh folder, or clear earlier local test rows first:

python spidra_async_evidence.py submit "https://this-domain-does-not-exist-spidra-test.invalid"
python spidra_async_evidence.py work
python spidra_async_evidence.py submit "https://books.toscrape.com/" --deadline-seconds 1
python spidra_async_evidence.py work

The .invalid domain is deliberately unusable, but Spidra might reject it during submission or accept the job and later report a failure. Record what the API actually returns. The one-second deadline demonstrates the local timeout path; depending on when the worker starts, it may expire before the first poll. That's a local timing test, not evidence that Spidra ran out of time.

Move from a single worker to a production backend

The SQLite implementation demonstrates recovery of accepted jobs across process restarts. It is not a ready-made distributed queue. If multiple servers read the same rows, they need atomic job claims or leases, plus idempotent final-result writes, so two workers don't publish the same result twice.

For a customer-facing integration, also plan for bounded submission concurrency, safe validation of user-supplied URLs, and authenticated access to each customer's local job ID. Don't expose raw provider errors or extracted private data in public logs. The companion runner redacts remote IDs in its console logs, but its saved JSON responses are unredacted; inspect and sanitize those files before sharing them. Monitor unfinished-job age, local timeouts, failed polls, unknown submissions, and results that cannot be persisted.

Use this polling design only where the API actually creates a background job. Spidra's Scrape and Crawl operations support job-based execution, but not every Spidra operation behaves identically. Map returns inline, while other operations can offer different waiting behavior. Check the specific endpoint's contract before building a polling loop around it.

For a one-off script, an SDK method that waits for completion can be simpler than a custom queue. When you need a customer request to survive application restarts, persist the accepted job ID and resume its status checks from storage. The example shows recovery of accepted jobs without pretending SQLite alone solves every production scheduling problem.

Create a Spidra API key and run the companion worker against a public URL you're permitted to scrape.

Frequently asked questions

No. asyncio lets a running Python process perform nonblocking waits, but it does not preserve job IDs after that process exits. Restart recovery requires external persistence and a worker that reloads unfinished records.

Share this article

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.