How to collect and refresh job postings for a job board

A job board needs more than an initial collection of job postings. New roles appear, employers update existing listings, and some positions close. Without a process for handling these changes, the board gradually becomes unreliable.

The challenge is collecting the data repeatedly without introducing duplicates, losing useful information, or treating every missing result as a closed vacancy.

This tutorial shows how to build a Python pipeline that collects structured job data with Spidra, stores listings in SQLite, identifies changes between runs, and monitors new postings.

We'll use software engineering jobs in Austin, Texas, as the example.

How the job data pipeline works

Our pipeline has four main parts:

1. Collect: Search for recent jobs using a structured data source.
2. Clean and store: Keep the fields we need and deduplicate listings using their source IDs.
3. Refresh: Run the same collection again and compare the results with what's already stored.
4. Monitor: Schedule checks for new listings and receive notifications when relevant changes appear.

The important distinction is between collecting jobs and maintaining them. A one-time collection gives us a snapshot. A refresh process records what has changed since that snapshot.

We'll use Spidra's Indeed Job Search and Job Details Templates to demonstrate the collection steps.

Step 1: Collect recent job postings

A raw job search page may include sponsored listings, older posts, and jobs that are less relevant than the search query suggests.

For a job board, it's helpful to narrow the collection by title, location, and posting date. Spidra provides an Indeed Job Search Template that accepts these inputs and returns structured job records.

For our example, we'll search for software engineering jobs in Austin using a seven-day posting window.

Start by creating a Spidra account and generating an API key. Then install the Python requests library:

pip install requests

Store your API key in an environment variable:

export SPIDRA_API_KEY="spd_YOUR_API_KEY"

Define the search configuration:

import os
import requests

API = "https://api.spidra.io/api"
HEADERS = {
    "Authorization": f"Bearer {os.environ['SPIDRA_API_KEY']}"
}

inputs = {
    "title": "software engineer",
    "locations": ["Austin, TX"],
    "postedWithin": "7",
    "site": "www",
    "fullDetails": False,
}

The postedWithin option limits the search by the posting window. It helps narrow the results, although it does not independently confirm that every returned vacancy is still open.

Before running the collection, request an estimate:

response = requests.post(
    f"{API}/templates/indeed-job-search/estimate",
    json={"inputs": inputs},
    timeout=30,
)
response.raise_for_status()

estimate = response.json()

The estimate includes the expected credits for the configured search. Checking estimates is free.

Now run the Template:

response = requests.post(
    f"{API}/templates/indeed-job-search/runs",
    headers=HEADERS,
    json={"inputs": inputs, "wait": True},
    timeout=70,
)
response.raise_for_status()

run = response.json()

For a small collection, the response may already contain the completed rows.

Longer runs return a runId, which you can use with GET /template-runs/{runId} until the status changes from running to completed, partial, or another terminal state.

A completed run includes job records with fields such as:

• id, title, company, and location
• postedAt and url
• salaryText and other available salary fields
• remote and sponsored

Not every employer provides salary information, and some fields may be missing. Preserve those missing values rather than replacing them with estimates.

Also check the run status. A partial result can contain useful rows alongside failed items. Don't discard the successful records simply because another page couldn't be read.

Step 2: Retrieve more details for selected jobs

Job search results are useful for discovery, but they may not contain everything needed for a detailed listing.

For example, a search result might include the job title, company, location, and a short description snippet. The individual job page may contain additional information about responsibilities, qualifications, benefits, or application requirements.

Spidra's indeed-job-details Template accepts a list of job URLs and retrieves additional fields where available.

You can use the url field from the search results:

job_urls = [
    job["url"]
    for job in run.get("rows", [])
    if job.get("url")
][:5]

These URLs become the jobs input for the indeed-job-details Template. You don't have to enrich every result immediately. A better approach is to select the records your application actually needs, retrieve their additional information, and keep the search result as the fallback when a detail page cannot be accessed.

For example, a job market dashboard that only compares hiring activity may not need complete descriptions. A job board displaying detailed vacancies probably will.

Spidra does not charge for pages that fail to return data, but some jobs may remain inaccessible. Treat those records as incomplete rather than assuming another attempt will succeed.

Step 3: Store listings and remove duplicates

When you collect job postings repeatedly, duplicate records become a problem.

The same role can appear in searches for different keywords or locations. A software engineering position might appear in both a general engineering search and a backend developer search.

Comparing job titles isn't sufficient because different employers can publish positions with identical titles. Even the same employer may advertise several openings under one title.

For Indeed data, use the source job ID as the primary identifier. A simple approach is to prefix the ID with the source name:

source_id = f"indeed:{job['id']}"

That helps distinguish IDs if you later introduce other job sources.

For the example, we'll use SQLite to store the listings. Each record contains the job ID, title, company, location, salary, posting date, source URL, and timestamps showing when it was first and last collected.

The table can start with this structure:

CREATE TABLE IF NOT EXISTS jobs (
    source_id TEXT PRIMARY KEY,
    title TEXT,
    company TEXT,
    location TEXT,
    salary_text TEXT,
    posted_at TEXT,
    source_url TEXT,
    first_seen_at TEXT NOT NULL,
    last_seen_at TEXT NOT NULL
);

On the first run, insert each unique source ID. On subsequent runs, look up the ID before inserting. If it already exists, compare the fields you care about and update the stored record when necessary.

The first_seen_at value should remain unchanged, while last_seen_at records the most recent time the listing appeared in a successful collection.

You can keep category membership in a separate table if one job belongs to several categories. That avoids creating multiple copies of the same job.

A source ID works well for deduplicating records within that source, although a reposted vacancy may receive a different ID. Deduplicating across multiple job platforms requires additional matching logic.

Step 4: Detect changes between collections

Once the data is stored, the next step is to identify what changed when the collection runs again.

For each returned job, compare its current values with the version already in the database.

The last case deserves attention. Suppose a job was published seven days ago and your search only returns postings from the past seven days.

Tomorrow, that job might disappear from your search results simply because it is outside the posting window.

That doesn't mean the employer has closed the vacancy.

The same problem can occur when search rankings change, a sponsored listing rotates out, or a relevant job is no longer included on the first results page.

A job missing from a search result is not proof that it has closed. Before removing a published vacancy, check its original job page or an authorized source that provides its current status. If the status cannot be verified, mark it for review rather than automatically deleting it.

This distinction is essential when building a public job board. Your refresh process should help prevent stale listings, not accidentally remove valid opportunities.

Testing the refresh logic

The Python example stores the listings in SQLite and compares their values during subsequent imports.

Its local self-test uses sample records to verify the change detection without making API requests. The output is:

First import: 2 new, 0 changed
Second import: 0 new, 1 changed

In the second import, one record has updated salary information while the other remains unchanged.

This demonstrates the database comparison logic. Your actual collection results will depend on the jobs available when you run the search.

Step 5: Monitor new job postings

You could schedule the Python script with a cron job or another task scheduler and rerun it daily.

However, if you also want notifications when new listings appear, Spidra provides Monitors.

A Monitor reruns a configured Template, preset, or set of pages on a schedule and compares its results with the previous check.

For this example, you can create a Monitor using the same Indeed Job Search inputs:

When the Monitor starts, its first check establishes a baseline. It records the starting data without treating all existing records as new alerts.

Later checks can identify newly appearing records. Depending on your configuration, Spidra can notify you through email, Slack, Discord, or a signed webhook.

You can also configure a Monitor to watch changes to selected fields, such as salary or title. The watchFields option is used with changed-item monitoring to focus on the fields that matter.

For this job board, I'd start by monitoring new listings and handle more complicated updates through the Python synchronization process.

Monitoring and database synchronization are separate jobs

A Monitor detects changes and sends notifications. It does not automatically insert records into your application's database or remove expired vacancies.

If your board needs automatic database updates, you have two options. You can schedule the Python collection script independently and let it update SQLite whenever it runs.

Alternatively, you can configure a Monitor webhook and build a receiver that triggers your application's synchronization process.

The second approach is useful when you want Spidra to detect the changes while your application controls database updates, enrichment, verification, and publishing.

See the Monitors documentation for configuration and scheduling options.

Expanding the job board to more categories and locations

One job search rarely gives a board complete coverage of a market. Indeed's publicly accessible search results are limited to the first page for signed-out visitors, so a single broad query may miss relevant jobs.

Instead, you can run several more specific searches. For example, a software engineering board could collect frontend developer, backend developer, DevOps engineer, and data engineer openings across several cities.

Each query produces another group of results, which can be merged using the source IDs.

The Indeed Job Search Template accepts multiple locations in one run. Each location adds another search operation, so check the estimate before increasing the scope.

The same Template also supports country-specific Indeed sites. If you want to collect results from another country, choose the appropriate site and review the returned listings, availability, and relevant restrictions.

As your board expands, track which searches produce useful, distinct records. Running overlapping queries indefinitely can increase collection work without substantially improving coverage.

Keeping published job listings accurate

A job board's responsibilities don't end when data reaches the database. There are a few things to address before making listings publicly available.

• Keep original posting dates. Don't replace a job's publication date with the date it was collected. Store both values separately so users can distinguish an employer's posting date from your last check.
• Don't invent missing details. Some employers omit salary information or employment requirements. Leave those fields empty or display an appropriate message such as "Salary not disclosed."
• Verify whether a job is still accepting applications. A recent search result isn't a guarantee of an open vacancy. Use a reliable status signal and a review process for uncertain cases.
• Preserve source attribution and follow publishing permissions. Collection and redistribution are different activities. Make sure you have the appropriate rights before publishing third-party listings or descriptions.
• Handle expired listings correctly. If your board uses JobPosting structured data, follow Google's job posting guidelines. Google requires accurate job details and expects expired positions to be removed or marked appropriately. Use validThrough only when an actual expiration date is known, not one inferred from the search window.

For more information about collection requirements, see our guide to web scraping legality.

What this approach does and doesn't solve

This pipeline handles the data collection and update process, but it isn't a complete job board application.

It gives you structured records, a way to avoid duplicate IDs, and a repeatable process for identifying changed values.

You'll still need to decide which jobs your board should display, how to verify expired listings, whether to moderate submissions, and how to handle applications.

It also doesn't provide exhaustive coverage of every job available in a location. The results depend on the sources, queries, access permissions, and available listing information.

If you're collecting jobs for internal hiring intelligence rather than public redistribution, you may not need the publishing layer at all. The same database and change detection approach can support a private research dashboard.

For a simpler starting point that focuses on collecting and exporting listings without building a database, see how to scrape Indeed job listings with Spidra.

Start with a small, reliable collection

You don't need to collect every vacancy in a country to build a useful job data product.

Start with a specific category and location. Collect the listings, store them using stable source identifiers, and confirm that subsequent runs update existing records instead of duplicating them.

Once that process works, introduce additional searches, selected detail-page enrichment, and scheduled monitoring.

Spidra handles the collection and monitoring operations, while your application remains responsible for validating, storing, and publishing the records appropriately.

To try the collection process, open the Indeed Job Search Template, configure a title and location, and check the estimate before running it.