How to scrape Indeed jobs with Python

You can write a 10-line Python scraper for Indeed. Getting those lines to return actual job listings is another story.

Indeed actively restricts automated traffic. A request that works against a simple website can return a 403 error or a security challenge on Indeed, leaving you with no HTML to parse.

That creates a problem if you're collecting job listings for hiring research, salary analysis, or an application that needs up-to-date opportunities. You want the job title, company, location, salary, and posting date, not an afternoon spent debugging browser challenges.

We'll start with a basic Python request to see what happens, then use Spidra to retrieve structured Indeed job listings. From there, we'll collect jobs across multiple cities, remove duplicates, and export the results to CSV. We'll also look at how to retrieve the full description of an individual job.

One important consideration before proceeding: Indeed's Terms of Service restrict automated scraping without permission. Using a third-party scraping API does not grant permission to access or reuse Indeed's data. Make sure your collection and intended use are authorized.

Why scraping Indeed with requests and BeautifulSoup often fails

For a typical website, Python's requests and BeautifulSoup libraries are often enough. You fetch the page, inspect its HTML, find the relevant elements, and extract their text.

Let's see what happens when we try that approach with Indeed.

Install requests:

pip install requests

Create a Python file and request the search results for Python developer positions in Austin, Texas.

# check_indeed.py
import requests

url = "https://www.indeed.com/jobs"

response = requests.get(
    url,
    params={"q": "python developer", "l": "Austin, TX"},
    timeout=20,
)

print(response.status_code)
print(response.headers.get("cf-mitigated"))

In our October 15, 2026 test, this request returned:

403
challenge

The response wasn't an Indeed job search page. It was a security check.

You might get a different response depending on your connection and when you make the request, but the underlying problem is the same: the page you receive isn't necessarily the content you see when browsing Indeed manually.

Adding BeautifulSoup won't fix that. BeautifulSoup parses the HTML you give it, and if that HTML contains a security challenge instead of job listings, there's nothing useful to extract.

A browser automation tool such as Selenium or Playwright can handle JavaScript-rendered pages, but browser automation alone doesn't guarantee access either. Indeed can challenge automated browsers, and even when a scraper works, you'll still need to maintain selectors as the site's HTML changes.

If you already have permission to collect the data, the next question is whether you want to maintain that scraping infrastructure yourself.

For occasional scraping on accessible pages, Python and BeautifulSoup may be sufficient. For this tutorial, we'll take a different approach.

Get structured Indeed job listings with Python and Spidra

Spidra's Indeed scraper supports Indeed search results, individual job listings, company profiles, and salary pages.

For supported Indeed URLs, Spidra can read the data directly instead of using its normal browser-based scraping flow. You provide the URL, and the response contains structured data from the page.

You don't need to inspect Indeed's CSS classes or write selectors to identify every job card.

For this example, you'll need Python 3.9 or later and a Spidra API key. Create an account, generate a key in the dashboard, and install the Python SDK:

pip install spidra

Set your API key as an environment variable. On macOS or Linux:

export SPIDRA_API_KEY="spd_YOUR_API_KEY"

Now create jobs.py and add the following code:

# jobs.py
import os
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

url = "https://www.indeed.com/jobs?q=python+developer&l=Austin%2C+TX"

result = client.scrape(url, output="json")

data = result.content

print(data["jobCount"], "jobs found")
print(data["sponsoredCount"], "sponsored jobs")

The call to client.scrape() submits the request and waits for the result.

There's one detail worth paying attention to: we set output="json" without supplying an extraction prompt.

Normally, Spidra can use AI to transform webpage content into custom structured data. Indeed's supported pages are different because Spidra already recognizes their data structure. Asking for JSON without a prompt returns that structure directly, avoiding the additional AI extraction cost.

Our recorded test of the Austin search returned:

38 jobs found
23 sponsored jobs

The request took approximately eight seconds and used one credit.

That's 38 job listings from one search, with no custom HTML selectors.

The numbers will change as listings are added, removed, or reordered, but the response structure gives us something we can work with in Python.

What data can you extract from an Indeed job listing?

The result.content object contains the search information and a jobs array.

Each job can include its title, employer, location, posting date, salary information, job type, and link to the full posting.

You can inspect the first few results with:

for job in data["jobs"][:3]:
    print({
        "title": job.get("title"),
        "company": job.get("company"),
        "location": job.get("location"),
        "salary": job.get("salaryText"),
        "url": job.get("url"),
    })

We're using .get() rather than indexing every field directly because employers don't always publish the same information.

For example, a job listing might contain a salary range such as $100,000 - $130,000 a year, while another listing has no salary information at all. When a value isn't available, Spidra returns None for that field rather than inventing an answer.

Salary data also includes separate fields such as salaryMin, salaryMax, and salaryPeriod. Keep the period when working with salary numbers, because hourly and annual rates shouldn't be compared as though they're the same thing.

You can find the full list of supported fields in the Indeed extraction documentation.

Separate sponsored jobs from other listings

There's another detail in our result that's easy to overlook.

Of the 38 jobs returned, 23 were sponsored. That's more than half the listings on that search page.

Spidra doesn't remove sponsored results automatically. Instead, every job includes a sponsored field, allowing you to decide what to keep.

To collect only nonsponsored listings:

jobs = [
    job
    for job in data["jobs"]
    if not job.get("sponsored", False)
]

print(f"{len(jobs)} nonsponsored jobs")

For our recorded search, this leaves 15 jobs.

This distinction matters when you're analyzing hiring activity. Sponsored listings reflect advertising placement as well as hiring demand, so you may want to analyze them separately rather than discard them entirely.

For a simple job feed, though, excluding sponsored results might be exactly what you want.

Indeed page two is behind a sign-in

At this point, you have a working scraper, but there's an obvious question.

What if you need more than the roughly 40 jobs returned by one search?

Normally, you would paginate by changing a parameter in the URL. Indeed uses a start parameter, so you might try requesting the next page like this:

page_two = (
    "https://www.indeed.com/jobs"
    "?q=python+developer&l=Austin%2C+TX&start=10"
)

result = client.scrape(page_two, output="json")

In our test, that request failed.

Indeed requires signed-out visitors to sign in before accessing subsequent result pages. Spidra reported that limitation instead of returning fabricated results, and the failed request cost zero credits.

The relevant distinction is that the scraper isn't failing because of a missing CSS selector. Indeed isn't providing the next page to an anonymous visitor.

Adding more requests with different start values won't solve that limitation.

A more useful approach is to narrow the searches themselves.

For example, instead of searching for all Python developer jobs in Texas, you could search separately in Austin, Dallas, Houston, and other cities.

You can also use more specific titles, such as Python backend developer, Django developer, or Senior Python engineer.

For recently posted jobs, Indeed supports the fromage query parameter. Setting fromage=1 requests listings from the past day.

One detail from Spidra's Indeed support is that some jobs outside the requested date window can still appear. These are marked with an outsideWindow field, so you can exclude them when you process the results.

This gives you a way to collect a broader dataset from multiple first-page searches without pretending that the sign-in limitation doesn't exist.

Scrape Indeed jobs from multiple cities

Let's extend our example to collect Python developer jobs from five cities in Texas.

We'll build the search URLs using Python's urlencode() function so that spaces, commas, and other characters are encoded correctly.

from urllib.parse import urlencode

cities = [
    "Austin, TX",
    "Dallas, TX",
    "Houston, TX",
    "San Antonio, TX",
    "Fort Worth, TX",
]

urls = [
    "https://www.indeed.com/jobs?" + urlencode({
        "q": "python developer",
        "l": city,
        "fromage": 1,
    })
    for city in cities
]

Instead of calling client.scrape() five times, we'll use Spidra's Batch scrape feature.

It accepts up to 50 URLs per batch, returning a separate result for each URL.

batch = client.batch_scrape(
    urls,
    output="json",
)

for item in batch.items:
    print(item.url, item.status)

This is slightly different from scraping one page.

Each batch item has its own status and result. The overall batch may finish successfully even if an individual URL failed, so your code needs to inspect the items before using them.

For a successful Indeed search, item.result contains the structured data, including the jobs array.

That brings us to another problem: duplicates.

A company might advertise a remote position across multiple cities, and the same listing could appear in several searches. Counting every result as a separate opening would inflate your dataset.

Fortunately, Indeed job listings have an identifier we can use to remove duplicates.

jobs_by_id = {}

for item in batch.items:
    if item.status != "completed":
        print(f"Failed: {item.url} ({item.error})")
        continue

    for job in item.result.get("jobs", []):
        job_id = job.get("id")
        if job_id:
            jobs_by_id[job_id] = job

jobs = list(jobs_by_id.values())

print(f"Found {len(jobs)} unique jobs")

We're using a dictionary keyed by the job ID. If the same job appears twice, the later result replaces the earlier one rather than creating another record.

In a separate recorded test using five city searches, Spidra returned 182 unique jobs after deduplication, of which 72 were nonsponsored.

The batch cost five credits. It completed in approximately 13 seconds on one run, while another took around 80 seconds.

The difference is useful to know. Scraping time isn't fixed, even when you're collecting from the same source. Your program should allow for that rather than assuming every batch will finish in a few seconds.

Export the Indeed job listings to CSV

Now that we have a list of unique jobs, we can save the data in a format that works with Excel, pandas, or another analysis tool.

Python's built-in csv module is enough for this. We don't need another dependency.

We'll export the job ID, title, company, location, salary, posting date, remote status, and URL.

import csv

fields = [
    "id", "title", "company", "location",
    "postedAt", "salaryText", "salaryMin",
    "salaryMax", "salaryPeriod", "remote",
    "sponsored", "url",
]

with open("indeed_jobs.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=fields)
    writer.writeheader()

    for job in jobs:
        if job.get("sponsored") or job.get("outsideWindow"):
            continue
        writer.writerow({field: job.get(field) for field in fields})

Running this creates an indeed_jobs.csv file in your project directory.

The file contains one row per retained listing, and jobs without published salary information have empty salary cells.

In one of our recorded collection runs, 126 of 184 exported jobs had salary data. That means 58 didn't.

This is a good example of why missing values should stay missing. Replacing them with zero would make a salary analysis inaccurate, while estimating values without distinguishing them from published salaries would create a different problem.

If you're planning to analyze pay across locations, preserve the original salary text and period, then normalize hourly, monthly, and annual rates separately.

The complete Python Indeed scraping script

Here's the entire script in one place.

It builds the search URLs, fetches the results in a batch, skips failed requests, removes duplicate job IDs, filters sponsored and out-of-window listings, and writes the remaining results to CSV.

# jobs.py
import csv
import os
from urllib.parse import urlencode

from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

cities = [
    "Austin, TX",
    "Dallas, TX",
    "Houston, TX",
    "San Antonio, TX",
    "Fort Worth, TX",
]

urls = [
    "https://www.indeed.com/jobs?" + urlencode({
        "q": "python developer",
        "l": city,
        "fromage": 1,
    })
    for city in cities
]

batch = client.batch_scrape(urls, output="json")

jobs_by_id = {}

for item in batch.items:
    if item.status != "completed":
        print(f"Failed: {item.url} ({item.error})")
        continue

    if not isinstance(item.result, dict):
        print(f"Unexpected result: {item.url}")
        continue

    for job in item.result.get("jobs", []):
        if job.get("id"):
            jobs_by_id[job["id"]] = job

jobs = [
    job for job in jobs_by_id.values()
    if not job.get("sponsored")
    and not job.get("outsideWindow")
]

fields = [
    "id", "title", "company", "location",
    "postedAt", "salaryText", "salaryMin",
    "salaryMax", "salaryPeriod", "remote",
    "sponsored", "url",
]

with open("indeed_jobs.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=fields)
    writer.writeheader()
    writer.writerows(
        {field: job.get(field) for field in fields}
        for job in jobs
    )

print(f"Saved {len(jobs)} jobs to indeed_jobs.csv")
print(f"Failed searches: {batch.failed_count}")

Run it with:

python jobs.py

The script will print the number of retained jobs and failed searches. If a search fails, the other results can still be saved.

This is enough for a small job-data collection workflow. If you intend to run it regularly, you'll also want to consider storing the date of each collection and comparing job IDs across runs.

That way, you can distinguish genuinely new postings from listings you've already collected.

Get the full description of an Indeed job

So far, we've been working with search results.

Those results include useful details, but they don't necessarily contain the full job description. If you're analyzing required skills, responsibilities, or qualifications, you'll probably need the individual job pages.

Indeed's job URLs commonly follow this pattern:

https://www.indeed.com/viewjob?jk=JOB_ID

Spidra supports these pages through the same Scrape method.

Assuming you still have the jobs list from the previous example, you can fetch the first listing that has a URL:

job_url = next(
    (job["url"] for job in jobs if job.get("url")),
    None,
)

if job_url:
    result = client.scrape(job_url, output="json")
    job = result.content["job"]

    print(job.get("title"))
    print(job.get("descriptionText"))

The response includes the full job description when Indeed makes it available, along with other job details.

In our recorded single-job test, the descriptionText field contained 1,968 characters. The response also included descriptionMarkdown, salary-related details, and hiring information.

This request is separate from collecting the search results, so it uses another credit.

If you only need titles, companies, dates, and salaries, don't fetch every description unnecessarily. Search results already contain those fields when Indeed provides them.

But if you're studying which skills employers request or building a search experience based on job descriptions, the individual pages give you much more material to work with.

What should you do when an Indeed request fails?

There are two failures worth treating differently.

The first is a request that Indeed won't serve, such as a second search results page that requires sign-in, an expired job, or an unavailable page.

In those situations, Spidra returns an error rather than valid job data. The recorded pagination failure was not charged, and Spidra's documented behavior is that failed jobs cost zero credits.

The second is a successful search that returns fewer results or fields than you expected.

A result containing ten jobs doesn't necessarily mean the scraper missed thirty. The query may be restrictive, job availability may have changed, or Indeed may have returned a smaller results page.

Likewise, an empty salary field doesn't mean extraction failed. The employer may not have published salary information.

Before retrying, inspect the job status, the search query, and the fields actually returned. Blind retries won't fix an intentionally restricted page.

And remember that Spidra returning data doesn't establish permission to collect or reuse it. Our guide to web scraping legality covers the broader issues, but you should also review Indeed's own rules and obtain appropriate authorization.