Python Requests vs Playwright vs a scraping API: Which should you use?

Python gives you several ways to collect data from websites. You can fetch a page with Requests, open it in a browser using Playwright, or use a scraping API that handles retrieval and extraction for you.

The right choice depends on where the data lives, whether the page requires interaction, and how often your application needs to collect it.

To compare Requests vs Playwright vs a scraping API, we'll work with three real websites: the Python.org job board, Package Pulse's npm download statistics, and Our World in Data's life expectancy dashboard. We'll look at the Python implementations, the data returned, and where each approach makes sense.

Requests vs Playwright vs a scraping API: The short answer

Start with Requests when the information you need is available in a normal HTTP response. It's lightweight, straightforward, and works well with Beautiful Soup for extracting information from HTML.

Use Playwright when the page needs a browser to execute JavaScript, interact with elements, or display the information you're collecting. It gives you control over navigation, clicks, forms, and the rendered page.

Consider a scraping API when you need those capabilities but don't want to maintain browser infrastructure, retrieval logic, and extraction workflows yourself. APIs such as Spidra can return page content or structured data without requiring you to manage browser instances.

Here's how they compare:

These are architectural differences, not benchmark results. A browser-based solution isn't automatically more accurate, and a scraping API doesn't guarantee that every site will work.

The best way to choose is to start with the data you actually need.

Extract job listings from Python.org using Requests

The Python.org job board publishes real Python-related positions, including job titles, companies, locations, posting dates, and individual job URLs.

It's a good example of a website that doesn't require a browser just to access its listings. The information is present in the server-rendered HTML.

We'll extract the first five positions using Requests and Beautiful Soup.

Install the dependencies:

pip install requests beautifulsoup4

Requests retrieves the HTML, while Beautiful Soup gives us a convenient way to find the listing elements.

# jobs_requests.py
import json
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

URL = "https://www.python.org/jobs/"

def extract_jobs(html):
    soup = BeautifulSoup(html, "html.parser")
    jobs = []

    for card in soup.select("ol.list-recent-jobs > li")[:5]:
        header = card.select_one(".listing-company-name")
        link = header.select_one("a") if header else None
        if not link:
            continue

        title = link.get_text(" ", strip=True)
        company_parts = [
            text for text in header.stripped_strings
            if text not in {title, "New"}
        ]
        location = card.select_one(".listing-location")
        posted = card.select_one(".listing-posted time")

        jobs.append({
            "title": title,
            "company": company_parts[-1] if company_parts else None,
            "location": location.get_text(" ", strip=True) if location else None,
            "posted_date": posted.get_text(" ", strip=True) if posted else None,
            "url": urljoin(URL, link["href"]),
        })

    return jobs

response = requests.get(URL, timeout=20)
response.raise_for_status()

jobs = extract_jobs(response.text)
print(json.dumps(jobs[:2], indent=2, ensure_ascii=False))

The script targets the job board's listing elements and converts relative links into absolute URLs.

The page includes an AI Engineer opening at webbyfox consultancy in London, posted on October 23, 2026. That record is a useful reference when checking the extracted fields.

For this particular job, Requests is a sensible choice. The data is already in HTML, so running an entire browser would add complexity without solving a problem that currently requires one.

There are still limitations. The script depends on Python.org's HTML structure, and a redesign could break its selectors. It also only processes the first page and doesn't automatically retry transient failures.

For a small, stable job board, those responsibilities may be perfectly manageable.

What if you use Playwright for the same page?

Playwright can extract the same information, but it needs to start a browser first.

Install Playwright and its Chromium browser:

pip install playwright
python -m playwright install chromium

You can keep the extract_jobs() function from the previous example and replace the Requests retrieval with the following code:

from playwright.sync_api import sync_playwright

with sync_playwright() as playwright:
    browser = playwright.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto(URL, wait_until="domcontentloaded")
    page.locator("ol.list-recent-jobs > li").first.wait_for()

    jobs = extract_jobs(page.content())
    browser.close()

The difference is how we retrieve the page. Requests downloads the response HTML directly, while Playwright loads it in Chromium and lets us inspect the resulting DOM.

Playwright is useful when browser behavior matters, but this job board doesn't require that extra step for the fields we're collecting.

That changes when a website builds its content after the initial response.

Compare browser and non-browser retrieval on Package Pulse

Package Pulse provides npm download statistics and comparisons between JavaScript packages.

The comparison for React, Vue, Angular, and Svelte includes monthly and yearly downloads, trends, and interactive visualizations.

This gives us a more realistic scenario: a data-heavy application where we need structured values from a table.

Before choosing Playwright, first inspect the page's HTML and network requests.

A website can use JavaScript extensively while still making the information you need available in its initial HTML or through an accessible data endpoint. If that's the case, Requests may be sufficient.

Read the comparison table with Playwright

Playwright can work directly with the table after the page loads.

The following example finds the table rows and extracts the download figures for the four packages.

# package_downloads.py
import re
from playwright.sync_api import sync_playwright

URL = (
    "https://www.package-pulse.com/compare/"
    "react-vs-vue-vs-%40angular/core-vs-svelte"
)

def number(text):
    match = re.search(r"\d[\d,\s\u00a0\u202f]*", text)
    return int(re.sub(r"\D", "", match.group())) if match else None

with sync_playwright() as playwright:
    browser = playwright.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL, wait_until="domcontentloaded")

    table = page.locator("table").first
    table.wait_for()

    packages = []
    for row in table.locator("tbody tr").all():
        cells = row.locator("td")
        if cells.count() < 4:
            continue

        packages.append({
            "name": cells.nth(0).inner_text().strip(),
            "monthly_downloads": number(cells.nth(2).inner_text()),
            "yearly_downloads": number(cells.nth(3).inner_text()),
        })

    browser.close()

Playwright offers two advantages here. It can wait for the table to appear, and it can interact with controls if the information changes when users select different packages or time ranges.

However, we're still responsible for identifying the table, parsing its cells, and handling changes to the page's structure.

Playwright's locator documentation explains the built-in waiting and retry behavior that makes these interactions more reliable than fixed delays.

Retrieve the same information with Spidra

Now consider a managed scraping API.

Spidra's Scrape API can retrieve a webpage and extract fields according to a prompt or JSON schema.

Instead of writing selectors for the table, we can describe the records we want and define their types.

Install the Python SDK:

pip install spidra

Set the SPIDRA_API_KEY environment variable, then create a schema for the download statistics.

# package_spidra.py
import os
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

schema = {
    "type": "object",
    "properties": {
        "packages": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "monthly_downloads": {"type": "integer"},
                    "yearly_downloads": {"type": "integer"}
                }
            }
        }
    }
}

result = client.scrape(
    "https://www.package-pulse.com/compare/"
    "react-vs-vue-vs-%40angular/core-vs-svelte",
    prompt=(
        "Extract monthly and yearly npm downloads for "
        "react, vue, @angular/core, and svelte from "
        "the Downloads by project table."
    ),
    schema=schema,
)

print(result.content)

The Spidra playground returned the following data for this task:

{
  "packages": [
    {
      "name": "react",
      "monthly_downloads": 24611745,
      "yearly_downloads": 890871734
    },
    {
      "name": "svelte",
      "monthly_downloads": 881681,
      "yearly_downloads": 40149360
    },
    {
      "name": "vue",
      "monthly_downloads": 3864256,
      "yearly_downloads": 94246383
    },
    {
      "name": "@angular/core",
      "monthly_downloads": 898638,
      "yearly_downloads": 30300240
    }
  ]
}

These figures matched those displayed on the Package Pulse comparison page when captured.

The records also have consistent field names and numeric types, making them easier to validate or store in a database.

Does Fast Mode change the result?

Spidra supports two retrieval modes.

Its default mode uses a browser, while Fast Mode begins with direct HTTP retrieval and falls back to a browser when necessary.

The same Package Pulse extraction was run in both modes:

The yearly figures also matched across both runs.

For this task, both modes returned the requested records with no observed differences in the extracted values.

That doesn't establish which mode was faster, or whether Fast Mode needed its browser fallback. Those questions require the execution metadata.

It does establish that the more browser-oriented configuration didn't improve the returned data in this particular comparison.

The decision should depend on the page and the workload, rather than assuming that a browser always produces better results.

Extract data from an interactive dashboard without scraping it

Our final example comes from Our World in Data's life expectancy visualization.

The dashboard includes a map, chart views, and a table of life expectancy figures for countries and regions.

An interactive interface might initially suggest Playwright. After all, you can use a browser to switch views, select countries, and inspect displayed values.

But there's another option.

First, find out whether the data has an API

Our World in Data publishes the underlying dataset as a downloadable CSV.

Its data API documentation explains how to retrieve chart data programmatically.

For the life expectancy dataset, we can use the CSV endpoint directly:

# life_expectancy.py
import csv
import io
import requests

URL = (
    "https://ourworldindata.org/grapher/"
    "life-expectancy.csv?v=1&csvType=full"
    "&useColumnShortNames=false"
)

response = requests.get(
    URL,
    headers={"User-Agent": "PythonDataExample/1.0"},
    timeout=30,
)
response.raise_for_status()

reader = csv.DictReader(io.StringIO(response.text))
value_column = next(
    col for col in reader.fieldnames
    if col not in {"Entity", "Code", "Year"}
)

countries = {
    "Monaco", "San Marino", "Hong Kong",
    "Japan", "South Korea"
}

for row in reader:
    if row["Year"] == "2023" and row["Entity"] in countries:
        print(row["Entity"], row[value_column])

This avoids the dashboard entirely.

Requests downloads the published dataset, and Python's CSV module reads it without launching a browser or depending on chart selectors.

For an application that needs the underlying time series, this is generally a better starting point than scraping the visualization.

What if you need the data currently displayed in the interface?

There are still cases where browser interaction makes sense.

You might need to reproduce a user's selected countries, current filters, or the order of rows displayed after someone interacts with the visualization.

Playwright can manipulate those controls and read the resulting table.

Spidra also supports browser actions, including clicking, waiting for elements, selecting options, and taking screenshots.

For example, the Spidra playground extracted these rows from the life expectancy table:

All five records matched the visible values in the captured table.

That establishes the accuracy of the extracted sample, although the captured view does not by itself establish that a browser action successfully changed the selected tab.

The more useful distinction is this: retrieving the underlying dataset and reproducing what a user sees in the interface are different tasks.

For the first, use the CSV. For the second, a browser may be appropriate.

What changes when you run these scrapers in production?

A script that successfully extracts five records is a useful start, but it doesn't tell you how much work the system will require after several months.

Each approach introduces different responsibilities.

Requests: Simple retrieval, application-owned extraction

With Requests, you handle the HTTP request and parse the response yourself.

You also decide how to manage connection errors, retries, rate limits, data validation, and changes in page structure.

The benefit is control without browser overhead.

For stable HTML pages and accessible APIs, this is often the simplest architecture.

Requests supports explicit timeouts and HTTP error handling, but you still need to implement retry policies and determine what to do when fields are missing.

If the site changes its CSS classes, your parser might return an empty list while the HTTP request still succeeds.

Playwright: Browser control, browser maintenance

Playwright gives you a browser automation environment, including navigation, locators, JavaScript execution, and page interaction.

Its automatic waiting helps reduce timing-related issues, but it doesn't eliminate them. Your code still needs to find the correct elements and recognize when the page is in the expected state.

In production, you'll also need to consider browser installation, updates, memory usage, concurrent sessions, and crash recovery.

Playwright's browser contexts can help you manage multiple isolated sessions without launching a separate browser process for every job. Our guide to scaling scraping with Playwright BrowserContext covers that architecture in more detail.

If your application needs precise interaction with a small number of websites, maintaining Playwright yourself may be the right decision.

Scraping API: Less retrieval infrastructure, but still your data pipeline

A managed scraping API moves some of the retrieval and browser responsibilities to a provider.

With Spidra, you can request pages through an SDK, run browser actions, and ask for structured extraction using a schema.

That reduces the infrastructure you manage, but doesn't eliminate your application's responsibilities.

You still need to check whether a job succeeded, whether the output contains the expected records, and whether those records are accurate.

For example, returning an object with a price field isn't sufficient if the value is null, outdated, or inconsistent with the source.

For recurring jobs across many URLs, you also need a strategy for handling partial results, duplicate records, and failed items. Our guide to scraping multiple URLs in Python covers those problems.

A scraping API also introduces a usage cost and dependency on an external service. Some websites can still refuse automated access, and extraction quality varies with the content.

The question is whether reducing infrastructure and maintenance work is worth those tradeoffs for your application.

How to choose between Requests, Playwright, and a scraping API

The examples give us a practical decision framework.

You also don't have to choose only one approach for an entire project.

A company enrichment application might use Requests for official APIs, Playwright for a specific interactive workflow, and a scraping API for extracting consistent information from many unrelated company websites.

This can be more practical than forcing every source through the same system.

When deciding whether to pay for a scraping API, compare more than the price per request. Look at the cost of producing a usable record, including failed attempts, parsing work, browser infrastructure, and maintenance.

A free HTTP request can become expensive if it requires significant engineering attention. A paid API call can also be unnecessary if a straightforward endpoint already provides the data.

Conclusion

Requests, Playwright, and scraping APIs solve related problems, but they shouldn't be interchangeable defaults.

Start with the simplest method that gives you reliable access to the data. Use Requests when HTTP responses are sufficient, Playwright when browser behavior is necessary, and a managed scraping API when outsourcing retrieval and extraction is more practical than maintaining the infrastructure yourself.

For applications that need recurring web data from different sources, you can try Spidra's Scrape API with your own URLs and compare the results against your existing approach.