How to scrape multiple URLs in Python and handle failures

Scraping three webpages with Python is easy enough. You write a loop, send a request to each URL, and extract the content you need. But what happens when you have 100 URLs, one takes 30 seconds to respond, and another returns an HTTP 403 error?

A simple loop can become painfully slow, and without proper error handling, one failed request can stop the entire script.

The better approach is to process URLs concurrently, retry temporary failures, and keep successful results even when some pages can't be scraped.

In this tutorial, you'll learn how to scrape multiple URLs in Python using requests and ThreadPoolExecutor, handle errors without losing results, and use Spidra's Batch scrape feature when you need to process multiple pages without managing the scraping infrastructure yourself.

Scrape multiple URLs using Python requests

Let's say you want to collect book titles and prices from several product pages. We'll use Books to Scrape, a website designed for web scraping practice.

First, install Requests and Beautiful Soup:

pip install requests beautifulsoup4

Create a file named scraper.py:

# scraper.py
import requests
from bs4 import BeautifulSoup

urls = [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    "https://books.toscrape.com/catalogue/soumission_998/index.html",
]

for url in urls:
    response = requests.get(url, timeout=15)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    title = soup.select_one("h1").get_text(strip=True)

    print(title)

The script visits each page, parses its HTML, and extracts the book title from the h1 element.

If all three requests succeed, you'll get:

A Light in the Attic
Tipping the Velvet
Soumission

There's nothing particularly wrong with this approach for a small number of pages. But every request runs sequentially, so Python waits for one page to finish before requesting the next.

If you're scraping 100 URLs and each takes two seconds, that's roughly 200 seconds spent on requests alone.

There's another problem. If the second URL returns an HTTP error, raise_for_status() raises an exception and stops the script before the third URL is processed.

We can improve both behaviors.

Run multiple scraping requests concurrently with ThreadPoolExecutor

Python's concurrent.futures module includes ThreadPoolExecutor, which lets you run multiple functions concurrently using threads.

This works well for web scraping because HTTP requests are I/O-bound operations. Most of the time is spent waiting for a response, so other threads can make progress while one is waiting.

Instead of making requests directly inside a loop, move the scraping logic into a function:

from concurrent.futures import ThreadPoolExecutor, as_completed

def scrape_page(url):
    response = requests.get(url, timeout=15)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    return {
        "url": url,
        "title": soup.select_one("h1").get_text(strip=True),
    }

Now use ThreadPoolExecutor to process the URLs:

with ThreadPoolExecutor(max_workers=5) as executor:
    futures = {
        executor.submit(scrape_page, url): url
        for url in urls
    }

    for future in as_completed(futures):
        result = future.result()
        print(result)

Here, max_workers=5 means up to five scraping functions can run concurrently. With our three URLs, all three requests can start without waiting for the previous one to finish.

The as_completed() function returns futures as they finish, so results don't necessarily appear in the same order as the original URL list.

You can adjust the worker count depending on the number of URLs and the website's capacity. More workers don't always mean better performance, especially when multiple requests are hitting the same server.

This solves our sequential request problem, but the script can still fail if one of the threads raises an exception.

Handle failed URLs without stopping the entire scraper

Imagine you're scraping 50 URLs. The first 20 requests succeed, but the next page returns HTTP 404 because the product has been removed.

You don't want one missing page to interrupt the remaining requests.

A better approach is to return a consistent result for each URL, whether the request succeeds or fails.

Update scrape_page():

def scrape_page(url):
    try:
        response = requests.get(url, timeout=(5, 15))
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        heading = soup.select_one("h1")

        if heading is None:
            raise ValueError("Page has no h1 heading")

        return {
            "url": url,
            "status": "success",
            "title": heading.get_text(strip=True),
        }

    except (requests.RequestException, ValueError) as error:
        return {
            "url": url,
            "status": "failed",
            "error": str(error),
        }

Now the function always returns a dictionary containing the URL and its status.

We're also handling a missing h1 element explicitly. Without that check, trying to call .get_text() on a missing element would raise an AttributeError.

Collect the results into separate lists:

results = []

with ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(scrape_page, url) for url in urls]

    for future in as_completed(futures):
        results.append(future.result())

successful = [r for r in results if r["status"] == "success"]
failed = [r for r in results if r["status"] == "failed"]

print(f"Successful: {len(successful)}")
print(f"Failed: {len(failed)}")

If three pages succeed and one fails, the script reports three successful requests and one failed request.

More importantly, you still have the successful results. You can save them, process them further, and investigate the failures separately.

But there's a distinction worth making here. A request can fail because a page no longer exists, because the server is temporarily unavailable, or because your connection timed out.

Those failures shouldn't all be handled the same way.

Retry temporary scraping failures with exponential backoff

Suppose a website returns HTTP 503 because its server is temporarily overloaded. Retrying the request might work.

Now suppose another URL returns HTTP 404 because the page was deleted. Retrying it five times probably won't change anything.

For temporary failures, a common approach is exponential backoff, where your script waits progressively longer between retry attempts.

The Requests library supports this through urllib3's Retry class and Requests' HTTPAdapter.

Add the following imports:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

Then create a function that configures retry behavior:

def create_session():
    retry = Retry(
        total=3,
        backoff_factor=1,
        backoff_jitter=0.5,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET"],
        respect_retry_after_header=True,
    )

    session = requests.Session()
    session.mount("https://", HTTPAdapter(max_retries=retry))
    session.mount("http://", HTTPAdapter(max_retries=retry))

    return session

This configuration allows up to three retries after the initial request for eligible failures.

The delay generally increases between repeated failures, with a small random adjustment to avoid making every retry at the same moment. urllib3 doesn't add a backoff delay before the first retry by default.

It also respects supported Retry-After instructions from the server, which is particularly relevant when you receive HTTP 429 responses.

Replace the request inside scrape_page() with:

with create_session() as session:
    response = session.get(url, timeout=(5, 15))
    response.raise_for_status()

The rest of the function can remain unchanged.

Each invocation creates its own session, so we're not sharing a mutable Requests session across worker threads. In a larger application, you could reuse a session within each worker instead of constructing one for every URL.

This is already a much more reliable scraper than our first loop. We can fetch multiple pages concurrently, retry certain temporary errors, and collect failures without interrupting the rest of the work.

Still, retries have limits.

If a website requires JavaScript to render the data, returns a bot challenge, or consistently denies automated access, repeating the same HTTP request may not help.

You could introduce browser automation, proxies, and additional extraction logic, but maintaining all of that becomes another project.

For those situations, a scraping API can be a more practical option.

Scrape up to 50 URLs in one batch with Spidra

Spidra is a web data API that retrieves content from webpages and returns it as Markdown or structured JSON. It supports browser rendering, proxy usage, and CAPTCHA solving, so you don't need to build those components yourself.

When you already have a list of URLs, you can use its Batch scrape feature to process multiple pages.

There's an important distinction between Spidra's two scraping approaches.

Regular Scrape accepts up to three URLs and combines their content into a single result. Batch scrape accepts up to 50 URLs per request and returns a separate result for each URL.

Let's try it with the same book pages.

First, install the Spidra Python SDK:

pip install spidra

Create an API key from your Spidra dashboard and set it as an environment variable.

On macOS or Linux:

export SPIDRA_API_KEY="spd_YOUR_API_KEY"

Create batch_scraper.py:

# batch_scraper.py
import os
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

urls = [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    "https://books.toscrape.com/catalogue/soumission_998/index.html",
]

batch = client.batch_scrape(urls, output="markdown")

for item in batch.items:
    print(item.url, item.status, item.credits_used)

The batch_scrape() method submits the URLs and waits for the batch to complete.

You don't need to create threads or poll the API manually. Each URL is processed individually, and the returned batch contains the status, result, and credit usage for each page.

Because we requested Markdown, Spidra returns the page content rather than requiring us to extract it using a CSS selector.

That makes it useful when you're collecting entire pages for research, indexing, or other applications that need more than a single HTML element.

But what happens when one of the URLs doesn't work?

A completed batch can still contain failed URLs

This is one of the more important details when working with batch scraping.

A batch can finish successfully as an operation while individual URLs inside it fail.

To see how that works, we tested Spidra with four URLs. Two were valid pages from Quotes to Scrape, one pointed to a nonexistent book page, and the last used a domain that couldn't be resolved.

Here's the request:

urls = [
    "https://quotes.toscrape.com/page/1/",
    "https://quotes.toscrape.com/page/2/",
    "https://books.toscrape.com/this-page-does-not-exist",
    "https://this-domain-does-not-exist-spidra-example.com/",
]

batch = client.batch_scrape(urls, output="markdown")

print(batch.status, "completed:", batch.completed_count, "failed:", batch.failed_count)

The recorded output was:

completed completed: 2 failed: 2

The batch completed, but only two of the four pages were scraped successfully.

We inspected the failed items:

for item in batch.items:
    if item.status == "failed":
        print(item.url, "|", item.credits_used, "|", item.error)

The first failure returned:

https://books.toscrape.com/this-page-does-not-exist | 0 | Failed to scrape https://books.toscrape.com/this-page-does-not-exist — The page was not delivered: the page answered HTTP 404. Nothing was charged. Please retry.

The second returned:

https://this-domain-does-not-exist-spidra-example.com/ | 0 | Failed to scrape https://this-domain-does-not-exist-spidra-example.com/: the domain could not be found (DNS lookup failed). Check the URL for typos (ERR_NAME_NOT_RESOLVED)

The batch took approximately 13 seconds in this test and consumed two credits, one for each successfully scraped page. The failed pages cost nothing.

The lesson isn't that every failed page needs to be retried. Neither of those URLs would benefit from immediately repeating the same request.

The important part is that we still received the two successful results, along with enough information to identify and investigate the failures.

Always inspect the status of individual batch items, not just the overall batch status.

Spidra also provides a batch retry operation, available through client.retry_batch_scrape(batch_id), which re-runs failed items from an existing batch.

To use it, you need the original batch ID. You can obtain one through start_batch_scrape() or the batch job records. The normal batch_scrape() method is simpler when you only need to submit a batch and wait for its results.

The retry operation is useful for temporary failures, but you should still inspect the errors before using it. Retrying a permanently missing page only repeats the same problem.

Extract structured data from multiple websites

Collecting Markdown is useful when you need the content of each page, but suppose you're scraping product information from several websites.

You want each result to contain a product name, price, currency, and stock status.

With Beautiful Soup, you'd typically write selectors for each website's HTML structure. That gets harder when the pages use different layouts.

Spidra supports structured extraction using a JSON schema.

For example, you can describe the fields you want:

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": "number"},
        "currency": {"type": ["string", "null"]},
        "in_stock": {"type": ["boolean", "null"]},
        "sku": {"type": ["string", "null"]},
    },
}

Then pass the schema to Batch scrape:

batch = client.batch_scrape(
    urls,
    prompt="Extract the product",
    schema=schema,
)

for item in batch.items:
    if item.status == "completed":
        print(item.result)
    else:
        print("Failed:", item.url, item.error)

In a separate test, we used this schema to extract product data from three websites with different page layouts.

The results included:

{
    'name': 'A Light in the Attic',
    'price': 51.77,
    'currency': '£',
    'in_stock': True,
    'sku': 'a897fe39b1053632'
}

{
    'name': 'Bulbasaur',
    'price': 63,
    'currency': '£',
    'in_stock': True,
    'sku': '4391'
}

{
    'name': 'Dell XPS 13',
    'price': 1281.99,
    'currency': '$',
    'in_stock': None,
    'sku': None
}

The three pages had different layouts, but the extracted data followed the same structure.

Notice the None values in the laptop result. The relevant fields weren't available in the page's visible content, so they were returned as missing rather than filled with assumed values.

That's useful for applications that consume structured data. You can validate the records consistently and distinguish missing fields from actual values.

For this test, the three-page batch consumed six credits and took approximately 14 seconds. Structured extraction can use additional credits for AI processing, depending on the content and extraction request.

You can find more examples in the Spidra structured output documentation.

Save successful results and failed URLs separately

One final improvement makes batch scraping much easier to work with: saving the results as you process them.

Suppose you're scraping 200 URLs. You don't want to keep everything in memory and only write the data after all 200 pages have finished.

If your script stops halfway through, you'll lose the results you've collected but haven't saved.

Instead, process the URLs in smaller batches and save completed records after each batch.

We'll use JSON Lines (JSONL), where each line is a separate JSON object. This makes it easy to read the results later, whether you're importing them into a database or processing them with Python.

Create bulk_scraper.py:

# bulk_scraper.py
import json
import os
from spidra import Spidra, SpidraError

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

urls = [
    "https://quotes.toscrape.com/page/1/",
    "https://quotes.toscrape.com/page/2/",
    "https://books.toscrape.com/this-page-does-not-exist",
    "https://this-domain-does-not-exist-spidra-example.com/",
]

def chunks(items, size=50):
    for start in range(0, len(items), size):
        yield items[start:start + size]

def save_record(file, record):
    file.write(json.dumps(record, ensure_ascii=False) + "\n")
    file.flush()

The chunks() function divides the URL list into groups of no more than 50, matching Spidra's batch limit.

Next, add the scraping function:

def scrape_urls(urls):
    with open("results.jsonl", "w", encoding="utf-8") as success_file, \
         open("failed.jsonl", "w", encoding="utf-8") as failed_file:

        for group in chunks(list(dict.fromkeys(urls))):
            batch = client.batch_scrape(group, output="markdown")

            for item in batch.items:
                if item.status == "completed":
                    save_record(success_file, {
                        "url": item.url,
                        "content": item.result,
                    })
                else:
                    save_record(failed_file, {
                        "url": item.url,
                        "status": item.status,
                        "error": item.error,
                    })

            print(
                f"Completed: {batch.completed_count}, "
                f"Failed: {batch.failed_count}"
            )

if __name__ == "__main__":
    try:
        scrape_urls(urls)
    except SpidraError as error:
        print(f"Batch operation interrupted: {error}")
        raise

Run it with:

python bulk_scraper.py

The script creates two files:

• results.jsonl contains successfully scraped pages, with the original URL and extracted Markdown content.
• failed.jsonl contains URLs that failed, their status, and the error message returned by Spidra.

It also removes duplicate input URLs while preserving their original order, avoiding unnecessary repeated scrapes within a single run.

For the four test URLs, the successful file will contain records for the two Quotes to Scrape pages, while the failed file will contain the missing book page and unresolved domain.

The files are opened in write mode, so each new execution starts with fresh output. Within an execution, the script writes each record as it becomes available and flushes it to the operating system.

You should also respect the API's submission limits. Spidra allows up to 20 batch submissions per minute, so large applications should pace submissions and back off when they receive a rate-limit response.

The Batch scraping documentation covers the operations available for checking and retrying existing batches.

When should you use requests instead of a scraping API?

You don't always need a scraping API to collect content from multiple URLs.

If the pages are static, accessible through ordinary HTTP requests, and follow predictable HTML structures, Requests with ThreadPoolExecutor may be all you need.

You can control concurrency, retries, HTML parsing, and output handling without paying for a separate scraping service.

The tradeoff appears when the websites become more complicated. Some pages load data through JavaScript, some return challenges, and others use different layouts that require maintaining separate extraction rules.

Spidra is useful when you want the scraping and extraction handled through one API, particularly when collecting data from many different pages.

It doesn't guarantee that every website will work. Some pages may still block automated access, and failures need to be inspected. However, failed scrape jobs don't consume credits, and the batch response gives you individual results to work with.

Whichever approach you choose, respect website terms, access restrictions, and applicable laws. Even a technically accessible page may have restrictions on automated collection or reuse of its data. For more context, see our guide to web scraping and legality.