Skip to main content
TutorialsNovember 19, 2026 · 12 min read

How to extract HTML tables into a Pandas DataFrame

Turn an HTML table into a pandas DataFrame and CSV with read_html, check the types and gaps, and render JavaScript tables when the page's HTML is empty

How to extract HTML tables into a Pandas DataFrame

You have a table on a public web page and you want it as a pandas DataFrame and a CSV file. This tutorial turns an HTML table into a DataFrame with pandas.read_html, checks the column types and missing values, and saves a CSV. It then covers the case where read_html finds nothing because the table is built by JavaScript.

Read every table on the page with pd.read_html

read_html parses every <table> in a page and returns a list of DataFrames. We'll use a page from Scrape This Site, a public sandbox for practising exactly this: 25 rows of NHL team stats.

# read_table.py
import requests
import pandas as pd
from io import StringIO

URL = "https://www.scrapethissite.com/pages/forms/"
html = requests.get(URL, timeout=30).text

tables = pd.read_html(StringIO(html))
print(len(tables), tables[0].shape)

Output:

1 (25, 9)

One table, 25 rows, 9 columns. A page with several tables gives you a longer list. Pick one by position, or narrow it down with match="some header text" or attrs={"id": "stats"}.

Wrap the HTML in StringIO. In pandas 3 a plain string is treated as a file path, so passing the page text directly fails:

pd.read_html(html)
FileNotFoundError: [Errno 2] No such file or directory: …

The message goes on to print the whole page, which is why it's easy to miss what went wrong.

Check the inferred types and missing values

pandas guesses a type for every column. info() shows the guess and the number of non-empty cells in one place:

df = tables[0]
print(df.head(3).to_string())
df.info()
        Team Name  Year  Wins  Losses  OT Losses  Win %  Goals For (GF)  Goals Against (GA)  + / -
0   Boston Bruins  1990    44      24        NaN  0.550             299                 264     35
1  Buffalo Sabres  1990    31      30        NaN  0.388             292                 278     14
2  Calgary Flames  1990    46      26        NaN  0.575             344                 263     81
<class 'pandas.DataFrame'>
RangeIndex: 25 entries, 0 to 24
Data columns (total 9 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   Team Name           25 non-null     str    
 1   Year                25 non-null     int64  
 2   Wins                25 non-null     int64  
 3   Losses              25 non-null     int64  
 4   OT Losses           0 non-null      float64
 5   Win %               25 non-null     float64
 6   Goals For (GF)      25 non-null     int64  
 7   Goals Against (GA)  25 non-null     int64  
 8   + / -               25 non-null     int64  
dtypes: float64(2), int64(6), str(1)
memory usage: 1.9 KB

Three things to read from this:

  • The header was found even though the page's header row sits inside <tbody> with no <thead>.
  • Numbers came out as numbers. Wins is int64 and Win % is float64, including the negative values in + / -. Text columns show as str. Older pandas versions show object here, and many tutorials still do.
  • OT Losses is empty on every row. The cells are blank in the HTML, so pandas fills the column with NaN and types it float64. When a whole-number column has even one blank cell, pandas does the same, because NaN is a float. Convert it with .astype("Int64") if you need integers with gaps.

If a numeric column arrives as str, a cell contains something that isn't a number, such as a footnote marker or N/A. Find it with pd.to_numeric(df["Wins"], errors="coerce"), which turns the bad cells into NaN. The thousands=, na_values= and parse_dates= options of read_html handle the common cases while reading.

Clean the column names and save a CSV

Short names are easier to type, and a column with no data is only noise in the file:

# clean_table.py (after reading the table as above)
df.columns = ["team", "year", "wins", "losses", "ot_losses", "win_pct", "gf", "ga", "diff"]
df = df.dropna(axis=1, how="all")   # ot_losses is empty on every row
print(df.shape)

df.to_csv("hockey.csv", index=False)
print("\n".join(open("hockey.csv").read().splitlines()[:3]))

Output:

(25, 8)
team,year,wins,losses,win_pct,gf,ga,diff
Boston Bruins,1990,44,24,0.55,299,264,35
Buffalo Sabres,1990,31,30,0.388,292,278,14

index=False keeps pandas' row numbers out of the file. That is the whole job for a table that's in the page's HTML, and it needs no API and no account.

The pager at the bottom of this page links to ?page_num=2 and so on, so you can loop over the pages with the same code.

Use Beautiful Soup when read_html throws something away

read_html keeps cell text and drops everything else: attributes, CSS classes, link targets. This page colours each Win % with a class, and Beautiful Soup can read it:

# soup_table.py (after reading and renaming as above)
from bs4 import BeautifulSoup

rows = BeautifulSoup(html, "html.parser").select("tr.team")
df["above_500"] = ["text-success" in r.select_one("td.pct")["class"] for r in rows]
print(df[["team", "win_pct", "above_500"]].head(3).to_string())
print((df["above_500"] == (df["win_pct"] > 0.5)).all())

Output:

             team  win_pct  above_500
0   Boston Bruins    0.550       True
1  Buffalo Sabres    0.388      False
2  Calgary Flames    0.575       True
True

The class turns out to mean "above .500", so here it only repeats what win_pct already says. It shows the mechanism: when the value you want lives in an attribute, a class or a link, read the table with pandas and add that column from the soup. The Beautiful Soup guide covers the library in more depth.

Find out why a table is missing from the HTML you downloaded

The same site has a second page, Oscar winning films, where you click a year to see its films. Running the same code on it fails:

URL = "https://www.scrapethissite.com/pages/ajax-javascript/"
html = requests.get(URL, timeout=30).text
pd.read_html(StringIO(html))
ValueError: No tables found matching pattern '.+'

Check what the server actually sent:

print(len(html), html.count("<table"), "Spotlight" in html)

Output:

13210 1 False

There is one <table>, and none of the films are in the page. The table in the downloaded HTML is an empty shell, hidden until a script fills it:

<table class="table" style="display:none"> <thead> <tr> <th>Title</th> <th>Nominations</th> <th>Awards</th> <th>Best Picture</th> </tr> </thead> <tbody id="table-body"> </tbody> </table>

requests and read_html don't run JavaScript, so they only ever see this shell. Pages built this way, and pages that want a click, a filter or a scroll before the rows appear, need a real browser. A blocked request (an HTTP 403, or a consent page instead of the content) looks similar from the outside: check what html contains before blaming the parser.

Render the page in a browser, then hand the rows to pandas

If you need this for one page, a headless browser such as Playwright is enough. When you need it across several sites, or you don't want to run browsers yourself, Spidra's Scrape does the browser step. It loads the page in a real browser, runs browser actions you give it (click, select, wait, scroll, run JavaScript), and returns the result.

Scrape returns Markdown or JSON, not raw HTML, so read_html has nothing to parse. Instead, the evaluate action runs a line of JavaScript in the page and returns the table's cells, and pandas builds the DataFrame from them. If you haven't used the SDK yet, the Python tutorial covers the install and your API key.

# render_table.py
import json
import os

import pandas as pd
from spidra import BrowserAction, ScrapeUrl, Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

# Returns every row of the first table as cell text. An icon with no text becomes "yes".
ROWS = """JSON.stringify([...document.querySelectorAll('table tr')].map(tr =>
  [...tr.cells].map(c => c.innerText.trim() || (c.querySelector('i') ? 'yes' : ''))))"""

The function runs your steps, then the evaluate step last, and reads that step's value from action_results:

def render_table(url, steps):
    result = client.scrape(
        [ScrapeUrl(url=url, actions=[*steps, BrowserAction(type="evaluate", script=ROWS)])],
        extract_content_only=True,
    )
    rows = json.loads(result.data[0].action_results[-1].result)
    return pd.DataFrame(rows[1:], columns=rows[0])

Now click the 2015 button, wait for the rows, and get the table:

oscars = render_table(
    "https://www.scrapethissite.com/pages/ajax-javascript/",
    [
        BrowserAction(type="click", selector="a.year-link[id='2015']"),
        BrowserAction(type="waitFor", selector="tr.film"),
    ],
)
print(oscars.shape)
print(oscars.head(3).to_string())

Output:

(16, 4)
                Title Nominations Awards Best Picture
0           Spotlight           6      2          yes
1  Mad Max: Fury Road          10      6             
2        The Revenant          12      3             

The films that were missing from the HTML are here. Everything arrives as text, so set the types yourself, then save the CSV exactly as before:

oscars = oscars.astype({"Nominations": "int64", "Awards": "int64"})
oscars["Best Picture"] = oscars["Best Picture"].eq("yes")
print(oscars.dtypes)

oscars.to_csv("oscars_2015.csv", index=False)
print("\n".join(open("oscars_2015.csv").read().splitlines()[:3]))
Title             str
Nominations     int64
Awards          int64
Best Picture     bool
dtype: object
Title,Nominations,Awards,Best Picture
Spotlight,6,2,True
Mad Max: Fury Road,10,6,False

Best Picture is drawn as an icon, so its cell has no text. The ROWS script turns a cell with an icon and no text into yes, which is why Spotlight, the 2015 winner, comes back True.

Why return the cells and not the page's Markdown

Scrape's default output is a Markdown version of the page, and it contains the table. It is also lossy for tables with blank cells. This is the hockey page again, header first, then the first row:

| Team Name | Year | Wins | Losses | OT Losses | Win % | Goals For (GF) | Goals Against (GA) | \+ / - |
| Boston Bruins | 1990 | 44 | 24 | 0.55 | 299 | 264 | 35 | |

The blank OT Losses cell vanished, so every value after it moved one column left: 0.55 now sits under OT Losses. The evaluate rows keep blank cells as empty strings, so the columns stay aligned.

Two more limits to know. An evaluate result is cut off at 10,000 characters. Returning a 36-row table's full outerHTML was truncated, so return cell text instead. And the first page of a paginated table only holds one page of rows, which the next section deals with.

Reuse the same function on a second site

render_table doesn't know anything about one site. DataTables, a popular table library, has a demo page where the rows come from a JavaScript array and only 10 of 36 are shown ("Showing 1 to 10 of 36 entries"). Only the steps change: wait for the table, set the page size to 100 with a select action, wait for the last row.

staff = render_table(
    "https://datatables.net/examples/data_sources/js_array.html",
    [
        BrowserAction(type="waitFor", selector="#example tbody tr"),
        BrowserAction(type="select", selector=".dt-length select", value="100"),
        BrowserAction(type="waitFor", selector="#example tbody tr:nth-child(36)"),
    ],
)
staff["Salary"] = staff["Salary"].str.replace(r"[$,]", "", regex=True).astype("int64")
staff["Start date"] = pd.to_datetime(staff["Start date"], format="%Y/%m/%d")
print(staff.shape)
print(staff.dtypes)
print(staff.head(3).to_string())

Output:

(36, 6)
Name                     str
Position                 str
Office                   str
Extn.                    str
Start date    datetime64[us]
Salary                 int64
dtype: object
             Name                       Position         Office Extn. Start date   Salary
0      Airi Satou                     Accountant          Tokyo 5407 2008-11-28   162700
1  Angelica Ramos  Chief Executive Officer (CEO)         London 5797 2009-10-09  1200000
2      Ashton Cox        Junior Technical Author  San Francisco 1562 2009-01-12    86000

All 36 rows came back. The salaries ($162,700) and dates (2008/11/28) arrive as text, so the last two lines convert them. Extn. stays text on purpose, since an extension number is an identifier and not a quantity.

The other way to get a table out of Scrape is to describe it: pass a prompt and a schema, and the page is read by AI, which suits layouts that vary from site to site. That route reads text, and on a wide table it can shift a column. Asked for the first rows of the 1990s table on Wikipedia's timeline of programming languages, it returned these two rows, where designed_by should hold the designers:

{'language': 'Haskell', 'first_appeared': 1990, 'designed_by': 'Miranda, Clean'}
{'language': 'EuLisp', 'first_appeared': 1990, 'designed_by': 'Common Lisp, Scheme'}

Those are the table's "Predecessor(s)" column. Check AI-extracted tables against the page, and prefer the cell-by-cell route above when the table is a plain <table>.

Choose an approach

Approach

Use it when

It stops working when

pd.read_html(StringIO(html))

The table is in the HTML you download

The rows are added by JavaScript, or the site blocks your request

requests with your own headers, then read_html

The site rejects pandas' default request

The page needs a click, a filter or a scroll

Beautiful Soup on top

The value is in an attribute, a class or a link

The table isn't in the downloaded HTML either

Playwright or similar, then read_html

One site, and you're happy running a browser

You have many sites, or you don't want to maintain browsers

Spidra Scrape with actions and evaluate

JavaScript tables, tables behind a click or a page-size selector, the same job on several sites

You only need one static table: pandas alone is simpler and free

Spidra Scrape with a prompt and schema

Layouts that differ from site to site, one output shape

A wide table where a shifted column would go unnoticed

If the table is in the HTML, use read_html and stop. Everything below it in the table exists for pages where that isn't true.

What this doesn't handle

  • Tables behind a login. Spidra can use your own session cookies, and you are responsible for the accounts you use.
  • Many pages of one table. A static pager can be looped with read_html. A JavaScript pager needs a click per page, or the forEach action with pagination (see the browser actions guide).
  • Tables inside PDFs. Those need a PDF parser, not an HTML one.
  • Several tables on one page. The ROWS script reads the first table. Change 'table tr' to '#your-id tr' to pick another.
  • Pages that still block you. Some sites refuse automated visits. Those pages fail, and failed pages aren't charged.
  • A site's terms. The Scrape This Site pages are built for scraping practice, and the DataTables page is the library's own demo. Check the terms of any other site first, and read our guide to the legal side of scraping.

Conclusion

You now have two ways to a DataFrame and a CSV. For a table that's in the HTML, pandas.read_html is the right tool: it infers the types, keeps the missing values as NaN, and costs nothing. Check info() before you trust the result, and reach for Beautiful Soup only when the value you need is in an attribute.

The trouble starts when the table isn't in the HTML. Then each site becomes its own piece of browser code: launching a browser, finding the right button, waiting for the rows, extracting the cells, and keeping it all running when the site changes or a request gets blocked. That is the work Spidra takes off your hands. Spidra opens the page in a real browser, runs your steps, and returns the rows, so your script stays a few lines of pandas. In this tutorial one render_table function read an AJAX page and a table library's page, and the only difference between them was a list of steps.

It goes further than tables. Batch scrape reads up to 50 URLs in one request, Crawl and Map find the pages first, and Parse reads PDFs and Office files. For popular sites there are over 90 ready-made templates that return a table with one row per result, so you can skip the scraping code entirely. Every plan includes every feature, with the plans differing in credits, proxy bandwidth, pages per crawl and monitors, so the way you prototype is the way you run it in production. Official SDKs cover ten languages, and an MCP server lets AI tools use it too.

Create a free account at app.spidra.io, set SPIDRA_API_KEY, and run render_table.py on the JavaScript table you couldn't read before.

Frequently asked questions

A page can hold any number of tables, so read_html returns one DataFrame per table, in page order. Use tables[0] for the first, or narrow the list at read time with match= (text inside the table) or attrs= (such as {"id": "stats"}).

Share this article

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.