How to convert arXiv PDFs to Markdown with Python

Research papers on arXiv contain valuable information for AI applications, but working directly with PDFs can make it difficult to extract text, preserve mathematical equations, and organize content for retrieval.

Converting these papers to Markdown gives you structured text that is easier to search, store, and process. With Spidra, you can parse an arXiv PDF directly from its URL, extract selected pages, and preserve equations, tables, and page references.

In this tutorial, you'll use Python and Spidra Parse to convert the Attention Is All You Need research paper into Markdown. You'll also save the paper's metadata and prepare the extracted content for reuse in research tools, AI agents, or RAG applications.

Get an arXiv research paper

Every arXiv paper has a unique identifier that you can use to access its abstract page and PDF.

For example, the paper Attention Is All You Need, which introduced the Transformer architecture, has the identifier 1706.03762.

Its abstract page is available at:

https://arxiv.org/abs/1706.03762

The PDF is available at:

https://arxiv.org/pdf/1706.03762

We'll use version 7 of this paper, 1706.03762v7, so the conversion refers to a specific revision.

This paper is a useful example because it includes mathematical equations, scientific tables, multi-column text, and multiple authors. We'll extract three PDF pages containing the title and abstract, the scaled dot-product attention equation, and the model evaluation results.

You don't need to download the PDF manually. Spidra accepts the document URL and returns its content as Markdown.

Set up Python and Spidra

You'll need Python 3.9 or later and a Spidra account with an API key.

Install the Spidra Python SDK and the arxiv package:

pip install spidra arxiv

The Spidra SDK handles document parsing, while the arxiv package retrieves bibliographic information such as the paper's title, authors, publication date, and version from the arXiv API.

Next, create an API key in your Spidra dashboard and save it as an environment variable:

export SPIDRA_API_KEY="spd_YOUR_API_KEY"

You can now submit PDF links to Spidra using Python.

Convert the arXiv PDF to Markdown

Create a file named parse_arxiv.py and add the following code:

# parse_arxiv.py
import os
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

doc = client.parse(
    "https://arxiv.org/pdf/1706.03762v7",
    pages="1,4,8",
    mode="ocr",
    include_pages=True,
    page_markers=True,
)

The parse() method accepts the paper's URL and returns the extracted content as Markdown. The Python SDK handles job submission and polling, so you don't need to implement those steps yourself.

A few options control the result:

• pages="1,4,8" selects three PDF pages instead of processing the entire document. You can also specify ranges such as "1-5" or "10-".
• mode="ocr" uses Spidra's AI-powered reading mode for every selected page. This is useful for research papers containing mathematical notation, tables, and complex layouts.
• include_pages=True returns each page separately with its page number and extracted Markdown.
• page_markers=True inserts page markers into the combined Markdown, making it easier to trace content back to the original document.

Spidra also offers auto mode, which uses a PDF's embedded text when available and reads scanned pages with AI. For this paper, we'll use OCR mode because it preserves the mathematical notation and table values particularly well.

What the extracted Markdown looks like

The conversion returns the paper's headings, paragraphs, author information, equations, and tables as Markdown.

For example, the extracted content includes the section:

### 3.2.1 Scaled Dot-Product Attention

We call our particular attention "Scaled
Dot-Product Attention" (Figure 2).

The mathematical equation from the original PDF is represented using LaTeX:

$$
\text{Attention}(Q, K, V) =
\text{softmax}\left(
\frac{QK^T}{\sqrt{d_k}}
\right)V
$$

This represents the scaled dot-product attention equation:

$$
\text{Attention}(Q,K,V)=
\text{softmax}\left(
\frac{QK^T}{\sqrt{d_k}}
\right)V
$$

Preserving the mathematical notation is useful when you want to index research papers, explain scientific concepts through an AI assistant, or render the extracted material in a Markdown viewer that supports LaTeX.

The results table is also converted into Markdown. Here is an excerpt from the paper's model comparison table:

| Model | BLEU | | Training Cost (FLOPs) | |
| :--- | :---: | :---: | :---: | :---: |
| | EN-DE | EN-FR | EN-DE | EN-FR |
| Transformer (base model) | 27.3 | 38.1 | $3.3 \cdot 10^{18}$ | |
| Transformer (big) | 28.4 | 41.8 | $2.3 \cdot 10^{19}$ | |

The model names, BLEU scores, and training-cost values are available as text rather than being locked inside the PDF.

You can access the complete result through doc.markdown, or work with individual pages through doc.pages.

Each page record includes its page number, Markdown content, and reading method. This lets you retain document context when building search or retrieval applications.

Retrieve the paper's metadata from arXiv

The Markdown contains the paper's text, but a useful research workflow should also preserve information about its source.

For example, if an AI assistant retrieves a passage from a research paper, you may want to display its title, authors, publication date, and the PDF page containing the answer.

The arxiv Python package provides this information through arXiv's API.

Add the following code:

import arxiv

arxiv_client = arxiv.Client()

search = arxiv.Search(
    id_list=["1706.03762v7"]
)

paper = next(arxiv_client.results(search))

The paper object contains bibliographic information, including:

• paper.title: The research paper's title.
• paper.authors: The authors associated with the paper.
• paper.published: The original submission date.
• paper.updated: The revision date.
• paper.entry_id: The URL identifying the paper on arXiv.
• paper.get_short_id(): The arXiv identifier, including its version.

The get_short_id() method is especially useful for keeping track of revisions. A paper may be updated after its initial submission, and its version identifier lets you associate the extracted text with the correct revision.

We'll use this metadata when saving the Markdown to disk.

Save the Markdown and paper metadata using Python

Let's combine document conversion and metadata retrieval into one Python script.

The script will:

1. Retrieve a paper's bibliographic information from arXiv.
2. Convert selected PDF pages into Markdown using Spidra.
3. Save the Markdown with the original source information.
4. Write the paper's metadata to a separate JSON file.

Create a new file named arxiv_to_markdown.py:

# arxiv_to_markdown.py
import json
import os
from pathlib import Path

import arxiv
from spidra import Spidra

ARXIV_ID = "1706.03762v7"
SELECTED_PAGES = "1,4,8"

# Retrieve bibliographic metadata.
paper = next(
    arxiv.Client().results(
        arxiv.Search(id_list=[ARXIV_ID])
    )
)

paper_id = paper.get_short_id()
pdf_url = f"https://arxiv.org/pdf/{paper_id}"

# Convert the PDF to Markdown.
client = Spidra(
    api_key=os.environ["SPIDRA_API_KEY"]
)

doc = client.parse(
    pdf_url,
    pages=SELECTED_PAGES,
    mode="ocr",
    include_pages=True,
    page_markers=True,
)

# Preserve source and page information.
metadata = {
    "arxiv_id": paper_id,
    "title": paper.title,
    "authors": [author.name for author in paper.authors],
    "published": paper.published.date().isoformat(),
    "updated": paper.updated.date().isoformat(),
    "source_url": paper.entry_id,
    "pdf_url": pdf_url,
    "pages": [page.page_number for page in doc.pages],
    "total_pages": doc.total_pages,
}

# Create the output directory.
output_dir = Path("papers")
output_dir.mkdir(exist_ok=True)
base_name = paper_id.replace("/", "_")

# Save Markdown with source attribution.
markdown = (
    f"# {paper.title}\n\n"
    f"**Authors:** {', '.join(metadata['authors'])}\n\n"
    f"**Source:** {paper.entry_id}\n\n"
    f"**PDF:** {pdf_url}\n\n"
    f"{doc.markdown}\n"
)

(output_dir / f"{base_name}.md").write_text(
    markdown,
    encoding="utf-8",
)

# Save structured paper metadata.
(output_dir / f"{base_name}.metadata.json").write_text(
    json.dumps(metadata, ensure_ascii=False, indent=2),
    encoding="utf-8",
)

Run the script:

python arxiv_to_markdown.py

It creates a papers directory containing two files:

papers/
├── 1706.03762v7.md
└── 1706.03762v7.metadata.json

The Markdown file contains the paper's extracted content, together with its title, authors, source URL, and PDF link.

Because page_markers=True is enabled, the combined Markdown also contains HTML comments identifying the source pages:

<!-- page 1 -->

<!-- page 4 -->

<!-- page 8 -->

These markers are inserted at the corresponding page boundaries, rather than being presented as a separate list. They remain available in the Markdown source even when a renderer hides HTML comments.

The JSON file provides the same paper identity in a format your application can read directly. Some of its fields look like this:

{
  "arxiv_id": "1706.03762v7",
  "title": "Attention Is All You Need",
  "pages": [1, 4, 8],
  "total_pages": 15
}

The complete JSON also includes the author list, publication and revision dates, and original source URLs.

To process a different research paper, change ARXIV_ID. You can also change SELECTED_PAGES to extract another range or remove the pages argument from client.parse() to convert the entire PDF.

The same script will retrieve the new paper's metadata and save its results under the corresponding arXiv identifier.

Prepare the Markdown for a research or RAG application

Once a research paper is available as Markdown, you can use the content in document search, summarization pipelines, or retrieval-augmented generation applications.

For RAG, preserving the source URL and page number alongside each passage is useful. It allows an application to identify where a retrieved answer came from rather than returning an unattributed block of text.

Spidra's include_pages=True option makes it straightforward to build page-aware records.

For example, add this code to the end of arxiv_to_markdown.py:

# Save individual pages as JSONL records.
pages_path = output_dir / f"{base_name}.pages.jsonl"

with pages_path.open("w", encoding="utf-8") as file:
    for page in doc.pages:
        record = {
            "text": page.markdown,
            "arxiv_id": paper_id,
            "page": page.page_number,
            "source_url": pdf_url,
        }
        file.write(
            json.dumps(record, ensure_ascii=False) + "\n"
        )

This creates a third file, 1706.03762v7.pages.jsonl, containing one JSON record per extracted page.

Each record connects the extracted text to the paper's arXiv identifier, the original PDF, and the source page.

From here, you can split longer pages into smaller chunks, generate embeddings, and store them in a vector database. Your retrieval layer can carry those metadata fields forward when presenting search results or generating citations.

For scientific content, retaining the original PDF link is also useful when an application needs to display the source equation, figure, or table alongside the extracted text.

If you're building a broader ingestion workflow, you can also use Spidra Search with the research source to discover papers from arXiv and other research repositories before passing their PDF links to Parse.

Process more arXiv papers

The example above processes one paper, but the same approach works for a collection of research papers.

For each paper, you can retrieve its metadata, submit its PDF URL to Spidra Parse, and save the Markdown under its arXiv identifier. Keeping the version in the filename also makes it easier to distinguish revisions in a recurring ingestion workflow.

If you already have multiple PDF links, Spidra's parse_batch() method can process up to 50 documents per batch. For workflows that need to process many documents, see our guide on batch processing PDFs with Python.

You can also estimate the pages and credits required with estimate_parse() before submitting a document. For plain Markdown extraction, Spidra charges one credit per parsed PDF page, including pages read using OCR.

For more information about PDF inputs, uploads, and reusing documents, see our guide to PDF URLs, uploads, and file IDs.

With Markdown files and source metadata stored together, you have the foundation for a repeatable research ingestion workflow that can support literature search, document analysis, and AI-powered research tools.

Convert your own arXiv papers with Spidra

You've now built a Python workflow that takes an arXiv paper, converts selected pages to Markdown, preserves mathematical expressions and scientific tables, and saves the result with its bibliographic metadata and page references.

You can use the same approach for other research papers or extend it into a larger document-processing pipeline.

Create a free Spidra account, get your API key, and run arxiv_to_markdown.py with a paper you want to process.