Skip to main content
TutorialsNovember 18, 2026 · 10 min read

How to give Claude Code web search and scraping capabilities with MCP

Connect Claude Code to Spidra MCP to search the web, scrape JavaScript pages, and extract structured JSON with real examples.

How to give Claude Code web search and scraping capabilities with MCP

Claude Code has built-in tools for working with code and retrieving information from the web. But when you need to extract structured data from JavaScript-rendered pages or combine search and scraping in one workflow, a dedicated scraping service can help.

You can connect Spidra to Claude Code through the Model Context Protocol (MCP). This lets Claude Code search for webpages, retrieve their content, and extract structured records without requiring you to write a separate scraper.

This tutorial shows how to set up Spidra MCP, search for official Python documentation, and extract JSON from a JavaScript-rendered website using real examples.

How MCP connects Claude Code to web data

Model Context Protocol (MCP) is an open standard that allows AI applications to connect to external tools and data sources.

Claude Code supports MCP servers, which expose tools it can call while working on a task.

Claude Code also has built-in web search and fetching tools. Spidra MCP adds a dedicated web data service for workflows that need search, JavaScript-rendered page retrieval, or structured extraction.

These capabilities serve different purposes:

  • Search: Find relevant webpages when you don't know their URLs.

  • Scrape: Retrieve content from a specific webpage, including pages that require JavaScript rendering.

  • Structured extraction: Extract requested information into fields your application can use.

You can combine them in a single coding session. For example, Claude Code can search for documentation, retrieve a relevant page, and return structured information from its contents.

Connect Spidra MCP to Claude Code

You'll need Claude Code installed, a Spidra account, and an internet connection.

Spidra provides a hosted MCP server, so you don't need to install or maintain a separate scraping server on your computer.

Step 1: Add the Spidra MCP server

Open your terminal and run:

claude mcp add --transport http spidra https://mcp.spidra.io/mcp

This registers Spidra as a remote MCP server in Claude Code.

The command uses HTTP transport, which Claude Code recommends for remote MCP servers.

By default, Claude Code saves the configuration with local scope, making the server available in your current project.

To use Spidra across your projects, add --scope user when registering the server.

Step 2: Authenticate your Spidra account

Run:

claude mcp login spidra

Follow the authentication flow in your browser and authorize access to your Spidra account.

You can also complete authentication from inside Claude Code.

Start a session:

claude

Then enter:

/mcp

Select Spidra and follow the sign-in instructions if authentication is required.

Spidra's hosted MCP server supports browser-based authentication, so you don't need to manually add an API key for this setup.

Step 3: Verify the connection

Run:

claude mcp list

This displays your configured MCP servers and their connection statuses.

Look for Spidra in the list. A healthy connection should show Connected.

You can also inspect the configuration directly:

claude mcp get spidra

If the server is registered but requires authentication, complete the login process before using its tools.

For additional configuration options, see the Claude Code MCP documentation.

Once connected, you can start using Spidra through ordinary prompts.

Search for Python documentation using Spidra

Suppose you're building a Python application and need to understand how HTTP request timeouts work in the Requests library.

You could manually search the web and paste documentation into Claude Code. Instead, ask Claude Code to search for the relevant pages.

Use this prompt:

Use Spidra Search to find the official Python Requests documentation about HTTP timeouts. Return the top 5 results with their titles, URLs, and descriptions. Show the Spidra tool used and the actual search results. Do not generate or invent results.

The captured Claude Code response identified spidra_search as the tool and reported the query:

Python Requests documentation timeouts

It requested five results from the web source without a domain filter.

What did Spidra Search return?

The returned results contained five pages:

#

Result

Source

1

Timeout in requests.get

Python Discussions

2

Session Objects

Requests documentation

3

Timeout for Python requests.get() entire response

Stack Overflow

4

Quickstart

Requests documentation

5

Guide to Handling Python Requests Timeout

Oxylabs

Two results came from the official Requests documentation website.

The first was the Advanced Usage documentation, which explains session behavior and request timeout configuration.

The second was the Requests Quickstart, which covers timeout settings and explains that a timeout doesn't necessarily limit the entire response download.

The remaining results were relevant to the search query but came from third-party sources.

No search errors were reported.

image(20261118-152143).png

image (40).png

The result illustrates a useful distinction: a webpage can be relevant to your query without being an authoritative source.

For development tasks that depend on official documentation, you should verify the returned URLs before using their contents.

Filter search results to official sources

The original search didn't include a domain filter, which explains why it returned documentation alongside discussions and blog posts.

Spidra Search supports including and excluding domains.

For a more targeted search, you can ask Claude Code:

Use Spidra Search to find documentation about HTTP request timeouts. Apply the available domain filter to requests.readthedocs.io and return only results from that domain. Verify the hostname of each returned URL.

This is a follow-up prompt to try, not a second completed test.

Domain filtering helps narrow the results, but you should still verify that the returned pages contain the information your application needs.

For more search options, see our guide to Spidra Search.

Scrape a JavaScript-rendered page with Claude Code

Search helps you discover URLs. But when you already know the webpage, you can retrieve its contents directly.

Let's use Quotes to Scrape, a demonstration website designed for practicing web scraping.

The JavaScript version loads its quote content through the browser, making it useful for demonstrating rendered-page extraction.

Ask Claude Code:

Use Spidra to scrape https://quotes.toscrape.com/js/ and extract the first 10 quotes with their authors as structured JSON. Show which Spidra tool you used, the actual returned records, any errors, and the number of complete records. Don't invent missing information.

Claude Code identified spidra_scrape as the tool used.

Its response reported browser rendering with scrapeMode: "default" and JSON extraction using the requested fields.

Extract quotes as structured JSON

For an application, retrieving the page content is only part of the task.

A quotes application might need each quote's text and author as separate fields that can be stored in a database.

The extraction request specified two fields:

  • author: the author of the quote.

  • text: the quote itself.

The result contained 10 records, each with both fields populated.

Here's an excerpt from the returned JSON:

{
  "quotes": [
    {
      "author": "Albert Einstein",
      "text": "“Try not to become a man of success. Rather become a man of value.”"
    },
    {
      "author": "André Gide",
      "text": "“It is better to be hated for what you are than to be loved for what you are not.”"
    }
  ]
}

The complete output contained 10 quote records.

All 10 had nonempty author and text fields. Claude Code reported no scraping errors.

image(20261118-151848).png

The result demonstrates how structured extraction can turn content from a JavaScript-rendered page into records an application can process.

It also reveals something worth checking before storing the data.

Normalize extracted values before using them

One record returned an encoded author name:

André Gide

The intended name is:

André Gide

The JSON structure was valid, and the author field was populated, but the value still needed normalization.

Python's built-in html module can decode HTML entities.

Run:

from html import unescape

author = "André Gide"
print(unescape(author))

Output:

André Gide

You can apply the same normalization when processing the full collection of extracted records.

For application data, also check for missing fields, duplicate records, unexpected types, and values that don't match the source page.

The quotes example returned 10 structurally complete records. That doesn't independently establish that every extracted value was correct.

If your application depends on consistent fields, see our guide to extracting structured data from websites.

Combine web search and scraping in one workflow

The previous examples used Search and Scrape separately. You can also combine them in one Claude Code task.

Suppose you're building a reference application for Python HTTP clients.

You want to collect documentation about request timeouts from Requests, HTTPX, and aiohttp, but you don't have the relevant URLs.

Give Claude Code this prompt:

I'm building a reference for Python HTTP clients.

Use Spidra to find the official documentation for Requests, HTTPX, and aiohttp.

For each library:

  1. Find its official documentation website.

  2. Locate the page explaining HTTP request timeouts.

  3. Scrape the relevant page.

  4. Extract the documented timeout behavior.

  5. Include the source URL with each record.

  6. Return the information as structured JSON.

Use only the retrieved documentation. If a value cannot be verified, return null instead of guessing.

Claude Code can use Spidra Search to discover relevant documentation pages, followed by Scrape to retrieve and extract their content.

The resulting records can then be used to generate application code, create a reference file, or populate a development dataset.

This combined workflow is an additional example to try. It wasn't part of the two completed demonstrations above.

When should you use Search, Scrape, or Crawl?

Choose the operation based on what you already know about the target.

Use spidra_search when you have a query but no URL.

Use spidra_scrape when you already know the webpage and need its content or structured fields.

For several known URLs, spidra_batch_scrape can process them together.

If you need to discover URLs within a website, spidra_map can help identify them before scraping. For broader website extraction, spidra_crawl can discover and process relevant pages.

You don't need to crawl an entire documentation website when you only need two pages.

Our guide to mapping websites before scraping selected pages covers this workflow in more detail.

Fix common MCP and scraping issues

Claude Code cannot find Spidra

Check your configured MCP servers:

claude mcp list

If Spidra isn't listed, register it again using the setup command.

If it is listed but disconnected, inspect its status using /mcp inside Claude Code.

Spidra needs authentication

Run:

claude mcp login spidra

Complete the browser authentication flow.

Avoid placing API keys in Claude Code prompts or committing credentials to a repository.

Search returns unrelated or nonofficial results

Narrow the query and use available domain filters when necessary.

Verify the source URLs before treating search results as authoritative documentation.

Scraping returns incomplete content

Check whether the target page requires JavaScript rendering, browser interactions, or authentication.

Spidra supports browser rendering and interactions, but successful extraction isn't guaranteed for every website.

Inspect the returned content and error information before retrying.

Claude Code answers without using Spidra

Make the tool requirement explicit:

Use Spidra Search for this task. Show the retrieved source URLs and identify anything you couldn't verify.

If Claude Code cannot access Spidra, check the MCP connection instead of accepting an answer that hasn't been retrieved from the requested source.

When should you use MCP instead of a direct scraping API?

MCP is useful when Claude Code needs to choose and invoke tools while working on a larger development task.

For example, you might ask it to discover documentation, retrieve the relevant pages, and create application code using the extracted information.

A direct scraping API may be simpler when the task is predictable.

If your backend already knows which URLs to retrieve and which fields it needs, calling the API directly avoids requiring an AI assistant to coordinate each request.

You can use MCP during development to explore a workflow and establish the required data structure, then use the Spidra API or SDK for recurring production operations.

Also treat retrieved webpage content as untrusted data. Websites may contain instructions intended to influence an AI assistant, so don't let scraped content override your application's instructions or permissions.

Before scraping third-party websites, review their access restrictions, applicable terms, and privacy requirements. Our web scraping legal guide covers the main considerations.

Start using Spidra MCP with Claude Code

Connecting Spidra through MCP lets Claude Code discover webpages, retrieve rendered content, and extract structured information as part of a coding workflow.

The examples in this tutorial show both sides of that process. Search returned relevant documentation and third-party sources for a Python question, while Scrape returned structured quote records from a JavaScript-rendered website.

Use Search when you need to discover pages, Scrape when you know the URL, and validate the returned values before relying on them.

Create a Spidra account and connect the hosted MCP server to try these examples with websites you're permitted to access.

Frequently asked questions

Yes. Claude Code has built-in web search and fetching tools in supported environments. MCP allows you to connect additional services such as Spidra for dedicated web search, scraping, and structured extraction workflows.

Share this article

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.