Co-founder

Joel Olawanle

Joel is a Software Engineer and Developer Advocate who builds product, create developer education, and help developers succeed through technical content and community.

Website X LinkedIn

Articles by Joel Olawanle

All articles

Python Requests vs Playwright vs a scraping API: Which should you use?

Compare Python Requests, Playwright, and scraping APIs using real websites, working code, and practical guidance on when to use each.

Joel OlawanleOctober 26, 2026 · 14 min read

How to handle untrusted web content safely in scraping and AI pipelines

Learn to handle untrusted web content safely in Python with URL validation, response limits, structured extraction, and prompt-injection defenses.

Joel OlawanleOctober 25, 2026 · 17 min read

HTTP2Shell (CVE-2026-111497): How to check if your Python web scraper is affected

Learn how HTTP2Shell affects Python web scrapers, check Brotli dependencies, and fix CVE-2026-111497 in Requests, HTTPX, and Scrapy.

Joel OlawanleOctober 24, 2026 · 14 min read

How to scrape Indeed jobs with Python

Learn how to scrape Indeed jobs with Python, handle blocked requests, collect structured listings from multiple searches, and export them to CSV.

Joel OlawanleOctober 24, 2026 · 14 min read

How to collect and compare PHP2Shell advisories with Python

Build a Python tool to find PHP2Shell security advisories, extract vendor patch information, and compare the results using Spidra.

Joel OlawanleOctober 23, 2026 · 15 min read

Web crawling vs. web scraping: What's the difference?

Learn the difference between web crawling and web scraping, when to use each, and how they work together to collect data from websites.

Joel OlawanleOctober 23, 2026 · 13 min read

How to map a website and scrape only the pages you need

Learn how to map a website, discover relevant URLs, filter unwanted pages, and scrape only the content you need using Python and Spidra.

Joel OlawanleOctober 15, 2026 · 13 min read

How to scrape multiple URLs in Python and handle failures

Learn how to scrape multiple URLs in Python using concurrent requests, retries, and error handling, then process failed pages with Spidra.

Joel OlawanleOctober 15, 2026 · 15 min read

How to Scrape PDFs and Extract Structured JSON with an API

Learn how to scrape PDFs from a URL, extract clean Markdown or structured JSON with a schema, and even search for PDF documents before scraping them with Spidra.

Joel OlawanleOctober 2, 2026 · 7 min read

Search API vs scraping API: when you need one, the other, or both

Learn the difference between a search API and a scraping API, when to use each one, and how search, scraping, crawling, browser actions, and extraction fit together in real web data workflows.

Joel OlawanleSeptember 30, 2026 · 5 min read

Introducing Spidra Search: Search the web before you scrape it

Spidra Search lets you find relevant pages before you scrape them. Search the web, news, images, research, and developer sources.

Joel OlawanleSeptember 30, 2026 · 7 min read

Get structured data from popular websites

Learn how to get structured data from popular websites like Amazon using a JSON Schema and AI prompt, no selectors or proxies required.

Joel OlawanleJuly 8, 2026 · 5 min read

How to scrape web data with Beautiful Soup: step-by-step guide in 2026

How to scrape web data with Beautiful Soup in Python. Covers requests, HTML parsing, CSS selectors, XPath, pagination, CSV and JSON export, and how to handle dynamic content.

Joel OlawanleJune 27, 2026 · 12 min read

How to scrape eBay with JavaScript and Node.js in 2026

How to scrape eBay with JavaScript and Node.js using the Spidra API. Covers item pages, search results, batch scraping, price monitoring, and CSV export with real tested output.

Joel OlawanleJune 27, 2026 · 14 min read

How to scrape eBay with the Spidra API (2026)

Complete guide to scraping eBay with the Spidra API. Covers item pages, search results, pagination, batch scraping 50 items in parallel, price monitoring, and error handling with real output.

Joel OlawanleJune 27, 2026 · 17 min read

How to scrape eBay product data with Python (2026)

How to scrape eBay product data with Python in 2026. Step-by-step BeautifulSoup tutorial covering price, images, specs, and seller details, plus a faster approach with real output and a price monitoring pipeline.

Joel OlawanleJune 27, 2026 · 13 min read

How to scrape Amazon with Selenium: step-by-step tutorial (2026)

Step-by-step tutorial for scraping Amazon product pages with Selenium in Python. Covers headless Chrome setup, extracting price, rating, and images, plus why Selenium struggles with Amazon at scale.

Joel OlawanleJune 26, 2026 · 10 min read

How to scrape Amazon with Python: step-by-step tutorial (2026)

Step-by-step tutorial for scraping Amazon product pages with Python in 2026. Covers BeautifulSoup selectors, search results, pagination, and working around blocks.

Joel OlawanleJune 26, 2026 · 14 min read

How to scrape Amazon reviews in 2026

How to scrape Amazon reviews in 2026 using the Spidra API and BeautifulSoup. Step-by-step guide covering anti-bot bypass, JSON schema, real output, and CSV export.

Joel OlawanleJune 26, 2026 · 13 min read

How to scrape Amazon product data with Python and Node.js (2026)

How to scrape Amazon product data with Python and Node.js in 2026. Covers product pages, search results, batch scraping 50 ASINs in parallel, and price monitoring.

Joel OlawanleJune 26, 2026 · 16 min read

How to crawl a website and extract data with the Spidra API

Crawl a site with one request and get typed JSON from only the pages you want. Real output, costs, the crawl report, and Python and Node.js examples.

Joel OlawanleJune 24, 2026 · 13 min read

Spidra browser actions: complete guide to clicking, scrolling, and interacting before scraping

Complete guide to Spidra browser actions. Learn how to click, scroll, type, and use forEach with real examples.

Joel OlawanleJune 23, 2026 · 15 min read

Spidra batch scraping API: how to scrape 50 URLs in parallel

Process up to 50 URLs in parallel with the Spidra batch scraping API. Covers AI extraction, JSON schema, retrying failures, and proxy routing.

Joel OlawanleJune 22, 2026 · 13 min read

What is loop engineering? The complete guide for developers

The complete guide to loop engineering. What it is, the 4-condition test before building one, the six building blocks, failure modes, and honest cost estimates.

Joel OlawanleJune 16, 2026 · 27 min read

HTML vs Markdown for AI: which format is better for LLMs?

Raw HTML wastes up to 90% of your LLM context window on boilerplate. Learn why Markdown is better for RAG and AI training, and when HTML is actually the right choice.

Joel OlawanleJune 12, 2026 · 12 min read

Spidra API Node.js tutorial: scrape any website with JavaScript and TypeScript

Scrape any website with Node.js using the Spidra API. AI extraction, JSON schema with Zod, browser actions, batch scraping, crawling, and Next.js examples included.

Joel OlawanleJune 12, 2026 · 16 min read

Spidra is live on Product Hunt today

We built Spidra to fix web scraping. No more broken selectors, proxy headaches, or CAPTCHA nightmares. Here's a short story behind what we shipped.

Joel OlawanleJune 11, 2026 · 5 min read

What is a headless browser? How it works, uses, and tools (2026)

Learn what a headless browser is, how it works, Playwright vs Puppeteer explained with real code, and when to use a managed API instead.

Joel OlawanleJune 11, 2026 · 15 min read

Spidra API Python tutorial: scrape any website with Python

Scrape any website with Python using the Spidra API. Covers AI extraction, JSON schema, browser actions, batch scraping, and crawling with working code examples.

Joel OlawanleJune 10, 2026 · 16 min read

How to scrape cf_clearance cookies from Cloudflare-protected websites

Learn how to extract cf_clearance cookies from Cloudflare-protected sites, use them in a requests session, and why the approach breaks at scale.

Joel OlawanleJune 10, 2026 · 10 min read

Spidra API tutorial: complete guide to web scraping with the Spidra API

Complete Spidra API guide. Scrape pages, extract structured JSON, run browser actions, and crawl entire websites with working Python and cURL examples.

Joel OlawanleJune 9, 2026 · 18 min read

What is AI scraping? How it works, use cases, and how to get started

Learn what AI scraping is, how it differs from traditional web scraping, common use cases, and how to extract structured data from any website in minutes.

Joel OlawanleJune 9, 2026 · 12 min read

How to scrape unstructured website data and turn it into structured JSON for AI

How to extract structured JSON from inconsistent web pages at scale, minimize token overhead, and build a vector DB search pipeline in Python.

Joel OlawanleJune 8, 2026 · 13 min read

5 best AI web scraper tools I've tested in 2026 (free + paid)

I tested 5 AI web scraper tools that actually work. Here is what each one is good for, where each one falls short, and which one I reach for depending on the job.

Joel OlawanleJune 5, 2026 · 9 min read

What is WebGL fingerprinting and how to bypass it when scraping

Learn how WebGL fingerprinting uses GPU hardware to identify scrapers, why it is harder to bypass than canvas fingerprinting, and what actually works at scale.

Joel OlawanleJune 4, 2026 · 10 min read

How to scale web scraping with Playwright BrowserContext

Learn how Playwright BrowserContext works, how to run concurrent sessions with asyncio, and what to do when local hardware becomes the bottleneck.

Joel OlawanleJune 3, 2026 · 11 min read

How to use pyppeteer_stealth for web scraping

Learn how pyppeteer_stealth patches Pyppeteer's bot signals, how to fix the Chromium setup error, and where it falls short against modern anti-bot systems.

Joel OlawanleJune 2, 2026 · 9 min read

How to use Puppeteer Cluster to scale up web scraping

Learn how to use puppeteer-cluster to run concurrent browser workers, when to use each concurrency model, and how to scale beyond local hardware limits.

Joel OlawanleJune 1, 2026 · 11 min read

How to use Ghost Cursor for web scraping

Learn how Ghost Cursor adds human-like mouse movement to Puppeteer, how every method works, and what to do when mouse movement alone is not enough.

Joel OlawanleMay 29, 2026 · 12 min read

Puppeteer real browser: How to avoid detection when scraping

Learn how Puppeteer Real Browser patches Puppeteer for stealth scraping, where it fails against real anti-bot protection, and what to use instead.

Joel OlawanleMay 28, 2026 · 9 min read

How to scrape with Patchright and avoid detection

Learn how Patchright patches Playwright for stealth scraping, where it falls short against real anti-bot protection, and what to use when you need more.

Joel OlawanleMay 27, 2026 · 8 min read

Web scraping with Scrapling: 2026 tutorial

Learn how to use Scrapling for web scraping with real code examples, and what to use when you need to scrape at scale or bypass stronger anti-bot protection.

Joel OlawanleMay 26, 2026 · 8 min read

How to bypass IP bans when web scraping in 2026

IP bans hit mid-run, not at the start. Learn why scrapers get blocked, how to rotate proxies and manage sessions, and when a managed API handles it better.

Joel OlawanleMay 25, 2026 · 12 min read

How to extract web data for AI training pipelines

Learn how to collect clean web data for AI training pipelines. Covers Markdown output, provenance metadata, anti-bot bypass, and scaling with Spidra's API.

Joel OlawanleMay 22, 2026 · 16 min read

How to scrape any website and get structured data with a single API call

Learn how to turn any URL into clean structured JSON using Spidra's API and official SDKs for Python, Node.js, Go, PHP, Ruby, and more.

Joel OlawanleMay 21, 2026 · 14 min read

Top 10 web scraping APIs for AI in 2026

Building AI apps that need web data? We ranked the 10 best scraping APIs on extraction quality, anti-bot bypass, browser actions, and true cost per page.

Joel OlawanleMay 20, 2026 · 19 min read

How I built a crypto data pipeline for 200+ tokens using Spidra

How I scraped yield and distribution history for 200+ crypto tokens using Spidra's Batch API and JSON Schema extraction.

Joel OlawanleMay 15, 2026 · 8 min read

Scraping business directories for lead generation with Spidra Node SDK

Learn how to scrape Yellow Pages across multiple pages, export leads to CSV, and enrich them with emails using the Spidra Node SDK. Real results included.

Joel OlawanleMay 7, 2026 · 13 min read

Web scraping with Node.js: a practical guide

Learn web scraping with Node.js. From Cheerio and Puppeteer to the Spidra SDK. Real working code and honest advice on when to use each tool.

Joel OlawanleMay 4, 2026 · 14 min read

Spidra SDKs and integrations: Node, Python, Go, PHP, Ruby, Rust, n8n, and more

Explore Spidra SDKs for Node.js, Python, Go, PHP, Ruby, and Rust, plus n8n integration for automation workflows and web data extraction.

Joel OlawanleApril 23, 2026 · 7 min read

What's new in Spidra: forEach actions, structured output, batch scraping, and more

Checkout the new forEach actions, structured output, batch scraping, authenticated scraping, and more shipped to Spidra.

Joel OlawanleApril 15, 2026 · 7 min read

7 best Firecrawl alternatives for AI web scraping in 2026

This guide covers the 7 best options, from open-source Python libraries to fully managed AI scraping platforms, with real pricing and honest trade-offs for each.

Joel OlawanleApril 15, 2026 · 11 min read

Navigating website anti-scraping: Strategies for robust data extraction

This guide details techniques like IP rate limiting, User-Agent analysis, CAPTCHAs, and JavaScript rendering, offering practical strategies for effective data extraction.

Joel OlawanleMarch 5, 2026 · 7 min read

7 essential use cases for web scraping data extraction

Discover seven transformative use cases of web scraping, from market analysis and AI development to SEO optimization, and learn how to leverage data for business growth.

Joel OlawanleMarch 4, 2026 · 9 min read

Essential practices for effective web scraping

Master the art of web scraping with essential best practices. Learn to build robust, efficient, and ethical data extraction tools, from IP rotation to handling dynamic content.

Joel OlawanleMarch 3, 2026 · 12 min read

You build the product.
Spidra handles the data work.

Start with 300 free credits, about 300 pages. No credit card required.

Using an AI agent? Connect the MCP server