Web scraping and crawling

14 tools for agents in web scraping and crawling: APIs, official MCP servers and Actors.

Tasks: Compare products · Search the web · Summarize a webpage · Search public social posts · Browse and act on a website

Compare

ToolMCPPricingJurisdictionAgent-readiness
Agent Payment Checker — x402, MPP, L402, UCP & AP2 DetectorNoNot yet verified-Not enough data
Apify MCP serverYesNot yet verified-Not enough data
Cheerio ScraperNoNot yet verified-Not enough data
Exa MCPYesNot yet verified-Not enough data
Firecrawl MCP serverYesNot yet verified-Not enough data
Oxylabs E-Commerce Scraper APINo$5 in welcome credits for new accounts-Not enough data
Oxylabs Web APIYes$5 in welcome credits for new accounts-Not enough data
Puppeteer ScraperNoNot yet verified-Not enough data
RAG Web BrowserNoNot yet verified-Not enough data
ScraperAPINoNot yet verified-Not enough data
Smart Article ExtractorNoNot yet verified-Not enough data
Web ScraperNoNot yet verified-Not enough data
Website Content CrawlerNoNot yet verified-Not enough data
Website Screenshot API — URL to PNG/JPEG/PDF for AI AgentsNoNot yet verified-60 / 100

Comparison fields come from the catalog listing. "Not yet verified" and "Not enough data" mean we have no checked value yet; nothing is guessed. Method.

Tools

Apify MCP server

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store.

MCP

Cheerio Scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code.

API or Actor

Exa MCP

Connect AI agents to Exa for web search, content fetching, and multi-step research.

MCP

Firecrawl MCP server

Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.

MCP

Oxylabs E-Commerce Scraper API

Scraper API for collecting public e-commerce product, search, pricing, and seller data, returned as HTML or structured JSON.

API or Actor

Oxylabs Web API

Web scraping API that returns structured data from sites, including JavaScript-heavy pages, with an official MCP server for AI agents.

MCP

Puppeteer Scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code.

API or Actor

RAG Web Browser

Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their…

API or Actor

ScraperAPI

Web scraping API that handles proxies, browsers and CAPTCHAs.

API or Actor

Smart Article Extractor

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click.

API or Actor

Web Scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function.

API or Actor

Website Content Crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines.

API or Actor

Agents: search_tools({"category":"Web pages and crawlers"}) · JSON · All categories · updated 2026-11-22.