# Web scraping and crawling

14 tools for agents in web scraping and crawling: APIs, official MCP servers and Actors. Family: Web and data access.

Tasks: [Compare products](https://agentready-tools.com/task/compare-products.md), [Search the web](https://agentready-tools.com/task/search-web.md), [Summarize a webpage](https://agentready-tools.com/task/summarize-webpage.md), [Search public social posts](https://agentready-tools.com/task/search-social-posts.md), [Browse and act on a website](https://agentready-tools.com/task/browse-website.md).

## Compare

| Tool | MCP | Pricing | Jurisdiction | Agent-readiness |
|---|---|---|---|---|
| [Agent Payment Checker — x402, MPP, L402, UCP & AP2 Detector](https://agentready-tools.com/tools/agentpay-checker) | No | Not yet verified | - | Not enough data |
| [Apify MCP server](https://agentready-tools.com/tools/apify-mcp) | Yes | Not yet verified | - | Not enough data |
| [Cheerio Scraper](https://agentready-tools.com/tools/apify-cheerio-scraper) | No | Not yet verified | - | Not enough data |
| [Exa MCP](https://agentready-tools.com/tools/exa-mcp) | Yes | Not yet verified | - | Not enough data |
| [Firecrawl MCP server](https://agentready-tools.com/tools/firecrawl-mcp) | Yes | Not yet verified | - | Not enough data |
| [Oxylabs E-Commerce Scraper API](https://agentready-tools.com/tools/oxylabs-ecommerce-scraper-api) | No | $5 in welcome credits for new accounts | - | Not enough data |
| [Oxylabs Web API](https://agentready-tools.com/tools/oxylabs-web-api) | Yes | $5 in welcome credits for new accounts | - | Not enough data |
| [Puppeteer Scraper](https://agentready-tools.com/tools/apify-puppeteer-scraper) | No | Not yet verified | - | Not enough data |
| [RAG Web Browser](https://agentready-tools.com/tools/apify-rag-web-browser) | No | Not yet verified | - | Not enough data |
| [ScraperAPI](https://agentready-tools.com/tools/scraperapi) | No | Not yet verified | - | Not enough data |
| [Smart Article Extractor](https://agentready-tools.com/tools/lukaskrivka-article-extractor-smart) | No | Not yet verified | - | Not enough data |
| [Web Scraper](https://agentready-tools.com/tools/apify-web-scraper) | No | Not yet verified | - | Not enough data |
| [Website Content Crawler](https://agentready-tools.com/tools/apify-website-content-crawler) | No | Not yet verified | - | Not enough data |
| [Website Screenshot API — URL to PNG/JPEG/PDF for AI Agents](https://agentready-tools.com/tools/url-screenshot) | No | Not yet verified | - | 60 / 100 |

Comparison fields come from the catalog listing. "Not yet verified" and "Not enough data" mean we have no checked value yet; nothing is guessed.

## Tools

- [Agent Payment Checker — x402, MPP, L402, UCP & AP2 Detector](https://agentready-tools.com/tools/agentpay-checker): Check which AI-agent payment protocols a URL supports: x402, MPP (Machine Payments Protocol), L402, UCP and A2A/AP2 discovery files.
- [Apify MCP server](https://agentready-tools.com/tools/apify-mcp) (MCP): Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store.
- [Cheerio Scraper](https://agentready-tools.com/tools/apify-cheerio-scraper): Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code.
- [Exa MCP](https://agentready-tools.com/tools/exa-mcp) (MCP): Connect AI agents to Exa for web search, content fetching, and multi-step research.
- [Firecrawl MCP server](https://agentready-tools.com/tools/firecrawl-mcp) (MCP): Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.
- [Oxylabs E-Commerce Scraper API](https://agentready-tools.com/tools/oxylabs-ecommerce-scraper-api): Scraper API for collecting public e-commerce product, search, pricing, and seller data, returned as HTML or structured JSON.
- [Oxylabs Web API](https://agentready-tools.com/tools/oxylabs-web-api) (MCP): Web scraping API that returns structured data from sites, including JavaScript-heavy pages, with an official MCP server for AI agents.
- [Puppeteer Scraper](https://agentready-tools.com/tools/apify-puppeteer-scraper): Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code.
- [RAG Web Browser](https://agentready-tools.com/tools/apify-rag-web-browser): Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their…
- [ScraperAPI](https://agentready-tools.com/tools/scraperapi): Web scraping API that handles proxies, browsers and CAPTCHAs.
- [Smart Article Extractor](https://agentready-tools.com/tools/lukaskrivka-article-extractor-smart): 📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click.
- [Web Scraper](https://agentready-tools.com/tools/apify-web-scraper): Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function.
- [Website Content Crawler](https://agentready-tools.com/tools/apify-website-content-crawler): Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines.
- [Website Screenshot API — URL to PNG/JPEG/PDF for AI Agents](https://agentready-tools.com/tools/url-screenshot): Screenshot any public URL to PNG, JPEG or PDF with Chromium. Predictable JSON output with a direct image URL and explicit errors, built for AI agents, API…

JSON: https://agentready-tools.com/categories/web-scraping-crawling.json. MCP: `search_tools({"category":"Web pages and crawlers"})`. All categories: https://agentready-tools.com/categories.md. Updated 2026-11-22.
