Crawl4AI

For self-hosting you install the Python package and run the crawler against your own infrastructure under Apache-2.0. For the hosted API you mint a key, which gets 20 free credits with no signup at all and 1,000 free credits a month once you verify an email address, and then call one of five endpoints with a bearer token: /scrape turns a URL into Markdown, /extract takes a URL and a plain-English ask and returns typed JSON, /search returns ranked results as JSON with a rich=1 flag for follow-ups, /serp returns Google's results page as structured data with true ranks localised to a place, and /answer returns a direct answer with its sources.

Billing on the hosted service is in credits where one credit costs $0.001. The page prices each operation: a page is 0.5 credits, a search is 1, a result from their archive is 0.25, a real browser render is 1 and a hard site is 2. Plans carry a monthly credit allowance plus rate and concurrency limits, and extra credits beyond the allowance are sold at a per-plan rate that falls as the plan rises. Purchased extra credits do not expire. The same key works from curl, from an MCP client through a one-line claude mcp add command, or by pointing a coding agent at the project's llms.txt and letting it configure itself.

• •Markdown Built For Prompts - /scrape returns a page as clean Markdown with boilerplate removed, which is the format a model actually wants rather than raw HTML.
• •Selector-Free Structured Extraction - /extract takes a URL and a description of what you want and returns typed JSON records, so an extraction survives a layout change that would break a CSS selector.
• •Search and SERP as Data - /search gives ranked results as JSON; /serp returns Google's results page as structured data with true ranks and every block, localised to a place, aimed at rank tracking. /serp is marked Beta on the page.
• •Answers With Sources - /answer returns a direct answer carrying its citations. The site marks this one Experimental.
• •Self-Host With No Gate - The crawler is Apache-2.0. pip install it and run it on your own boxes and proxies with no key, no token and no account.
• •Agent and MCP Wiring - One MCP command registers every endpoint as an agent tool, and the published llms.txt lets a coding agent set itself up from a single line of chat.

Developers building retrieval pipelines, RAG systems and agents that need the open web as input, and teams who want the option to run the crawler themselves rather than depend on a vendor. The self-hosted path is the strongest argument here: there is a real, unrestricted, permissively licensed crawler underneath, and the hosted plan is a convenience you can walk away from. Rank trackers and SEO tooling are the obvious audience for /serp, and the free tier at 1,000 credits a month is enough to evaluate the API properly before paying anything.

It is a poor fit if you need price stability. The vendor says in writing that these prices and the free credit can change while they learn, so a quote taken today has a short shelf life. It is also a poor fit if you need an SLA on the newer endpoints, because /serp and /answer are marked Beta and Experimental by the vendor rather than by us. Teams who want a managed crawler with a long track record and a settled price list are better served elsewhere, and teams who simply want the crawler can take the open source one and skip the service entirely.