Scraping#MCP#Web scraping#Extraction

Firecrawl: a self-hostable web data API for agents

Firecrawl bundles scrape, crawl, sitemap and search behind one API, and now runs as a single Rust binary on your machine. 205,483 stars, AGPL-3.0 with MIT SDKs.

Project facts

GitHub Ecosystem
Repositorygithub.com/firecrawl/firecrawl
License
AGPL-3.0
Language
TypeScript
Stars
205,483
Data checked
2026-11-21

Snapshot figures reflect the check date and may change over time.

GitHub repository card for firecrawl/firecrawl: the repo name and "knowledge infrastructure for agents" description, a flame icon, and counts for contributors, dependents, discussions, stars and forks

Feeding an agent a web page is easy until you have to keep the pipeline alive: something has to run JavaScript, follow pagination, strip navigation, and hand back text a model can read. Firecrawl packages that pipeline as one v2 API — /v2/scrape for a single page, /v2/crawl for a site, /v2/map for its URLs, /v2/search for results with content attached. The firecrawl/firecrawl repository was created on 2024-04-15 and had 205,483 stars and 10,745 forks on 2026-11-21, written mostly in TypeScript and licensed AGPL-3.0. It was sixth on GitHub’s daily trending page that day, with 1,241 stars added in 24 hours.

GitHub's repository card for firecrawl/firecrawl, showing the repo name, description and 205k stars

Key points

  • Three ways to run it, not one product: Firecrawl Local is a single Rust binary, published to crates.io as firecrawl-local 0.1.0 on 2026-11-11 and 0.1.1 on 2026-11-19. The full self-hosted server still lives in apps/api behind Docker Compose, and the cloud service runs at firecrawl.dev.
  • The data library is cloud-only: Firecrawl’s README puts Alexandria at 550+ data providers, 250M+ people, 75M+ companies and 41M+ research papers. None of that ships with the self-hosted server.
  • Two licenses stacked: the main repo is AGPL-3.0, while the SDKs and some UI components are MIT (npm firecrawl 4.54.0, PyPI firecrawl 4.57.0, checked 2026-11-21).
ItemValue (checked 2026-11-21)
Stars / forks205,483 / 10,745
LicenseAGPL-3.0; MIT for the SDKs
Primary languageTypeScript (Local is written in Rust)
Created / last push2024-04-15 / 2026-11-21
Open issues / PRs128 / 762
Latest GitHub releasev2.11.0 (2026-06-19)
Latest tagv2.11.809 (committed 2026-11-20)

Core features

The feature list splits cleanly along one line — what runs on your machine and what needs Firecrawl’s servers — and that line decides whether self-hosting is worth it.

  • Single-page scrape: scrape returns markdown, html, rawHtml or links, and parses PDFs and Office documents locally too. The cloud version adds JSON extraction, summaries and browser actions, which Local skips with a warning; screenshots need Chrome installed locally.
  • Crawl and map: crawl takes --limit (100 by default), --max-depth, --include/--exclude path regexes and --entire-domain. map collects URLs from sitemaps and the start page, and --search filters them, which saves time when you need a candidate list for site search.
  • No separate API to learn: firecrawl-local serve answers the Firecrawl v2 API on port 3002. Point an SDK at it with api_url / apiUrl and existing code runs unchanged; local routes need no key.
  • One command into your editor: firecrawl-local init finds Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI and Codex, backs up each config before touching it, and supports --dry-run so you can read the diff first.
  • Search and data providers go to the cloud: Local has no search index, so /v2/search and Alexandria calls are forwarded to Firecrawl Cloud. Keyless search is free at 1,000 credits a month without signup, rate-limited.

Three ways to run

All three expose the same API surface; they differ in who carries the browser, the queue and the index.

OptionRuns onNeedsMissing
Firecrawl LocalOne binary on your machineRust and a C compiler; optionally Chrome and Node 22+ for MCPSearch, JSON extraction, Interact, Agent; screenshots need Chrome
Cloudfirecrawl.devNothing to try keyless; a key for full quotaPage content leaves your machine
Full self-hostYour own DockerCompose v2, Playwright, Redis, RabbitMQ, PostgreSQLScreenshots and page actions need Fire-engine; no Alexandria

The official self-hosting guide is blunt about the full stack: it is a “trusted-network quickstart”, not a production architecture. It pins v2.11.162, sets USE_DB_AUTHENTICATION=false — that is, no authentication — and lists what is absent: durable storage, TLS, high availability, and any verified minimum host size.

Self-hosting requirements

The full server is not a single docker run. Per SELF_HOST.md, Compose starts the API and workers, Playwright, Redis, RabbitMQ and NuQ PostgreSQL, plus an optional FoundationDB queue backend. Only the API’s port 3002 is published; the database, cache and broker stay on the internal network.

That stack covers scrape, crawl, map and search, with real gaps. Screenshots and page actions need Fire-engine, which is not in the default stack, and Agent, Interact, Browser, feedback and the audio/video formats are cloud-only. The guide also notes the Compose file defines no persistent volumes for PostgreSQL, Redis or RabbitMQ — replace a container and the data goes with it — and that the unauthenticated API must not be exposed publicly.

Local is the low-effort path. It has no database, queue or container, and the costs are listed in apps/local: the json and summary formats are skipped, browser actions are ignored, crawl and batch jobs live in memory and die when serve stops, and the v0 and v1 routes are not served.

Typical use cases

  • Give a coding agent current docs: point Cursor or Claude Code at a library’s latest documentation. After init, the MCP tools work, and fetching and cleaning happen on your machine.
  • Turn a docs site into RAG material: crawl https://docs.example.com --limit 200 --out ./pages writes one markdown file per page, which removes the boilerplate-stripping step before chunking.
  • Teams that cannot ship page content out: finance and healthcare teams that will not hand URLs and text to a third party can run Local with FIRECRAWL_CLOUD=off, so a blocked page fails loudly instead of quietly going to the cloud.
  • Work that needs structured fields: prices, job posts, SKUs — anything that wants JSON — is beyond Local. Use the cloud, or self-host and connect an OpenAI-compatible endpoint.

Quick start

On macOS, install the compiler toolchain first, then the binary from crates.io:

xcode-select --install          # Debian / Ubuntu: build-essential

cargo install firecrawl-local
firecrawl-local scrape https://example.com --markdown
firecrawl-local map https://docs.firecrawl.dev --limit 20
firecrawl-local serve           # http://localhost:3002
firecrawl-local init --dry-run  # see which config files would change

If you would rather not install Rust, the documented alternative is npx -y firecrawl-cli@latest scrape https://firecrawl.dev, which goes through the cloud. With serve running, a curl POST to /v2/scrape returns the same JSON shape as the hosted API; the Python SDK only needs api_url="http://localhost:3002", and local routes need no key. The MCP layer is an npm package and needs Node.js 22 or newer.

Outside tests and doubts

On 2026-10-23, Tech Stackups ran 40 searches, 40 page fetches and two site crawls across Firecrawl, Exa, Tavily and DeepSeek, and the verdict split. On fetching, Firecrawl won: 37 of 40 pages came back usable, every PDF worked, and it was the only tool that crawled a documentation site reliably. On search it lost: asked about Python 3.15.1, it returned pages still naming 3.15.0 and 3.14, while Exa found the release. The tester also recorded the cost — one crawl cost 25 credits, and the free allowance ran out fast.

Hacker News is blunter. In a 2026-03-30 thread titled “I replaced Firecrawl with 2,700 lines of Elixir”, the author said he was paying about $190 a month, that it worked but “felt half baked”, and that the CLI was awkward. In an August 2026 Show HN for a single-binary Rust alternative, the maintainer framed the choice as paying for an expensive API or running a fleet of headless Chrome instances that eat 1GB of RAM per page and still get blocked by Cloudflare.

Both complaints point at the same thing: Firecrawl sells the engineering you skip — browser fleet, cleaning, retries — and the price includes it. The company’s side of that trade is in its own blog: on 2026-09-22 it announced a $75M Series B led by Smash Capital, with part of the money going to pay data providers and expand Alexandria. The split between the open repo and the hosted service makes more sense in that light.

Local answers the “half baked” critique directly, since one binary now runs the same cleaning code as the hosted API. Keeping search in the cloud is also an admission: that index is not something a laptop can hold.

License and compliance

The main repository is AGPL-3.0; the SDKs and some UI components are MIT. The split matters in practice. Calling the SDK from a closed-source product is fine under the MIT layer. Modifying the server code and offering it as a network service triggers AGPL section 13, which requires offering the corresponding source to those users. If you plan to embed the cleaning logic in a commercial product, check which layer you are changing.

The scraping boundary is stated in two places. The README says Firecrawl respects robots.txt by default and puts the responsibility for site policies and terms of use on the end user. The Local docs add that it identifies itself as FirecrawlLocal in its User-Agent and reads robots.txt when crawling. The defaults lean compliant; the caller still owns the rest — read the target’s terms, keep concurrency and request rates sane, and do not treat logged-in private pages as public data.

One default is easy to miss: pages Local cannot fetch are handed to Firecrawl Cloud and marked "source": "firecrawl-cloud" in the result. Teams with data-residency rules should either set FIRECRAWL_CLOUD=off or document the fallback explicitly.

Summary

Firecrawl fits teams already feeding web pages to agents who would rather not run a Playwright cluster, and individual developers who just want a few docs sites as markdown: cargo install, one scrape, results in five minutes. It fits two needs poorly. If you want a search index, Exa beat it in independent testing. If nothing may leave your network, search and the blocked-page fallback both point at the cloud.

The license is AGPL-3.0 with MIT SDKs, the primary language is TypeScript, and the repo had 205,483 stars on 2026-11-21. We covered its MCP wrapper in Firecrawl MCP, which is about wiring the tools into a chat; this entry is about running the server yourself. For other self-hosted collection tools, AnyCrawl is a much smaller TypeScript project that switches engines per page, and Obscura is worth a look if you want a browser engine that does not depend on Chromium.

Three weak spots. Release cadence is uneven: GitHub releases stop at v2.11.0 (2026-06-19), and after that there are only tags up to v2.11.809, so pinning a version means picking a tag or commit yourself. With 128 open issues and 762 unmerged pull requests, outside contributions queue for a long time and review capacity looks like the bottleneck. And the open repo increasingly reads as the front door to a paid service: search, structured extraction, screenshots and the Agent stay in the cloud, leaving scrape, crawl and map as the genuinely local parts.