Siftdog vs Firecrawl vs crw: self-hosted web search for AI agents

In short: all three are self-hostable and ship an MCP server. Pick Siftdog for a Tavily-compatible API under MIT, Firecrawl or crw when you need JavaScript-rendered pages.

At a glance

Facts from each project’s own repository and docs, September 2026.

When to pick each

Siftdog if you already use Tavily or its SDKs and want the same API on your own servers, or you need a permissive MIT license. It’s a small Python service; search comes from SearXNG, so there’s no search API key.

Firecrawl if you need a full scraping platform: JavaScript-heavy sites, browser interaction before extraction, and a large feature set. It’s the most widely used of the three.

crw if you want JavaScript rendering in a single binary of about 6 MB, and AGPL fits your plans.

How we compare with Tavily

We benchmarked Siftdog against Tavily itself on 45 queries: both found all 25 factual answers in advanced mode, and Siftdog returned official documentation for 9 of 10 technical questions. Full results and caveats.

Try Siftdog

git clone https://github.com/khsarvar/siftdog
cd siftdog
docker compose up

Or add it to Claude Code: one-command MCP setup.