Web search for AI agents, on your own servers.

Siftdog is a self-hosted, Tavily-compatible API to search, extract, crawl and map the web. No search API key, no per‑search bill.

Self-host free →Join hosted waitlist

Python · Docker · MCP · works with the Tavily SDK

terminal

$ claude mcp add siftdog -- uvx siftdog mcp
Added stdio MCP server siftdog with command:
uvx siftdog mcp to local config

$ claude mcp get siftdog
siftdog:
  Status: ✔ Connected
  Type: stdio

> Use siftdog to find what's new in Python 3.14
  Called siftdog

Works with

Claude Code
Cursor
Claude Desktop
tavily-python
LangChain

Features

Everything an agent needs from the web

The four Tavily endpoints, an MCP server and the safety checks for running it in public.

Search

Ranked web results through SearXNG, with no search API key. BM25 relevance blended with the engines' order.

Extract

Clean markdown or text from up to 20 pages per request, with trafilatura.

Crawl & map

Follow a site's links with depth, breadth, limit and regex filters, or just list its URLs.

MCP server

Four read-only tools for Claude Code, Cursor and other clients, over stdio or HTTP at /mcp.

Tavily-compatible

Same endpoints and response shapes. The official tavily-python SDK and LangChain work unchanged.

Safe to expose

API keys, plus private-address blocking on every redirect and at connect time (stops DNS rebinding).

Drop-in

Switch from Tavily by changing one line

Same /search, /extract, /crawl and /map endpoints, same request and response fields. Point the official client at your server and keep your code.

Tested with tavily-python 0.8.4 (search, extract, crawl, map) and langchain-tavily 0.2.18 (search, extract).

Python

from tavily import TavilyClient

client = TavilyClient(
    api_key="your-siftdog-key",
    api_base_url="http://localhost:8000",
)
results = client.search("latest python release", search_depth="advanced")

MCP

Give your agent web tools in one command

Four read-only tools: siftdog_search, siftdog_extract, siftdog_crawl and siftdog_map. Listed in the official MCP Registry as io.github.khsarvar/siftdog.

Step-by-step guide for Claude Code →

Run locally over stdio with uv:

claude mcp add siftdog -- uvx siftdog mcp

Or connect to a running server over HTTP:

claude mcp add --transport http siftdog http://localhost:8000/mcp

Benchmark

Measured against Tavily

45 queries through the same client, 25 September 2026.

25/25factual answers found in advanced mode, same as Tavily

9/10official docs returned (Tavily: 7 basic, 3 advanced)

$0per search, vs $8–16 per 1,000 on Tavily

Full results, method and caveats →

Quick start

Running in a minute

The Compose stack includes SearXNG for search. The image and the Python package work on their own for extract, crawl and map.

git clone https://github.com/khsarvar/siftdog
cd siftdog
docker compose up

API on port 8000 with SearXNG. OpenAPI docs at /docs.

docker run -p 8000:8000 ghcr.io/khsarvar/siftdog

Add -e SEARXNG_URL=… to enable search. Images for amd64 and arm64.

pip install siftdog
siftdog --port 8000

Python 3.11+. Configure with environment variables or a .env file.

Compare

Siftdog vs Tavily, Firecrawl and crw

Pick Tavily if you'd rather not run infrastructure, Firecrawl or crw if you need JavaScript rendering. Siftdog is the Tavily API on your own servers, MIT licensed.

	Siftdog	Tavily	Firecrawl	crw
License	MIT	Proprietary (hosted service)	AGPL-3.0	AGPL-3.0
Self-hosted	Yes	No	Yes	Yes (also a managed API)
Tavily-compatible API	Yes	—	No (own API)	No (own API)
Search source	SearXNG metasearch	Proprietary	—	SearXNG (bundled)
JavaScript rendering	No (static HTML)	—	Yes	Yes (Lightpanda, Chrome fallback)
MCP server	Yes (HTTP and stdio)	Yes	Yes	Yes
Language	Python	—	TypeScript	Rust

Don't want to run servers?

Hosted Siftdog is planned: the same API and MCP endpoint, licensed search sources and no infrastructure. Join the waitlist to hear when it opens.

Join hosted waitlist Self-host free

FAQ