Self-Hosted Search API — A DevOps Guide (2026)

Why self-host search at all

Self-hosting a search API is rarely fun. The reasons people do it anyway:

1. Data residency. User queries are PII in regulated industries. Sending them to a US cloud breaks GDPR, HIPAA, or SOC 2 in ways your security team won't sign off on.
2. Vendor risk. Tavily was acquired by Nebius in February 2026; the deal hadn't closed by May. Healthy companies still get acquired, sunset products, change pricing. Self-hosted means you control the timeline.
3. Cost at scale. The break-even between API credits and a self-hosted server lands somewhere around 5K–10K req/mo. Below that, paid APIs win. Above that, the math compounds in your favor.
4. Regulatory. Some workloads (gov, defense, finance) literally cannot ship outbound queries.
5. Operational simplicity. One fewer API key, one fewer rate-limit page, one fewer external SLA in your dependency graph.

If none of these apply, don't self-host search. Use Tavily, Serper, or SerpAPI and move on. The ops cost is real; this page is for teams where the trade-off is favorable.

OSS self-hosted search APIs — comparison matrix

Project	License	Architecture	Auth/Limits	Content extraction	MCP	Hardening config
fastCRW	AGPL-3.0	Rust + bundled search backend	Bearer token (optional self-host)	Yes (/v1/scrape)	Yes	Read-only rootfs, dropped caps, mem limits, pinned image
OrioSearch	MIT	Python + metasearch aggregator + Redis	Bearer token, timing-safe	Yes (trafilatura/readability)	No	Compose default
agent-search	MIT	FastAPI + metasearch aggregator	Bearer token	Yes (9-strategy fallback chain)	Yes	Compose default + optional Tor
Vane (was Perplexica)	MIT	Chat UI + metasearch aggregator	UI auth	Built-in	No	Chat product, not API
A bare metasearch daemon	AGPL-3.0	Aggregator only	None	No	No	Compose default

The matrix highlights the gap: a bare aggregator gives you search results but ships none of the wrapper concerns. Each of these projects is a different opinion on what wrapper to build.

Hardening — what fastCRW's compose stack actually does

The search sidecar in docker-compose.yml ships with these applied:

• read_only: true for a read-only rootfs.
• cap_drop: [ALL] so every Linux capability is dropped.
• security_opt: [no-new-privileges:true] to block privilege escalation.
• mem_limit, memswap_limit and pids_limit as memory, swap and PID caps.
• tmpfs for /tmp and the runtime cache, so nothing else in the container is writable.
• A pinned image tag, never :latest, and its config mounted read-only.

What this gets you:

• A compromised search sidecar can't write to its own filesystem (read-only rootfs).
• A compromised search sidecar can't gain privileges (no-new-privileges, dropped caps).
• A compromised search sidecar can't fork-bomb the host (pids_limit).
• Memory pressure is bounded (mem_limit).
• Image pinning prevents supply-chain drift — no auto-updating to a :latest tag.

The full file is in the repo, so you can audit every line before deploying.

What it does NOT cover:

• Application-level vulnerabilities in the search backend or fastCRW (those need patching via image bumps).
• Prompt injection in scraped content — that's an application concern, not infrastructure.
• DDoS at the edge — put fastCRW behind a CDN or load balancer with rate limiting.

Threat model

User query → fastCRW HTTP layer → search sidecar → upstream engines
                ↓                         ↓
       (auth, rate limit)         (no internet egress
       (input validation)          beyond search engines)

Threat	Mitigation
Prompt injection in scraped content reaching LLM	Content sanitization on /v1/scrape (agent-search calls this "prompt-injection scrubbing"; fastCRW does its own version)
SSRF — /v1/scrape accepting http://localhost, http://169.254.169.254/, etc.	URL validation, reject loopback/link-local/private, configurable allowlist
Resource exhaustion — 10 GB page download	Response size cap (fastCRW caps at 10MB by default)
Compromised sidecar attempts host escape	cap_drop, no-new-privileges, read-only rootfs
Vulnerable search backend version	Pinned image tag forces deliberate version bumps; subscribe to upstream security feed
API key leakage	Bearer auth optional in self-host (intentional — many self-hosters run inside a private network)

If your environment requires more, layers above this stack (network policies, mTLS, egress firewalls) are where to add them — none are blocked by the compose default.

How to deploy fastCRW (the 2-minute path)

# 1. Clone
git clone https://github.com/us/crw && cd crw

# 2. Configure
cp .env.example .env
# Optional: set CRW_API_TOKEN if you want bearer auth on
vim .env

# 3. Boot
docker compose up --build
# Stack: fastCRW (:8080) + search sidecar + Redis

# 4. Smoke test
curl -X POST http://localhost:8080/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query": "site:nist.gov password rotation guidance", "limit": 5}'

# 5. Verify health
curl http://localhost:8080/health

For production: put it behind your existing reverse proxy / WAF / CDN. The fastCRW binary speaks plain HTTP on :8080 by design — TLS termination is your edge layer's job.

Operational concerns

Upstream rate limits

The search backend queries upstream engines directly from your server's IP. At meaningful QPS, two things happen:

1. Captchas. The big engines fingerprint repeat-querier traffic. Mitigation: enable engine rotation in the backend's settings, configure Brave Search API for a paid lane.
2. Soft bans. Some engines simply rate-limit known IPs. Mitigation: rotate egress IPs (residential proxies, multiple VPS), or accept lower throughput.

This is the operational tax. fastCRW Cloud absorbs it; self-host means you own it.

Observability

The compose stack ships:

• /health endpoint (open, JSON)
• /tool-schema endpoint (open, JSON, lists MCP tool surface)
• structured tracing via OpenTelemetry env vars (set OTEL_EXPORTER_OTLP_ENDPOINT)

If you have a Grafana stack, point it at the OTLP endpoint and you get latency, error rate, and per-engine breakdown out of the box.

Backup and disaster recovery

Stateless. fastCRW, the search backend, and Redis are all stateless within a deployment — no database to back up. Configuration lives in your .env and the search backend's settings file, both of which should live in your config-management repo. To DR: re-deploy the compose stack on a new host.

Where each option fails

• fastCRW: at high QPS, upstream rate limits hit (above). At very low resource budgets, the Rust runtime is overkill — but with a small image and low idle RAM, it's not really a constraint anyone hits.
• A bare metasearch daemon: zero auth, zero rate limit, zero extraction. You build all of it. That's not a flaw — it's the explicit shape — but plan for it.
• OrioSearch: small project (~22 stars). Maintenance risk. If your team has Python expertise and the project's compatibility shape matches your needs, it's a real option; budget for forking later.
• agent-search: ~25 stars, similar maintenance concern. The Tor stack adds operational surface area.
• Vane: not an API. Chat UI. Different shape entirely.

When to pay for hosted instead

Self-hosting is the right call when one of the five reasons at the top of this page applies. If none does, paying for Tavily, Serper, SerpAPI, or fastCRW Cloud is cheaper than your team's time. The ops tax of self-hosting is real:

• ~30 minutes of attention per week (engine rotation, image bumps, captcha investigation),
• on-call rotation if it's user-facing,
• one engineer who actually understands the stack when it breaks.

Most teams underestimate this until they're three months in. Plan for it honestly.

Recommended next reads

• Tavily alternative hub — Tavily-shape API surface, hosted plan option.
• Open-source Tavily alternatives — narrower OSS-focused comparison.
• Tavily vs Serper — when paid APIs make sense.

Three calls to action

• Deploy now — docker compose up. Quickstart →
• Audit the hardening config — docker-compose.yml on GitHub →
• Skip the ops — hosted plan, same API surface →