Self-Host CRW With Docker Compose (2026): Production Setup in 10 Minutes

Why Self-Host

Self-hosting CRW means zero per-request cost, full data locality, and no credit meter. Unlike self-hosting Firecrawl — which needs Redis plus a 5+ service Docker Compose stack and a closed-source anti-bot engine — CRW is a single ~6MB Rust binary. The compose file below is small, but production-shaped: auth, limits, healthchecks, TLS, and log rotation.

Step 1: Minimal Run (Sanity Check)

Before composing anything, confirm the image works:

docker run -p 3000:3000 ghcr.io/us/crw:latest

In another terminal:

curl -s -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}' | head

You should get clean markdown back. Now make it production-grade.

Step 2: Environment File

Keep secrets out of the compose file. Create .env:

# .env
CRW_API_KEY=fc-change-me-to-a-long-random-string
OPENAI_API_KEY=sk-...        # only needed for /v1/extract LLM features
CRW_LOG_LEVEL=info

Generate a strong key:

openssl rand -hex 32

Step 3: docker-compose.yml

services:
  crw:
    image: ghcr.io/us/crw:latest
    restart: unless-stopped
    expose:
      - "3000"
    environment:
      CRW_API_KEY: ${CRW_API_KEY}
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      CRW_LOG_LEVEL: ${CRW_LOG_LEVEL:-info}
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 512M
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      crw:
        condition: service_healthy

volumes:
  caddy_data:
  caddy_config:

Note the 512M memory limit — generous for CRW, which has a low idle footprint. The Rust binary will not approach this; the cap simply protects the host from a pathological page.

Step 4: TLS With Caddy (Automatic HTTPS)

Caddy provisions and renews Let's Encrypt certificates automatically. Create Caddyfile:

scrape.yourdomain.com {
    encode gzip
    reverse_proxy crw:3000

    # Basic abuse protection at the edge
    rate_limit {
        zone api {
            key    {remote_host}
            events 120
            window 1m
        }
    }
}

Point an A record for scrape.yourdomain.com at the server, then:

docker compose up -d
docker compose ps        # both services should be healthy
docker compose logs -f crw

Step 5: Verify End-to-End

curl -s -X POST https://scrape.yourdomain.com/v1/scrape \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

A request without the bearer token should be rejected — confirm auth is enforced before exposing the box.

Step 6: Point Any Firecrawl SDK at It

Your self-hosted CRW is a drop-in Firecrawl endpoint. In Python:

from firecrawl import FirecrawlApp

app = FirecrawlApp(
    api_key="fc-YOUR-KEY",
    api_url="https://scrape.yourdomain.com",
)
doc = app.scrape_url("https://example.com", params={"formats": ["markdown"]})
print(doc["markdown"][:300])

Step 7: Updates and Rollback

# Pull a new image and recreate only the crw service
docker compose pull crw
docker compose up -d crw

# Pin a known-good tag instead of :latest for reproducible rollbacks
# image: ghcr.io/us/crw:v0.7.0

Pin an explicit version tag in production. :latest is fine for trying things; a pinned tag means an update is a deliberate, reversible action.

Step 8: Backups and Observability

• State — CRW is stateless for scrape/search; nothing to back up beyond your .env and compose files. Keep them in a private git repo.
• Logs — json-file driver with rotation is configured; ship to Loki/ELK with a log agent if you need retention.
• Health — the /health endpoint drives the Docker healthcheck and can feed an uptime monitor.

Hardening the Deployment

The compose file above is production-shaped, but a public endpoint deserves a few more guards. None of these are exotic — they are the difference between "works on my machine" and "safe to expose."

Run as a non-root user and read-only filesystem. CRW does not need to write to its own image at runtime, so lock it down. Add to the crw service:

    user: "10001:10001"
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true

Do not publish CRW's port directly. The compose file uses expose (internal only) rather than ports for the crw service, so the engine is reachable only through Caddy. Never bind 3000:3000 to the host on a public box — that bypasses TLS, the rate limit, and the reverse proxy's request logging.

Rotate the API key without downtime. Because the key lives in .env and is read at process start, rotating is: add the new key to your client, update .env, then docker compose up -d crw to recreate just that container. Keep the rotation window short and never commit .env — add it to .gitignore and store the value in your secret manager.

Monitoring and Alerting

The Docker healthcheck restarts a wedged container, but it does not tell you anything. Add an external probe so a sustained outage pages a human. Any uptime monitor that can POST works; here is a minimal self-hosted check you can cron from a separate host:

#!/usr/bin/env bash
# health-probe.sh — run from a DIFFERENT machine than the CRW host
set -euo pipefail
URL="https://scrape.yourdomain.com/health"
if ! curl -fsS --max-time 10 "$URL" >/dev/null; then
  curl -fsS -X POST "$SLACK_WEBHOOK" \
    -d "{\"text\": \"CRW health check FAILED for $URL\"}"
  exit 1
fi

# crontab on the monitoring host — every 2 minutes
*/2 * * * * /opt/probe/health-probe.sh >> /var/log/crw-probe.log 2>&1

Probe from a different machine than the one running CRW — a probe co-located with the service cannot tell you when the whole box is down, which is the failure you most need to hear about.

Capacity Planning

Because CRW is a single binary with a tiny idle footprint, capacity planning is unusually simple: the constraint is concurrent in-flight scrapes, not memory. A 1 vCPU / 1 GB box comfortably serves a steady stream of sequential scrapes plus the Caddy proxy. If your workload is bursty — an agent fleet that fires fifty requests at once — you have two levers: scale vertically (more vCPU lets CRW process more pages in parallel) or run two CRW replicas behind Caddy with a simple round-robin upstream. The point is that you reach for capacity deliberately, with a known cheap baseline, rather than over-provisioning a heavyweight stack "just in case." Start on the smallest box, watch CPU under real traffic, and grow only when the graph tells you to.

Resource Footprint

The whole stack — CRW plus Caddy — comfortably fits a 1 vCPU / 1 GB VPS. CRW's open-core Rust engine is a small single binary, lower-latency, and local-first. Compare that to a Firecrawl self-host that needs Redis and multiple worker containers.

Cost Math vs a Metered API

The reason teams self-host is usually economic, and the math is worth making explicit because it is stark. A metered scraping API charges per page, and the per-page number quietly multiplies: a "JSON" or "stealth" mode can stack to several credits per page, and a separate extraction product is often billed on top. At a few hundred thousand pages a month — modest for an AI ingestion pipeline — that becomes a four-figure monthly bill that grows with usage and gives you nothing durable. The self-hosted alternative is a fixed server cost. The single-binary stack in this guide runs comfortably on a 1 vCPU / 1 GB instance that costs a few dollars a month, and that number does not move whether you scrape ten thousand pages or ten million within the box's throughput. The crossover point where self-hosting wins is low, and past it the savings compound every month.

There is operational cost too, and it should be counted honestly — you own uptime, updates, and the box. But because CRW is one stateless binary with no Redis, no worker fleet, and no browser to babysit, that operational surface is small: the entire system is the compose file, the env file, and a reverse proxy. The realistic comparison is not "free vs paid" but "a few dollars of server plus an hour of setup, once" against "a usage-metered bill that scales with success forever." For any workload past prototype scale, the open-core path is the rational default, and the managed cloud remains there as a one-URL escape hatch if your priorities change.

Why CRW Self-Host

• One binary — no Redis, no worker fleet, no Chromium per request.
• Zero per-request cost — AGPL-3.0, unlimited self-host. The optional fastCRW cloud free tier is a one-time lifetime 500 credits, never a recurring meter.
• No lock-in — same Firecrawl-compatible API as the managed cloud; move either direction with one URL change.

Two-Replica Variant for Bursty Load

If a single CRW process becomes the bottleneck under bursty agent traffic, scale out before scaling up — Caddy load-balances across replicas with a one-line change. Run two CRW containers and let the proxy distribute:

services:
  crw1:
    image: ghcr.io/us/crw:latest
    restart: unless-stopped
    expose: ["3000"]
    env_file: .env
  crw2:
    image: ghcr.io/us/crw:latest
    restart: unless-stopped
    expose: ["3000"]
    env_file: .env
  caddy:
    image: caddy:2-alpine
    ports: ["80:80", "443:443"]
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
volumes:
  caddy_data:

# Caddyfile — round-robin across both replicas
scrape.yourdomain.com {
    encode gzip
    reverse_proxy crw1:3000 crw2:3000 {
        lb_policy round_robin
        health_uri /health
    }
}

Because CRW is stateless, replicas need no coordination — there is no shared cache or session to synchronize. This is the entire horizontal-scaling story: add containers, list them in the proxy, done. Start with one replica, watch CPU under real traffic, and add the second only when the graph justifies it.

Next Steps

• Read Run CRW on a $5 VPS
• See Self-Host a Firecrawl-Compatible API

Self-host CRW from GitHub for free, or use fastCRW for managed cloud scraping.