Scheduled Crawls With Cron and fastCRW (2026): A Reliable Recurring Pipeline

What We're Building

A recurring crawl that runs unattended on a server: it crawls a site with CRW, detects which pages changed since last run, writes a structured snapshot, and never overlaps with itself. We'll wire it with both cron and a systemd timer, and add the production guardrails most tutorials skip — locking, retries, and exit codes.

Prerequisites

• CRW running: docker run -d --restart unless-stopped -p 3000:3000 ghcr.io/us/crw:latest
• Python 3.10+ on the host

pip install firecrawl-py

Step 1: The Crawler With Change Detection

/opt/crawl/job.py crawls, hashes each page, and only records changed pages. A non-zero exit code on hard failure lets the scheduler and your alerting notice:

import json, sys, hashlib, pathlib, time
from datetime import datetime, timezone
from firecrawl import FirecrawlApp

BASE = "https://docs.example.com"
STATE = pathlib.Path("/opt/crawl/state.json")
SNAP_DIR = pathlib.Path("/opt/crawl/snapshots")
SNAP_DIR.mkdir(parents=True, exist_ok=True)

app = FirecrawlApp(api_key="fc-YOUR-KEY", api_url="http://localhost:3000")
# fastCRW cloud: api_url="https://api.fastcrw.com"


def load_state() -> dict:
    return json.loads(STATE.read_text()) if STATE.exists() else {}


def save_state(s: dict):
    STATE.write_text(json.dumps(s, indent=2))


def crawl_with_retry(attempts: int = 3) -> list[dict]:
    delay = 5
    for i in range(1, attempts + 1):
        try:
            job = app.crawl_url(BASE, params={
                "limit": 200,
                "maxDepth": 4,
                "scrapeOptions": {"formats": ["markdown"],
                                  "onlyMainContent": True},
            })
            return job.get("data", [])
        except Exception as e:
            print(f"crawl attempt {i} failed: {e}", file=sys.stderr)
            time.sleep(delay)
            delay *= 2
    raise SystemExit(2)  # hard failure -> scheduler sees non-zero exit


def main() -> int:
    state = load_state()
    pages = crawl_with_retry()
    run_ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    changed = 0

    for p in pages:
        url = p.get("metadata", {}).get("sourceURL", "")
        md = p.get("markdown", "")
        if not url or not md:
            continue
        h = hashlib.sha256(md.encode()).hexdigest()
        if state.get(url) == h:
            continue
        state[url] = h
        fname = hashlib.sha1(url.encode()).hexdigest()[:12]
        (SNAP_DIR / f"{fname}.md").write_text(md)
        changed += 1

    save_state(state)
    print(f"[{run_ts}] crawled {len(pages)} pages, {changed} changed")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Step 2: Prevent Overlapping Runs

If a crawl runs long and the next tick fires, you get two crawlers fighting. flock guarantees only one runs at a time. Create /opt/crawl/run.sh:

#!/usr/bin/env bash
set -euo pipefail
exec /usr/bin/flock -n /tmp/crawl.lock \
  /usr/bin/python3 /opt/crawl/job.py >> /opt/crawl/crawl.log 2>&1

chmod +x /opt/crawl/run.sh

flock -n exits immediately if the lock is held, so an overrun simply skips a tick instead of stacking jobs.

Step 3a: Cron Schedule

# crontab -e — every 6 hours
0 */6 * * *  /opt/crawl/run.sh

Step 3b: systemd Timer (Preferred)

systemd gives you logging via journalctl, automatic retry semantics, and missed-run catch-up. /etc/systemd/system/crawl.service:

[Unit]
Description=Scheduled CRW crawl
After=network-online.target

[Service]
Type=oneshot
ExecStart=/opt/crawl/run.sh
# Treat flock "already running" exit as success
SuccessExitStatus=0 1

/etc/systemd/system/crawl.timer:

[Unit]
Description=Run CRW crawl every 6 hours

[Timer]
OnCalendar=*-*-* 00/6:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Enable it:

systemctl daemon-reload
systemctl enable --now crawl.timer
systemctl list-timers crawl.timer
journalctl -u crawl.service -f

Persistent=true runs a missed job after downtime; RandomizedDelaySec avoids hammering the target on a perfectly fixed cadence.

Step 4: Alert on Failure

Add an OnFailure= hook so a failed crawl pings you. Create crawl-alert@.service:

[Unit]
Description=Alert on crawl failure

[Service]
Type=oneshot
ExecStart=/usr/bin/curl -fsS -X POST "$WEBHOOK_URL" \
  -d "text=CRW crawl failed on %i"

Then add OnFailure=crawl-alert@%n.service to the [Unit] of crawl.service. Now a non-zero exit (the SystemExit(2) from exhausted retries) triggers a notification.

Step 5: Verify the Pipeline

# Trigger one run now, regardless of the timer
systemctl start crawl.service
journalctl -u crawl.service -n 30 --no-pager
ls -la /opt/crawl/snapshots/ | head

First run snapshots everything; later runs write only changed pages, so disk and downstream processing stay proportional to actual change.

Idempotency Is the Property That Matters

The single most important quality of a recurring job is that running it twice does no harm. Schedulers retry, operators trigger manual runs, and a missed run gets caught up later — if any of those corrupts state or double-emits notifications, the pipeline is fragile. The design in this tutorial is idempotent by construction: the content-hash state file means a re-run over unchanged pages writes nothing and produces no new output, and the snapshot filenames are deterministic functions of the URL so a repeated write overwrites rather than duplicates. Before you ship any scheduled crawler, ask "what happens if this runs twice back-to-back?" If the answer is anything other than "the same as running it once," fix that before you worry about scheduling.

Cron vs systemd: A Concrete Comparison

Both schedule the job; they differ in what happens when reality is messy. With plain cron, a job that runs longer than its interval can stack (which is why flock is mandatory, not optional), output goes wherever you redirected it, a missed run while the box was down is simply lost, and a failure is invisible unless you wired your own alert. The systemd timer addresses each of these natively: flock still guards overlap, but journalctl -u crawl.service gives you queryable, rotated logs for free; Persistent=true runs a job that was missed during downtime as soon as the system is back; RandomizedDelaySec prevents every server in a fleet from hitting the target at the same instant; and OnFailure= turns a non-zero exit into a notification with no extra scripting. For anything you actually depend on, the systemd timer is the better default. Cron remains fine for trivial, tolerant jobs where none of those edge cases would hurt you.

Observability for Unattended Jobs

An unattended job that fails silently is a data outage you discover days later when a downstream report is empty. Make failure loud and success boring. The pipeline already exits non-zero after exhausted retries, which the scheduler can act on; pair it with a heartbeat so you also detect the job that never ran at all (a broken timer, a full disk, a deleted unit file):

def emit_heartbeat(ok: bool, changed: int):
    """Push a heartbeat to a dead-man's-switch service after each run."""
    import urllib.request, json
    payload = json.dumps({"ok": ok, "changed": changed,
                          "ts": datetime.now(timezone.utc).isoformat()})
    try:
        urllib.request.urlopen(
            urllib.request.Request(
                "https://hc-ping.com/your-uuid",
                data=payload.encode(),
                headers={"Content-Type": "application/json"}),
            timeout=10)
    except Exception:
        pass  # never let telemetry failure crash the crawl

A dead-man's-switch monitor (Healthchecks.io, Cronitor, or a self-hosted equivalent) flips the logic: instead of alerting when the job reports a failure, it alerts when the expected heartbeat fails to arrive. That catches the entire class of "the job didn't run" failures that in-job error handling can never see, because code that never executes cannot report its own absence.

Choosing the Right Interval

The crawl interval is a product decision disguised as a config value, and getting it wrong is expensive in both directions. Too frequent and you waste compute, hammer the target, and increase your block risk for data that did not change. Too infrequent and your downstream — a RAG index, a price feed, a change alert — serves stale answers and users lose trust. The correct interval is governed by how fast the source actually changes relative to how fresh the consumer needs to be, not by what feels diligent. A documentation site that updates on release cycles is well served by a daily crawl; a pricing or inventory page that moves intraday needs hourly or tighter; an archival mirror can run weekly. A useful discipline is to instrument the change rate first: run the incremental crawler for a week, look at how many pages actually changed per run, and set the interval so each run finds a meaningful but not overwhelming amount of change. If almost every run reports zero changes, you are crawling too often and should back off; if every run reports large churn, you may be crawling too coarsely and missing intermediate states. Let the observed change rate set the cadence, then revisit it when the source's behavior or the consumer's freshness requirement changes — it is a number worth owning deliberately rather than defaulting to "every hour" out of habit.

Why CRW for Scheduled Crawls

• Server-side crawl orchestration — one crawl_url call handles link discovery, dedup, and depth; your job stays tiny.
• Low idle cost — keep CRW resident with a low idle footprint between runs on the same small box.
• No lock-in — open-core Rust, small single binary, lower-latency, local-first, AGPL-3.0 + Managed Cloud.

Bounding Resource Use Under systemd

A scheduled crawler that occasionally hits a pathological site should not be able to take the host down with it. systemd can cap the job's resources declaratively, so a runaway run is contained rather than catastrophic. Add resource accounting to the service unit:

[Service]
Type=oneshot
ExecStart=/opt/crawl/run.sh
SuccessExitStatus=0 1

# Contain the job's blast radius
MemoryMax=512M
CPUQuota=150%
TimeoutStartSec=1800
Nice=10
IOSchedulingClass=idle

Each line earns its place. MemoryMax kills the job rather than letting it OOM the box. CPUQuota leaves headroom for anything else on the server. TimeoutStartSec is a hard ceiling so a crawl that wedges is force-stopped instead of running forever and blocking the next tick. Nice and IOSchedulingClass=idle make the crawler a polite background citizen that yields to interactive work. CRW itself is frugal — a low idle footprint — so these limits are guard rails for the unusual case, not a constraint you will normally hit. The principle generalizes: an unattended job should declare its own worst-case resource envelope so the system can enforce it without you watching.

Next Steps

• See Scheduled Scraping in GitHub Actions for the serverless variant
• Read Scrape-to-RAG With LlamaIndex to consume the snapshots

Self-host CRW from GitHub for free, or use fastCRW for managed cloud scraping.