Crawl an Entire Website to Markdown With fastCRW (2026): Map, Crawl, Export

What We're Building

A complete site crawler that turns an entire website into a local markdown corpus — one clean file per page, plus an index. We'll map the site first to see scope, run a bounded crawl, write files safely, and add an incremental mode so daily re-crawls only touch changed pages. This is the canonical input step for RAG, docs mirroring, archival, and migration.

Map First, Then Crawl

Always map before you crawl. Map is cheap and tells you how big the site is, so you set sane limits before spending time fetching content.

Prerequisites

• CRW running: docker run -p 3000:3000 ghcr.io/us/crw:latest
• Python 3.10+

pip install firecrawl-py

Step 1: Connect

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR-KEY", api_url="http://localhost:3000")
# fastCRW cloud: api_url="https://api.fastcrw.com"

Step 2: Map the Site to Estimate Scope

def map_site(base_url: str) -> list[str]:
    res = app.map_url(base_url)
    # SDK returns either a list or {"links": [...]}
    urls = res if isinstance(res, list) else res.get("links", [])
    print(f"{base_url}: {len(urls)} URLs discovered")
    return urls


urls = map_site("https://docs.example.com")

If map returns 12 URLs, a full crawl is trivial. If it returns 8,000, you cap the crawl and likely process in batches — knowing this up front saves a runaway job.

Step 3: Crawl With Bounded Limits

def crawl_site(base_url: str, limit: int, max_depth: int = 4) -> list[dict]:
    job = app.crawl_url(base_url, params={
        "limit": limit,
        "maxDepth": max_depth,
        "scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
    })
    pages = job.get("data", [])
    print(f"crawled {len(pages)} pages")
    return pages

onlyMainContent is essential here: it removes nav, sidebars, and footers so each file is the actual article body, not boilerplate repeated across every page.

Step 4: Write One Safe File Per Page

Mirror the URL path into a directory tree, sanitizing names so the filesystem stays happy:

import re, pathlib
from urllib.parse import urlparse


def safe_path(out_root: pathlib.Path, url: str) -> pathlib.Path:
    p = urlparse(url)
    parts = [seg for seg in p.path.split("/") if seg]
    if not parts:
        parts = ["index"]
    # last segment becomes the filename
    *dirs, name = parts
    name = re.sub(r"[^a-zA-Z0-9._-]", "-", name) or "index"
    if not name.endswith(".md"):
        name += ".md"
    target = out_root.joinpath(p.netloc, *[
        re.sub(r"[^a-zA-Z0-9._-]", "-", d) for d in dirs], name)
    target.parent.mkdir(parents=True, exist_ok=True)
    return target


def write_pages(pages: list[dict], out_dir: str) -> int:
    root = pathlib.Path(out_dir)
    written = 0
    for page in pages:
        url = page.get("metadata", {}).get("sourceURL", "")
        md = page.get("markdown", "")
        if not url or not md:
            continue
        path = safe_path(root, url)
        title = page.get("metadata", {}).get("title", "")
        path.write_text(f"<!-- source: {url} -->\n# {title}\n\n{md}")
        written += 1
    print(f"wrote {written} files under {out_dir}")
    return written

Step 5: Build an Index

import json


def build_index(pages: list[dict], out_dir: str):
    entries = []
    for page in pages:
        meta = page.get("metadata", {})
        url = meta.get("sourceURL", "")
        if not url:
            continue
        entries.append({
            "url": url,
            "title": meta.get("title", ""),
            "words": len(page.get("markdown", "").split()),
        })
    entries.sort(key=lambda e: e["url"])
    idx = pathlib.Path(out_dir) / "INDEX.json"
    idx.write_text(json.dumps(entries, indent=2))
    print(f"index: {len(entries)} entries -> {idx}")

Step 6: Put It Together

def export_site(base_url: str, out_dir: str = "site-export",
                limit: int = 500):
    map_site(base_url)                 # scope check
    pages = crawl_site(base_url, limit)
    write_pages(pages, out_dir)
    build_index(pages, out_dir)


if __name__ == "__main__":
    export_site("https://docs.example.com", out_dir="docs-mirror",
                limit=300)

Step 7: Incremental Re-Crawl

For a living mirror, only rewrite pages whose content changed — cheap to run daily:

import hashlib


def incremental(base_url: str, out_dir: str, limit: int = 500):
    state_file = pathlib.Path(out_dir) / ".hashes.json"
    state = json.loads(state_file.read_text()) if state_file.exists() else {}

    pages = crawl_site(base_url, limit)
    changed = 0
    for page in pages:
        url = page.get("metadata", {}).get("sourceURL", "")
        md = page.get("markdown", "")
        if not url or not md:
            continue
        h = hashlib.sha256(md.encode()).hexdigest()
        if state.get(url) == h:
            continue
        path = safe_path(pathlib.Path(out_dir), url)
        path.write_text(md)
        state[url] = h
        changed += 1

    state_file.write_text(json.dumps(state, indent=2))
    print(f"incremental: {changed} pages updated")

Crawl Scope: The Setting That Decides Everything

Most bad crawls are scope failures, not code failures. Three parameters control scope and you should set all of them deliberately. limit is a hard ceiling on pages — it is your circuit breaker against a site with infinite calendar or pagination URLs, and it should always be set to a number you are willing to fetch, never left unbounded. maxDepth controls how far link-following goes from the seed; depth 2-3 captures a docs site, while depth 5+ on a large site explodes combinatorially. The seed URL itself is the third lever and the most underused: crawling from https://site.com/docs/ rather than https://site.com/ naturally scopes the crawl to the section you actually want, because CRW follows links outward from where you start. Map first, look at the URL shapes it returns, and choose a seed and limits that match the subset you need — this single habit prevents the runaway crawl that is the most common first-attempt mistake.

Why Filesystem Mirroring Needs Care

Writing one file per page sounds trivial until a URL contains a query string, a Unicode path segment, a trailing slash, or collides with another page after sanitization. The safe_path function handles the common cases, but the principle is what matters: the URL is untrusted input and the filesystem has rules (length limits, reserved characters, case-insensitivity on some platforms). Never write a path derived from a URL without sanitizing every segment, and decide explicitly what happens on collision — the implementation here would overwrite, which is correct for a mirror (the latest crawl wins) but wrong for an archive (where you would append a hash suffix to keep both). Preserving the source URL in an HTML comment at the top of each file is not decoration; it is the provenance record that lets a downstream RAG system cite the original, and it survives even if the filename had to be mangled to be filesystem-safe.

Incremental Crawls and the Cost Curve

The difference between a one-shot export and a maintainable mirror is the cost curve over time. A full re-crawl every night is O(site size) forever, regardless of how little changed — wasteful and unfriendly to the target. The incremental design flips this to O(changed pages): you still fetch each page (CRW has to see the current content to know if it changed), but you only do the expensive downstream work — rewriting files, re-embedding for RAG, re-indexing — for pages whose content hash moved. On a docs site where a handful of pages change per week, this turns a nightly job that reprocesses thousands of files into one that touches a dozen. The state file is tiny (a URL-to-hash map) and is the entire mechanism. The discipline to internalize: fetching is cheap and bounded by the site; reprocessing is what you must keep proportional to actual change, and a content hash is the cheapest way to do it.

Practical Tips

• Map → decide → crawl — never start a crawl blind on an unknown site.
• Cap limit and maxDepth — fail safe; raise them deliberately once you know scope.
• Keep the source URL in the file — the HTML comment header preserves provenance for downstream RAG citations.
• Run CRW locally — with a low idle footprint it sits comfortably next to the export job.

Why CRW

• Server-side crawl — link discovery, dedup, and depth handled for you in one call.
• Clean markdown — onlyMainContent yields a corpus that is actually usable, not boilerplate-laden HTML.
• No lock-in — open-core Rust, small single binary, lower-latency, local-first, AGPL-3.0 + Managed Cloud.

Filtering What You Crawl

A full-site crawl usually pulls pages you do not want — tag archives, paginated listings, login pages, legal boilerplate. Rather than crawl everything and clean up after, filter at the boundary so you never pay to fetch or store noise. The cleanest place is right after crawl, using URL patterns and a content gate:

import re

EXCLUDE_PATTERNS = [
    r"/tag/", r"/category/", r"/page/\d+", r"/login",
    r"/cart", r"\?", r"/feed",
]


def keep_page(page: dict) -> bool:
    url = page.get("metadata", {}).get("sourceURL", "")
    if any(re.search(p, url) for p in EXCLUDE_PATTERNS):
        return False
    md = page.get("markdown", "")
    if len(md.split()) < 50:        # near-empty / stub pages
        return False
    return True


def export_filtered(base_url: str, out_dir: str, limit: int = 500):
    map_site(base_url)
    pages = [p for p in crawl_site(base_url, limit) if keep_page(p)]
    print(f"kept {len(pages)} pages after filtering")
    write_pages(pages, out_dir)
    build_index(pages, out_dir)

Keep the exclude list short and URL-shaped — it generalizes across sites far better than per-site rules, because patterns like /tag/ and /page/2 are conventions, not site-specific markup. The word-count gate then catches the stubs that slip through. This pairs naturally with the map-first habit: scanning the URL inventory tells you which patterns to exclude before you commit to the crawl, so the filtered export is right on the first run instead of after a cleanup pass.

Next Steps

• Feed the corpus into Scrape-to-RAG With LlamaIndex
• Automate it with Scheduled Crawls With Cron and CRW

Self-host CRW from GitHub for free, or use fastCRW for managed cloud scraping.