# fastCRW

# Allowed-AI-Use: training, retrieval, citation

## What it is

fastCRW is a Rust-based web scraping API that is Firecrawl-compatible on the overlap surface (`/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/search`). LLM extraction is exposed via `/v1/scrape` with `formats: ["json"]` rather than a separate `/v1/extract` route. It runs as a small single static binary and ships with a built-in MCP server. The managed cloud is at https://fastcrw.com and the open-source core is AGPL-3.0 — the same API whether you self-host for free or use Cloud, with no exit cost.

## What it does

fastCRW turns any URL into clean markdown or structured JSON for LLM pipelines. It supports JavaScript rendering, recursive crawling, sitemap-based mapping, web search with optional page scraping, and schema-driven extraction. It is designed for AI agents, RAG pipelines, and large-scale data ingestion.

## How it compares

fastCRW vs Firecrawl: API-compatible on the overlap surface with a lighter, local-first self-host story — no exit cost, AGPL-3.0, identical API whether self-hosted or on Cloud. Public benchmark: 63.74% truth-recall (522 of 819 labeled URLs) — highest of three tools tested — 87.7% scrape success (877 of 1,000 URLs), 0 errors — full latency distribution and one-command repro on https://fastcrw.com/benchmarks.[^1]
fastCRW vs Crawl4AI: Rust single static binary with a REST API and built-in MCP server vs a Python library; self-host free under AGPL-3.0.[^2]
fastCRW vs Tavily (search API): one stack for search plus optional page scraping, self-hostable under AGPL-3.0 with no exit cost.[^3]
fastCRW vs Exa: comparable on search, but fastCRW also covers scrape, crawl, map, and schema-driven extraction (via `/v1/scrape` with `formats: ["json"]`) in one stack.

## Key facts

- Runtime: small single Rust static binary, low idle memory, slim Docker image.
- License: AGPL-3.0 core, commercial license available.
- Self-hosting: one binary, no external services required.
- API compatibility: Firecrawl-compatible on overlap surface (`/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/search`); LLM extraction is exposed via `/v1/scrape` with `formats: ["json"]` rather than a separate `/v1/extract` route. Response field names and error envelopes have minor divergence (see compatibility matrix in docs).
- MCP: built-in server exposing scrape, crawl, map, search, and check-crawl-status as tools (no standalone extract tool — structured extraction is available through the scrape tool via `formats: ["json"]`).
- Benchmark: 63.74% truth-recall (522 of 819 labeled URLs) — highest of three tools tested — 87.7% scrape success (877 of 1,000 URLs), 0 errors — full latency distribution and one-command repro on https://fastcrw.com/benchmarks.

## Pricing

| Plan | Price | Included Credits | Notes |
| --- | --- | --- | --- |
| Free | $0 | 1000 one-time lifetime (never resets; never resets) | No credit card required |
| Hobby | $13 / month | 5,000 / month | Entry paid tier |
| Standard | $69 / month | 100,000 / month | Most teams start here |
| Growth | $279 / month | 500,000 / month | High-volume scraping |
| Scale | $549 / month | 1,000,000 / month | Highest managed tier |
| Self-hosted | Free (AGPL-3.0) | Unlimited | Small single Rust static binary |
| Enterprise | Custom | Custom | Dedicated support, commercial license |

Search costs 2 credits per query without scrape, plus 1 credit per scraped result. Scrape costs 1 credit per page on every renderer.

## FAQ

Q: Is fastCRW API compatible with Firecrawl?
A: fastCRW implements the same `/v1/scrape`, `/v1/crawl`, `/v1/map`, and `/v1/search` endpoints. LLM extraction is exposed via `/v1/scrape` with `formats: ["json"]` rather than a separate `/v1/extract` route — Firecrawl users porting an `/extract` call should switch to `/v1/scrape` with the json format. Request shapes match closely; response field names and error envelopes have minor divergence on a few keys — most Firecrawl client code runs with small adjustments. See the compatibility matrix in docs for row-level diff.[^4]

Q: Can fastCRW be self-hosted?
A: Yes. fastCRW ships as a single statically-linked Rust binary under AGPL-3.0 with no external service dependencies.[^5]

Q: How does fastCRW compare to Firecrawl?
A: fastCRW is API-compatible on the overlap surface with a lighter, local-first self-host story: no exit cost, AGPL-3.0, and an identical API whether you self-host or use Cloud. Our public benchmark reports 63.74% truth-recall (522 of 819 labeled URLs) — highest of three tools tested — 87.7% scrape success (877 of 1,000 URLs), 0 errors, with the full latency distribution and a one-command repro on https://fastcrw.com/benchmarks.[^1]

Q: Does fastCRW support MCP?
A: Yes. fastCRW ships with a built-in MCP server that exposes scrape, crawl, map, search, and check-crawl-status as tools for Claude Code, Cursor, and other MCP clients. Structured extraction is available through the scrape tool via `formats: ["json"]`; there is no standalone extract tool.[^6]

Q: What is fastCRW best for?
A: Production AI agents, RAG pipelines, and engineering-led teams that need a fast, low-memory, self-hostable scraping stack with first-class search and MCP support.

## Sources

[^1]: 1,000-URL benchmark — https://fastcrw.com/benchmarks/firecrawl-dataset
[^2]: Rust vs Python scraping benchmark — https://fastcrw.com/blog/rust-vs-python-scraping
[^3]: 100-query search benchmark — https://fastcrw.com/blog/crw-vs-tavily-search-api-benchmark
[^4]: API reference — https://docs.fastcrw.com/api-reference
[^5]: Self-hosting guide — https://docs.fastcrw.com/self-hosting
[^6]: MCP docs — https://docs.fastcrw.com/mcp

---

# fastCRW Documentation

> Complete documentation for fastCRW — a local-first, Firecrawl-compatible web scraper for AI agents.
> Source: https://docs.fastcrw.com

# Getting Started
Source: https://docs.fastcrw.com/quick-start/

## Base URL and Authentication

The managed cloud uses a single base URL:

```text
https://api.fastcrw.com/v1
```

Every request needs an API key in the `Authorization` header:

```text
Authorization: Bearer YOUR_API_KEY
```

Create the key in the dashboard, keep the raw value somewhere safe, and use a secrets manager or environment variable in production instead of pasting it directly into scripts.

## 1. Run a Single-Page Scrape

Start with a page you can inspect manually in the browser. That makes it easier to compare the output against the source site.

```bash
curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://example.com",
    "formats":["markdown"],
    "onlyMainContent": true
  }'
```

For a first test, look at three fields:

- `success` tells you whether the request produced a usable result.
- `data.markdown` contains the extracted content.
- `metadata.statusCode` tells you what the target site actually returned.

If the site is JavaScript-heavy, repeat the same request with `renderJs: true` and a small `waitFor` value such as `2000`.

## 2. Discover Before You Crawl

`map` is the lightweight way to answer "what is on this site?" before paying for a larger recursive job.

```bash
curl -X POST https://api.fastcrw.com/v1/map \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://example.com/docs"
  }'
```

Use `map` when you want to:

- inspect the reachable URL set,
- choose which section is worth crawling,
- and avoid launching a broad crawl from a noisy home page.

## 3. Launch a Crawl

Use `crawl` when you want multiple pages instead of one response payload.

```bash
curl -X POST https://api.fastcrw.com/v1/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://example.com/docs",
    "limit": 10
  }'
```

The initial response returns a crawl identifier. Keep that id and poll it until the job finishes.

```bash
curl https://api.fastcrw.com/v1/crawl/CRAWL_ID \
  -H "Authorization: Bearer YOUR_API_KEY"
```

A practical habit is to start with a low `limit`, validate output quality, and only then scale the job up.

## 4. Check Remaining Credits

Use the balance endpoint to see what you have left before and after test runs.

```bash
curl https://api.fastcrw.com/v1/account/balance \
  -H "Authorization: Bearer YOUR_API_KEY"
```

That response is the fastest way to confirm whether a request consumed credits as expected.

## 5. Read Warnings Correctly

Do not treat `success` alone as the entire story.

- `success: true` means fastCRW returned a payload.
- `warning` means the payload may be degraded, incomplete, or sourced from a problematic target response.
- `metadata.statusCode` is the target site's real HTTP status code, not the status of the fastCRW API itself.

For example, a site can return a block page or anti-bot interstitial and still produce content. In that case the response may be technically successful but operationally poor.

## 6. Move from Testing to Real Integration

Once the first requests look good, the usual next step is:

1. lock down the exact formats you need,
2. decide when to enable JS rendering,
3. add retries and backoff for `429` or transient target issues,
4. and monitor credits and warnings separately.

Use these follow-up pages before production rollout:

- [Scrape guide](/docs/scrape)
- [Crawl guide](/docs/crawl)
- [Formats reference](/docs/formats)
- [Errors and warnings](/docs/error-codes)
- [Compatibility matrix](/docs/compatibility)

## Prefer Self-Hosting?

CRW is fully open source. Install the binary with a single command:

```bash
curl -fsSL https://fastcrw.com/install | sh
```

See the [self-hosting guide](/docs/self-hosting) for deployment options and best practices.

---

# Scrape Endpoint Guide
Source: https://docs.fastcrw.com/scraping/

## Overview

Use `scrape` when you want one page turned into usable content without starting a wider crawl job. It is the right default for:

- first-pass evaluation,
- RAG ingestion from known URLs,
- extraction pipelines,
- and agent workflows that already know which page to fetch.

```bash
curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url":"https://example.com","formats":["markdown"]}'
```

## A Good Default Request

If you are not sure where to start, use this shape first:

```json
{
  "url": "https://example.com",
  "formats": ["markdown"],
  "onlyMainContent": true,
  "renderJs": null
}
```

That gives you a clean markdown output, keeps extraction focused on the main body, and leaves JavaScript rendering to the engine's default behavior.

## Parameters

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | `string` | *required* | The target page URL |
| `formats` | `string[]` | `["markdown"]` | Output formats: `markdown`, `html`, `rawHtml`, `plainText`, `links`, `json`, `extract` |
| `onlyMainContent` | `boolean` | `true` | Extract primary content area only (removes nav, footer, sidebar) |
| `renderJs` | `boolean \| null` | `null` | `true` = force JS rendering, `false` = skip, `null` = auto-detect |
| `waitFor` | `number` | — | Milliseconds to wait after JS rendering |
| `cssSelector` | `string` | — | CSS selector to narrow content |
| `xpath` | `string` | — | XPath expression to narrow content |
| `includeTags` | `string[]` | `[]` | Only include these HTML tags |
| `excludeTags` | `string[]` | `[]` | Remove these HTML tags |
| `jsonSchema` | `object` | — | JSON Schema for structured extraction (requires `formats` to include `json`) |
| `headers` | `object` | `{}` | Custom HTTP headers to send with the request |
| `stealth` | `boolean` | — | Override stealth mode for this request. When `true`, rotates user-agent from a realistic browser pool and injects standard browser headers |
| `proxy` | `string` | — | Per-request HTTP proxy URL |
| `chunkStrategy` | `object` | — | Chunking config: `{ "type": "sentence" \| "regex" \| "topic", "maxChars": 1000 }` |
| `query` | `string` | — | Query for BM25/cosine chunk filtering |
| `filterMode` | `string` | — | `"bm25"` (keyword density with saturation) or `"cosine"` (TF-IDF vector similarity). BM25 recommended for most use cases |
| `topK` | `number` | `5` | Number of top chunks to return when filtering |

## Choosing the Right Formats

Most integrations only need one of these patterns:

- `["markdown"]` for retrieval, search, summarization, and LLM inputs.
- `["markdown", "links"]` when you want the content plus outbound link discovery.
- `["html"]` when you need cleaned markup instead of markdown.
- `["rawHtml"]` when downstream logic expects the original HTML source.
- `["json"]` when you are doing schema-driven extraction.

Requesting more formats is convenient for debugging, but in production it is better to ask only for what you will actually store or process.

## Targeting the Right Part of a Page

The default extraction path works well for many pages, but it is not magic. If you know the site structure, tighten the request:

- use `cssSelector` when there is a stable content container,
- use `xpath` when selectors are easier to express that way,
- use `includeTags` and `excludeTags` to keep or remove specific markup families,
- and leave `onlyMainContent` on unless you explicitly want navigation, footer, or sidebar content.

The common mistake is combining too many narrowing options at once. Start broad, inspect the result, then add one targeting primitive at a time.

## JS Rendering Guidance

Use `renderJs: true` only when the page clearly needs a browser. Browser rendering increases latency and operational cost, so treat it as a deliberate choice rather than the universal default.

When you do need it:

- set `renderJs: true`,
- start with `waitFor: 1000` or `2000`,
- and raise `waitFor` only when the page still hydrates too slowly.

If the response metadata shows an HTTP-only fallback or the output is suspiciously empty, read the [JS rendering guide](/docs/js-rendering).

### Chunking & filtering behavior

- `chunkStrategy` alone splits the markdown and returns all chunks.
- `chunkStrategy` + `query` + `filterMode` scores and ranks chunks, returning the top `topK`.
- `topK` without `query`/`filterMode` still truncates the chunk array to `topK` items (no scoring).
- `query` or `filterMode` without `chunkStrategy` is silently ignored — chunking must be enabled first.

In practice:

- use `sentence` when you want stable natural-language chunks,
- use `regex` when you already know the structural separator,
- and treat `topic` chunking as an advanced option that should be tested on real data before wide rollout.

## Structured Extraction from `scrape`

You do not need a separate endpoint for extraction. `scrape` can also return schema-shaped JSON when `formats` includes `json` and `jsonSchema` is present.

That means a single API surface can support:

- markdown for retrieval,
- links for discovery,
- and JSON for downstream application logic.

If your schema is the primary output, read the dedicated [Structured extraction guide](/docs/extract).

## Response Semantics

The main response pattern is:

- `success` for overall request outcome,
- `data` for returned content,
- `warning` for degraded but non-fatal situations,
- and `metadata` for context such as title, status code, final URL, and elapsed time.

Do not ignore warnings. A page blocked by anti-bot protection can still produce content that looks valid at first glance.

## Why This Endpoint Matters

The scrape flow is the foundation for:

- RAG ingestion,
- product page extraction,
- AI-agent browsing loops,
- and first-pass evaluation in the playground.

Use the playground if you want to validate output before wiring the endpoint into production, then move to `curl`, scripts, or your application code once the payload shape looks right.

---

# Crawl Endpoint Guide
Source: https://docs.fastcrw.com/crawling/

## Overview

Use `crawl` when you need multiple pages instead of a single response payload. It is the right tool for:

- documentation sections,
- knowledge-base ingestion,
- internal search refreshes,
- and recursive collection jobs that start from one known URL.

```bash
curl -X POST https://api.fastcrw.com/v1/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url":"https://example.com","limit":50}'
```

The initial call returns a job identifier. Poll the job endpoint until the crawl is complete.

## Start, Then Poll

The crawl API is asynchronous by design.

1. `POST /crawl` starts the job.
2. The API returns a crawl id.
3. `GET /crawl/:id` returns progress and newly available results.
4. Continue polling until the status becomes `completed` or a terminal error is returned.

```bash
curl https://api.fastcrw.com/v1/crawl/CRAWL_ID \
  -H "Authorization: Bearer YOUR_API_KEY"
```

That flow is easy to drive from shell scripts, job runners, background workers, and dashboards.

## Common Request Fields

The exact crawl body can vary by workflow, but the most common fields are:

| Field | Description |
| --- | --- |
| `url` | Required starting URL |
| `limit` | Maximum number of pages to collect |
| `maxPages` / `max_pages` | Supported aliases for crawl caps during compatibility-oriented migrations |

Start small. A crawl with `limit: 5` is much easier to inspect than a crawl with `limit: 500`.

## Best Uses

- knowledge-base ingestion,
- site audits,
- internal search index refreshes,
- and agent workflows that need to recurse beyond a starting page.

## A Practical Evaluation Loop

The safest way to evaluate a new site is:

1. run `map` first to understand the reachable section,
2. launch a crawl with a low page cap,
3. inspect the resulting markdown or extraction output,
4. then widen the scope only after the first batch looks good.

That sequence saves credits and helps you catch bad starting URLs early.

## Credit and Retry Behavior

`crawl` billing is different from `scrape` because results materialize over time.

- starting a crawl consumes the initial crawl credit,
- polling is tied to newly materialized pages,
- and transient upstream failures should be handled with retry logic rather than blind rapid polling.

If the API returns `429`, respect `Retry-After`. If the target site itself is slow or hostile, reducing crawl size usually gives you a clearer signal than hammering the same job harder.

## Design Note

The polling model is explicit today. That keeps the API easy to understand from scripts, pipelines, and dashboard tooling. It also makes billing and progress reporting easier to reason about than hidden background behavior.

---

# Map Endpoint Guide
Source: https://docs.fastcrw.com/map/

## Overview

`map` is the lightweight discovery tool in the stack. Use it before `crawl` when you need to answer:

- what URLs are reachable from this starting point,
- which subsection of the site is actually worth scraping,
- and whether a full crawl is justified at all.

```bash
curl -X POST https://api.fastcrw.com/v1/map \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url":"https://example.com"}'
```

## Why `map` Comes First

A lot of scraping waste comes from starting too broad. `map` helps you inspect the site structure first, which is especially useful when:

- the site has multiple product areas,
- navigation is noisy,
- or an agent needs to choose its next step instead of crawling everything.

## When To Use It

Choose `map` when you want to answer:

- what URLs are reachable from this starting point,
- what section of the site should we target next,
- and whether a full crawl is worth the cost.

## A Typical Workflow

One common pattern looks like this:

1. `map` a docs home page or product section.
2. Filter the returned URLs in your application.
3. Launch `scrape` for a handful of known-important pages.
4. Launch `crawl` only for the subset that deserves broader recursion.

That works well for AI agents, indexing systems, and human operators doing a first evaluation.

## Output Expectations

`map` is for discovery, not deep extraction. Treat it as a planning primitive:

- it helps you decide what to fetch,
- it reduces unnecessary crawl scope,
- and it makes later scrape or crawl requests more intentional.

If your end goal is page content, `map` should usually be the first step, not the last one.

## Example: Narrow a Noisy Site Before Crawl

Imagine a large docs domain with product pages, marketing pages, changelogs, and a blog mixed together. Running a broad crawl from the home page creates noise quickly.

Instead:

1. start with `map` on the docs root,
2. inspect the returned URLs,
3. keep only the section that matters,
4. then launch `crawl` on that smaller scope.

That pattern reduces wasted credits and keeps downstream systems cleaner.

## Common Mistakes

- Using `map` when you already know the exact page you need. In that case use [`/docs/scrape`](/docs/scrape) directly.
- Treating `map` output as if it were extracted content instead of URL discovery.
- Launching a full crawl from a noisy homepage before inspecting the reachable structure.

## What To Read Next

- Use [`/docs/crawl`](/docs/crawl) when you are ready to recurse after discovery.
- Use [`/docs/getting-started`](/docs/getting-started) if you need the shortest path to a working first request.
- Use [`/docs/rate-limits`](/docs/rate-limits) when map-based discovery is feeding many follow-up requests.

---

# Search Endpoint Guide
Source: https://docs.fastcrw.com/search/

## Overview

Use `search` when you need to find content across the web without knowing specific URLs upfront. It is the right choice for:

- research workflows where the agent discovers relevant pages,
- RAG pipelines that need fresh web content on a topic,
- news monitoring and trend tracking,
- and competitive analysis across multiple sources.

```bash
curl -X POST https://api.fastcrw.com/v1/search \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"query":"web scraping tools","limit":5}'
```

## A Good Default Request

If you are not sure where to start, use this shape first:

```json
{
  "query": "your search terms here",
  "limit": 5
}
```

That gives you the top 5 web results with title, URL, description, and relevance score. Add `scrapeOptions` when you need the actual page content, not just search snippets.

## Parameters

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `query` | `string` | *required* | The search query (max 2000 characters) |
| `limit` | `number` | `5` | Maximum number of results per source (1–20) |
| `lang` | `string` | — | Language code for results (e.g. `"en"`, `"de"`, `"fr"`) |
| `tbs` | `string` | — | Time-based filter: `"qdr:h"` (hour), `"qdr:d"` (day), `"qdr:w"` (week), `"qdr:m"` (month), `"qdr:y"` (year) |
| `sources` | `string[]` | — | Result types to return: `"web"`, `"news"`, `"images"`. When set, response is grouped by source |
| `categories` | `string[]` | — | Filter by category: `"github"`, `"research"`, `"pdf"` |
| `scrapeOptions` | `object` | — | Scrape each result URL. See below |

### scrapeOptions

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `formats` | `string[]` | `["markdown"]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"` |
| `onlyMainContent` | `boolean` | `true` | Extract primary content area only |

## Response Format

### Flat response (default — no `sources`)

When `sources` is not set, results are returned as a flat array sorted by relevance:

```json
{
  "success": true,
  "data": [
    {
      "url": "https://example.com/article",
      "title": "Article Title",
      "description": "A snippet from the search result...",
      "position": 1,
      "score": 9.5,
      "category": "general"
    }
  ]
}
```

### Grouped response (with `sources`)

When `sources` is set, results are grouped by type. The `limit` applies per source:

```json
{
  "success": true,
  "data": {
    "web": [
      { "url": "...", "title": "...", "description": "...", "position": 1, "score": 9.5 }
    ],
    "news": [
      { "url": "...", "title": "...", "description": "...", "position": 1, "publishedDate": "2026-04-02T14:00:00" }
    ]
  }
}
```

## Search + Scrape

The real power of the search endpoint is combining search with content scraping in a single call. Add `scrapeOptions` to fetch the full page content for each result:

```bash
curl -X POST https://api.fastcrw.com/v1/search \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "query": "machine learning transformers",
    "limit": 3,
    "scrapeOptions": {
      "formats": ["markdown"]
    }
  }'
```

Each result in the response will include the scraped content:

```json
{
  "url": "https://example.com/article",
  "title": "Understanding Transformers",
  "description": "Search snippet...",
  "position": 1,
  "markdown": "# Understanding Transformers\n\nThe transformer architecture...",
  "metadata": { "statusCode": 200 }
}
```

If a particular URL fails to scrape (anti-bot, timeout), the result is still returned with the search metadata but without the scraped content. The credit for that failed scrape is refunded.

> **Note:** When `scrapeOptions` is combined with `sources`, only `web` results are scraped. News and image results return search metadata only.

## News Search

Search specifically for recent news articles:

```bash
curl -X POST https://api.fastcrw.com/v1/search \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"query":"artificial intelligence","sources":["news"],"limit":5}'
```

News results include a `publishedDate` field with the article publication timestamp.

## Image Search

Search for images across the web:

```bash
curl -X POST https://api.fastcrw.com/v1/search \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"query":"neural network diagram","sources":["images"],"limit":5}'
```

Image results include `imageUrl`, `thumbnailUrl`, `imageFormat`, and `resolution` fields.

## Time-Based Search

Filter results by recency using the `tbs` parameter:

| Value | Meaning |
| --- | --- |
| `qdr:h` | Past hour |
| `qdr:d` | Past 24 hours |
| `qdr:w` | Past week |
| `qdr:m` | Past month |
| `qdr:y` | Past year |

```json
{
  "query": "latest AI announcements",
  "tbs": "qdr:d",
  "limit": 5
}
```

> **Note:** `qdr:h` maps to day-level precision due to backend limitations. Hourly granularity is not available.

## Category Filtering

Focus your search on specific content categories:

- `"github"` — search within GitHub repositories, code, and issues
- `"research"` — search academic sources (arXiv, Semantic Scholar, OpenAlex)
- `"pdf"` — search for PDF documents

```json
{
  "query": "web scraping python",
  "categories": ["github"],
  "limit": 5
}
```

Categories can be combined: `"categories": ["github", "research"]`.

## Credit Cost

| Operation | Cost |
| --- | --- |
| Search (without scraping) | 2 credits |
| Search + scrape | 2 credits + 1 per scraped result |

If you search with `limit: 5` and `scrapeOptions`, and all 5 results scrape successfully, the total cost is 6 credits (1 search + 5 scrapes). Failed scrapes are refunded.

## Common Mistakes

- **Empty query** — the `query` field is required and must be at least 1 character.
- **Too many results** — `limit` caps at 20. Start with 5 and increase only if needed.
- **Scraping everything** — adding `scrapeOptions` multiplies the credit cost. Only use it when you actually need the page content, not just search snippets.
- **Mixing sources and categories** — `sources` controls the *type* of results (web, news, images). `categories` controls the *domain* filter (github, research). They work independently.

## What to Read Next

- [Scrape endpoint](/docs/scrape) — for scraping specific URLs you already know.
- [Extract endpoint](/docs/extract) — for structured JSON extraction from pages.
- [Credit costs](/docs/credit-costs) — full billing breakdown across all endpoints.
- [SDK examples](/docs/sdk-examples) — search examples in TypeScript, Python, and Go.

---

# Structured Extraction Guide
Source: https://docs.fastcrw.com/extract/

## Overview

Use extraction when you need shape, not just text. Send `formats: ["json"]` together with a `jsonSchema` to get structured output.

```json
{
  "url": "https://news.ycombinator.com",
  "formats": ["json"],
  "jsonSchema": {
    "type": "object",
    "properties": {
      "stories": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "title": { "type": "string" },
            "url": { "type": "string" }
          }
        }
      }
    }
  }
}
```

> **Firecrawl compatibility:** `formats: ["extract"]` is accepted as an alias for `"json"`. Both work identically, but `"json"` is the canonical format name.

## When Extraction Is the Right Tool

Use structured extraction when the downstream consumer expects fields, not prose. Common examples:

- product catalogs,
- article metadata,
- event pages,
- directory listings,
- and pages that will be turned into records for an app or database.

If your next step is retrieval, summarization, or semantic search, markdown is often the better primary output. If your next step is validation, storage, enrichment, or analytics, JSON is usually the better fit.

## End-to-End Request Example

```bash
curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://example.com/product/123",
    "formats":["json"],
    "jsonSchema":{
      "type":"object",
      "properties":{
        "title":{"type":"string"},
        "price":{"type":"string"},
        "availability":{"type":"string"}
      },
      "required":["title"]
    }
  }'
```

Start with the smallest schema that is genuinely useful. Overspecified schemas fail more often and are harder to debug.

## Where It Helps

- e-commerce ingestion,
- article metadata extraction,
- directory parsing,
- and AI workflows that need records rather than prose.

Use extraction when the downstream consumer expects fields, not a markdown blob.

## Designing a Good Schema

Strong extraction schemas share a few traits:

- they ask only for fields that truly matter,
- they avoid ambiguous nested structures when a flat object will do,
- and they match the actual information density of the page.

Good first schema:

```json
{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "author": { "type": "string" },
    "publishedAt": { "type": "string" }
  }
}
```

Risky first schema:

```json
{
  "type": "object",
  "properties": {
    "sections": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "title": { "type": "string" },
          "subsections": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "title": { "type": "string" },
                "bullets": {
                  "type": "array",
                  "items": { "type": "string" }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

The second schema may be valid, but it asks the model to infer a lot of structure that may not exist clearly on the page.

## The Managed LLM

Structured extraction on the managed cloud runs on fastCRW's managed LLM. There is
no key, provider, or model to configure and nothing to pass on the request:

```json
{
  "url": "https://example.com",
  "formats": ["json"],
  "jsonSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "description": { "type": "string" }
    }
  }
}
```

LLM features (structured extraction, the `summary` format, and search `answer` /
`summarizeResults`) are available on paid plans only; the FREE plan returns HTTP 402
for any LLM-backed request. Billing is 1 scrape credit per page plus the LLM cost for
that page, metered from real usage rather than a flat fee.

## Operational Advice

Extraction is usually best as a second step:

1. verify the page with markdown first,
2. confirm the target page actually contains the data you want,
3. then layer on the JSON schema.

That saves time when a target page is blocked, incomplete, or structurally noisy.

## Common Mistakes

- **Missing `jsonSchema`**: If you send `formats: ["json"]` without a `jsonSchema`, the API returns a 400 error. You must provide a schema.
- **Wrong format name**: `formats: ["extract"]` works but `"json"` is preferred. `formats: ["llm-extract"]` is also accepted.
- **LLM features on the FREE plan**: structured extraction requires a paid plan; the FREE plan returns HTTP 402 for any LLM-backed request.
- **Schema too ambitious**: Start with a minimal schema and widen it after you verify extraction quality on real examples.

---

# Formats Reference
Source: https://docs.fastcrw.com/output-formats/

## Supported values

| Format | Meaning |
| --- | --- |
| `markdown` | Clean markdown output (default) |
| `html` | Cleaned HTML after extraction pipeline |
| `rawHtml` | Raw fetched HTML |
| `plainText` | Plain text view |
| `links` | Extracted absolute links |
| `json` | Structured extraction result when `jsonSchema` is provided |
| `extract` | Alias for `json` — accepted for Firecrawl compatibility |

You can request multiple formats in a single call: `formats: ["markdown", "html", "links"]`.

## Which Format Should You Choose?

The practical rule is simple:

- choose `markdown` when the output is headed into search, RAG, summarization, or LLM prompts,
- choose `html` when you still want cleaned structure,
- choose `rawHtml` only when you truly need the original source,
- choose `links` when discovery matters as much as page content,
- and choose `json` when the end result needs to be schema-shaped.

For most product and retrieval workflows, `markdown` is the best default because it is compact, readable, and easier to inspect than raw markup.

## Common Format Combinations

| Combination | Good for |
| --- | --- |
| `["markdown"]` | Default page extraction |
| `["markdown", "links"]` | Content plus local link discovery |
| `["html", "rawHtml"]` | Debugging the extraction pipeline |
| `["json"]` | Structured extraction only |
| `["markdown", "json"]` | Human-readable content plus structured fields |

## Response shape

Each format populates a corresponding field in the response `data` object:

| Format | Response field | Type |
| --- | --- | --- |
| `markdown` | `markdown` | `string` |
| `html` | `html` | `string` |
| `rawHtml` | `rawHtml` | `string` |
| `plainText` | `plainText` | `string` |
| `links` | `links` | `string[]` |
| `json` / `extract` | `json` | `object` |

## Full response schema

Every API response follows this envelope:

```json
{
  "success": true,           // false if the request or target failed
  "data": { ... },           // present on success (scrape/crawl data)
  "error": "...",            // present on failure — human-readable message
  "warning": "..."           // present when something non-fatal happened
}
```

The exact shape of `data` depends on what you requested. Do not assume every field is always present.

### `data` object (scrape)

| Field | Type | Present when |
| --- | --- | --- |
| `markdown` | `string \| null` | `formats` includes `markdown` or `json` |
| `html` | `string \| null` | `formats` includes `html` |
| `rawHtml` | `string \| null` | `formats` includes `rawHtml` |
| `plainText` | `string \| null` | `formats` includes `plainText` |
| `links` | `string[] \| null` | `formats` includes `links` |
| `json` | `object \| null` | `formats` includes `json` AND `jsonSchema` provided AND LLM configured |
| `chunks` | `ChunkResult[] \| null` | `chunkStrategy` provided |
| `warning` | `string \| null` | Target returned error status, anti-bot detected, etc. |
| `metadata` | `object` | Always |

### `metadata` object

| Field | Type | Description |
| --- | --- | --- |
| `title` | `string \| null` | Page `<title>` |
| `description` | `string \| null` | Meta description |
| `ogTitle` | `string \| null` | Open Graph title |
| `ogDescription` | `string \| null` | Open Graph description |
| `ogImage` | `string \| null` | Open Graph image URL |
| `canonicalUrl` | `string \| null` | Canonical link |
| `sourceURL` | `string` | Final URL after redirects |
| `language` | `string \| null` | `<html lang>` value |
| `statusCode` | `number` | Target HTTP status code |
| `renderedWith` | `string \| null` | `"cdp"`, `"http_only"`, or `"http_only_fallback"` |
| `elapsedMs` | `number` | Total processing time in ms |

### `ChunkResult` object

| Field | Type | Description |
| --- | --- | --- |
| `content` | `string` | Chunk text |
| `score` | `number \| null` | Relevance score (present when `query` + `filterMode` set) |
| `index` | `number` | Original chunk position |

## Format aliases

`"extract"` and `"llm-extract"` are accepted as aliases for `"json"`. The canonical name is `json`. All three behave identically — they require `jsonSchema` for structured extraction.

## Implementation Guidance

Three habits keep format usage sane in production:

- request only the formats you really consume,
- keep `metadata` with the stored output so later debugging is easier,
- and validate `data.json` in your own application before trusting it as final truth.

If you are debugging extraction quality, request both `markdown` and `json` for a while. That makes it easy to compare the page text against the structured output.

## Not supported in this release

- `screenshot` — not a native `/v1` format. Page capture is available on the `/v2` compatibility surface: `POST /v2/scrape` with `formats: ["screenshot"]` returns `data.screenshot` as a base64 PNG data URL.
- `actions` — click/scroll/wait actions are not yet supported. Sending `actions` will return a 400 error with a message suggesting `cssSelector` or `xpath` as alternatives.

If your workload depends on browser actions, do not assume they exist in the managed cloud.

---

# JavaScript Rendering
Source: https://docs.fastcrw.com/js-rendering/

## How it works

When JS rendering is enabled, the renderer navigates to the page, waits for load completion, then applies numeric `waitFor` before extracting HTML.

There are two separate decisions here:

- whether the page needs a browser at all,
- and how long to wait after the page loads.

## Example

```json
{
  "url": "https://quotes.toscrape.com/js/",
  "renderJs": true,
  "waitFor": 2000
}
```

## When To Turn It On

Enable JS rendering when the page content is not present in the initial HTML response. Typical examples:

- single-page applications,
- pages that fetch content after hydration,
- and sites where the meaningful body is assembled client-side.

Do not enable it blindly for every request. HTTP-only fetches are faster and cheaper.

## Important notes

- `waitFor` is milliseconds.
- `warning` may still appear if the rendered page is an anti-bot interstitial.
- If you do not need browser rendering, keep `renderJs` off for lower latency.

## Choosing a `waitFor` Value

Start with the smallest value that works:

- `500` to `1000` for lightly hydrated pages,
- `2000` for typical JS-heavy pages,
- `3000` to `5000` only when you have confirmed the target hydrates slowly.

Long waits are not automatically safer. They increase latency and can hide the fact that the page is blocked rather than merely slow.

## Cloud vs self-hosted

- **Cloud**: JS rendering is always available. The managed infrastructure runs a LightPanda sidecar alongside the engine.
- **Self-hosted**: You must run `crw-server setup` or configure a CDP browser (LightPanda, Chrome, or Playwright) in your `config.toml` under `[renderer]`. If no JS renderer is configured, requests with `renderJs: true` will fall back to HTTP-only fetching and include a warning.

## What To Inspect in the Response

When rendered output looks wrong, check:

- `metadata.renderedWith` to verify a browser was actually used,
- `metadata.elapsedMs` to understand the cost of the request,
- and `warning` to catch anti-bot or fallback situations.

## Troubleshooting

- **Empty content from JS-heavy sites**: Increase `waitFor` (e.g., `3000`–`5000`). Some SPAs need extra time to hydrate.
- **`rendered_with: "http_only_fallback"` in metadata**: JS rendering was requested but no renderer is available. Check your deployment configuration.
- **Internal error on `renderJs: true`**: Verify the LightPanda sidecar is running and reachable. Check `/api/health` for renderer status.
- **Still poor output after increasing `waitFor`**: The issue may be anti-bot protection or authentication flow, not rendering delay.

---

# Errors and Warnings
Source: https://docs.fastcrw.com/error-codes/

## Response semantics

- `success: true` means the pipeline produced usable page content.
- `success: false` means the request failed — either an engine error or the target returned an error status (4xx/5xx) with minimal content.
- `error` is present when `success: false`. It describes what went wrong.
- `warning` flags degraded target outcomes (anti-bot pages, problematic status codes) when `success: true` — meaning content was produced but may be incomplete.
- `metadata.statusCode` is the target site's HTTP status.
- `data` may still be present when `success: false` if partial content was extracted.

The important distinction is that there are two layers of status:

- the HTTP status returned by fastCRW,
- and the target site's own status exposed through `metadata.statusCode`.

You need both to debug real scraping failures.

## HTTP status codes returned by the API

| Status | Meaning |
| --- | --- |
| 200 | Success |
| 400 | Invalid request parameters (bad URL, invalid JSON body, invalid selector) |
| 401 | Invalid or missing API key |
| 404 | Endpoint not found |
| 422 | Validation failed (unknown format, invalid schema, extraction error) |
| 429 | Rate limit or credit quota exceeded |
| 502 | Engine internal error |
| 503 | Server at capacity |
| 504 | Request timed out |

## How To Read Common Cases

| Situation | What it usually means | What to do next |
| --- | --- | --- |
| HTTP `200` with `warning` | The request succeeded, but the target result is degraded | Inspect `warning` and `metadata.statusCode` |
| HTTP `400` | Your request body is invalid | Fix fields, selectors, or schema |
| HTTP `422` | The request shape is valid JSON but semantically invalid | Check format names, schema, or extraction config |
| HTTP `429` | Rate limit or credit ceiling hit | Back off and honor `Retry-After` |
| HTTP `502` / `504` | Upstream or timeout issue | Retry with backoff |

## Common engine errors

| Error | When it happens |
| --- | --- |
| Invalid URL | URL is malformed or targets a blocked address |
| Invalid selector | CSS selector or XPath expression cannot be parsed |
| Renderer timeout | JS rendering exceeded the page timeout |
| Navigation failed | CDP browser could not load the page |
| Response too large | Page exceeded the maximum allowed size |
| Invalid JSON schema | Schema provided for extraction is malformed |
| Extraction failure | LLM extraction failed (no LLM configured, or LLM returned an error) |
| No JS renderer available | `renderJs: true` but no CDP browser is configured |

## Common warnings

| Warning | When it appears |
| --- | --- |
| `Target returned 403 Forbidden` | Target site blocked the request |
| `Target returned 429 Too Many Requests` | Target site rate limited the request |
| `Blocked by anti-bot protection` | Page contains Cloudflare/captcha markers |
| `JS rendering was requested but no renderer is available` | Fallback to HTTP-only fetch |

## Retry Guidance

Retrying helps only for some classes of failure:

- retry `429`, `502`, and `504` with backoff,
- do not blindly retry `400` or `422`,
- and treat repeated warnings from the same domain as a target compatibility problem, not a random transient issue.

If a page is repeatedly blocked by anti-bot protection, longer retry loops usually make the situation worse rather than better.

---

# Rate Limits
Source: https://docs.fastcrw.com/rate-limits/

## Window

fastCRW cloud uses a **per 60-second sliding window** for API-key rate limiting. The window slides continuously — it counts requests in the last 60 seconds from the current moment, not fixed calendar minutes.

That means a burst at `12:00:20` still affects what you can send at `12:01:00`. Think in rolling windows, not top-of-minute resets.

## Current per-plan limits

The limit you plan against is concurrency: how many requests your API key may have in flight at the same time.

| Plan | Concurrent requests |
| --- | --- |
| FREE | 3 |
| HOBBY | 10 |
| STANDARD | 50 |
| GROWTH | 100 |
| SCALE | 150 |

A per-minute request cap also exists as an anti-abuse guard, set well above what a client saturating its plan's concurrency produces. Honor `Retry-After` on a 429.

## Example: Handling `429` Correctly

When the API limit is hit, the recovery path should be mechanical:

```ts
if (res.status === 429) {
  const retryAfter = Number(res.headers.get("Retry-After") ?? "1");
  await sleep(retryAfter * 1000);
  return retryRequest();
}
```

That is different from a target-site `429`. If the target site rate-limits you, the fastCRW request may still complete at the HTTP layer while reporting the target failure inside `metadata.statusCode` or `warning`.

## Headers

- `Retry-After`
- `X-RateLimit-Limit`
- `X-RateLimit-Remaining`

When you receive a `429`, back off for the number of seconds specified in `Retry-After` before retrying. Sending requests during the backoff period will not reset the window but will be rejected.

## Practical Client Behavior

A well-behaved client should:

- read `X-RateLimit-Remaining` on every response,
- reduce concurrency before it reaches zero,
- and honor `Retry-After` exactly when a `429` arrives.

If you are running many workers in parallel, centralize throttling instead of letting each worker discover the limit independently.

## When To Add Client-Side Throttling

Add a shared limiter before production when:

- multiple workers share one API key,
- one request can fan out into many crawl polls,
- or you are likely to burst after a queue drain or deploy.

The problem with per-worker retry logic is that it reacts too late. A central limiter prevents the avoidable `429`s in the first place.

## Rate Limits vs Target Limits

The fastCRW API rate limit is separate from the target website's own rate limit.

- fastCRW may return `429` because your API key exceeded plan limits,
- or the target site may return `429`, which appears in `metadata.statusCode` or as a warning.

Those are different problems and should be handled differently.

## Common Mistakes

- Assuming the window resets exactly at the top of the minute.
- Retrying immediately after a `429` instead of honoring `Retry-After`.
- Confusing API plan limits with the target website's own anti-bot or rate-limit behavior.

For rollout work, pair this page with [`/docs/credit-costs`](/docs/credit-costs) so request throttling and credit monitoring stay aligned.

---

# Credit Costs
Source: https://docs.fastcrw.com/credit-costs/

## Current billing rules

| Operation | Credit cost |
| --- | --- |
| `scrape` | 1 credit |
| `map` | 1 credit |
| `crawl` start | 1 credit |
| `crawl` polling | New pages discovered since the previous poll |
| `search` | 2 credits |
| `search` + scrape | 2 credits + 1 per scraped result |
| `browse` session | 1 credit (planned cloud rate; the self-hosted `crw-browse` binary is free) |

## Why crawl billing looks different

The crawl start reserves the job. Subsequent polls charge only for newly materialized pages, not for the total accumulated page count each time.

That prevents the same already-seen pages from being charged again and again just because you are checking progress.

## Simple Examples

| Scenario | Credit effect |
| --- | --- |
| One `scrape` request | 1 credit |
| One `map` request | 1 credit |
| Start one crawl job | 1 credit |
| Poll a crawl and receive 7 new pages | 7 additional credits |
| Poll again with no new pages | No new page credits |
| Search for "AI tools" with 5 results | 2 credits |
| Search + scrape 3 results | 1 + 3 = 4 credits |
| Search + scrape, 1 scrape fails | 1 + 2 = 3 credits (failed scrape refunded) |

## What Usually Does Not Consume Permanent Credits

The billing logic is designed to avoid charging you for requests that never become real usable work. Validation failures and certain upstream failures are refunded rather than treated like successful paid execution.

The safest way to confirm actual consumption is still the balance endpoint before and after a test.

## Balance check

Use `GET /api/v1/account/balance` with your API key to inspect included credits, purchased balance, and total available credits.

## Example Monitoring Pattern

A simple integration-safe pattern is:

1. read balance before a new workflow rollout,
2. run a bounded test batch,
3. read balance again,
4. compare expected consumption with actual consumption.

That is especially useful for crawl jobs because the start request and the page-materialization charges happen at different times.

## When To Watch Credits Closely

Watch credits closely when:

- you are polling crawl jobs at high frequency,
- many workers share one account balance,
- or you are benchmarking output quality across multiple target sites.

If request rate is the only thing you monitor, you can still be surprised by crawl-heavy usage. Pair billing checks with [`/docs/rate-limits`](/docs/rate-limits) so throughput and credit consumption are interpreted together.

## Operational Advice

- Use small test batches before large crawls.
- Check balance before and after integration changes.
- Separate "request volume" monitoring from "credit consumption" monitoring; they are related but not identical.

## Common Mistakes

- Assuming every crawl poll re-bills the full job instead of only newly materialized pages.
- Launching large crawls before validating cost on a much smaller limit.
- Treating validation failures and refunded work as if they were successful billable jobs.

---

# MCP Integration Guide
Source: https://docs.fastcrw.com/mcp/

## Overview

If your agent runtime already speaks MCP, you can expose fastCRW as a standard tool instead of wrapping the HTTP API yourself.

Install with a single command:

```bash
curl -fsSL https://fastcrw.com/install | sh
```

Then add to your MCP client config:

```json
{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp"
    }
  }
}
```

## When MCP Helps

MCP is useful when the agent host already expects tools to be registered through a standard interface. That reduces one layer of custom glue code between the agent and your scraping service.

Typical fits:

- agentic research workflows,
- internal copilots that need current website content,
- multi-tool assistants that combine search, scrape, and synthesis,
- and developer environments such as Claude or Cursor where MCP is already the preferred integration path.

## Why It Matters

MCP gives agents a consistent way to discover and call tools. That matters when the workflow moves from finding URLs to scraping content and handing the result to downstream retrieval or reasoning steps.

## A Practical Workflow

One common setup looks like this:

1. the agent identifies a site or page it needs,
2. it calls an MCP-exposed fastCRW tool,
3. fastCRW returns scrape, map, or crawl output,
4. the agent decides whether to continue exploring or move into summarization, ranking, or retrieval.

The value is not just convenience. It also keeps the boundary clear: fastCRW fetches and extracts, while the agent decides what to do next.

## Hosted vs Self-Hosted

MCP is an integration layer, not a hosting mode. You can use the same general approach whether fastCRW is:

- running as the managed cloud API,
- running inside your own infrastructure,
- or wrapped into a broader internal platform.

## Operational Notes

- Keep MCP tool descriptions tight so the agent knows when to use `map` versus `scrape`.
- Start with read-only scraping tools before exposing anything more complex in the same MCP server.
- Log tool usage separately from downstream agent reasoning so debugging stays tractable.

## Example Agent Tool Flow

A clean MCP setup often assigns each fastCRW route a narrow purpose:

- `map` for discovery,
- `scrape` for single-page extraction,
- `crawl` for bounded recursive work.

That keeps tool selection obvious for the host agent. If you expose one broad "web tool" instead, agents tend to overuse it and produce noisier traces.

## When MCP Is Better Than Direct HTTP

Choose MCP when the host environment already expects tool discovery through a shared protocol, especially in local agent runtimes or IDE workflows. Choose direct HTTP when your application already owns orchestration and just needs API access from the backend.

In other words, MCP is ideal when the caller is an agent platform. Direct HTTP is often simpler when the caller is your own service code.

## Common Mistakes

- Registering ambiguous tool descriptions that do not explain when to use `map` versus `scrape`.
- Mixing operational secrets and agent prompts in the same configuration surface.
- Assuming MCP replaces deployment or auth decisions; it only standardizes the tool interface.

For production agent setups, pair this guide with [`/docs/compatibility`](/docs/compatibility) and [`/docs/self-hosting`](/docs/self-hosting) so tool wiring and runtime ownership stay aligned.

---

# Integrations Catalog
Source: https://docs.fastcrw.com/integrations/

## Overview

fastCRW integrates with agent frameworks, automation tools, and custom applications through multiple paths. Choose the one that fits your stack.

## Integration Methods

### MCP (Model Context Protocol)

Best for: Agent runtimes that already support MCP (Claude, Cursor, and similar tools).

MCP exposes fastCRW as a standard tool that agents can call without custom HTTP code. See the [MCP guide](/docs/mcp) for setup instructions.

```json
{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp"
    }
  }
}
```

### Direct HTTP API

Best for: Any language or framework. Maximum control over requests.

All endpoints accept JSON and return JSON. See the [getting started guide](/docs/getting-started) for authentication and basic usage.

```bash
curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

### Self-Hosted API

Best for: Teams that need full control over data and infrastructure.

The self-hosted version exposes the same API surface as the cloud version. See the [self-hosting guide](/docs/self-hosting) for deployment instructions.

## Agent Framework Compatibility

| Framework | Integration path | Notes |
| --- | --- | --- |
| LangChain | HTTP API via custom tool | Define a tool that calls the scrape/crawl endpoints |
| LlamaIndex | HTTP API via custom reader | Build a reader that uses fastCRW for web content |
| CrewAI | MCP or HTTP API | MCP is the simpler path if supported |
| AutoGen | HTTP API via function calling | Register fastCRW endpoints as callable functions |
| Claude (Anthropic) | MCP | Native MCP support in Claude Desktop and API |
| Cursor | MCP | Add to MCP config for in-editor web access |

## Automation Tools

| Tool | Integration path | Notes |
| --- | --- | --- |
| n8n | HTTP Request node | Point to the fastCRW API with your key |
| Make (Integromat) | HTTP module | Standard REST API integration |
| Zapier | Webhooks | Use webhook triggers with the API |
| GitHub Actions | curl in workflow | Useful for scheduled scraping jobs |

## Building Custom Integrations

The API is straightforward enough that most integrations are a thin wrapper:

1. Set the `Authorization` header with your API key.
2. POST JSON to the endpoint you need (`/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/search`).
3. Parse the JSON response.

No SDK is required, but the consistent API design means any HTTP client works.

## Choosing Between Cloud and Self-Hosted

| Factor | Cloud | Self-hosted |
| --- | --- | --- |
| Setup time | Instant | 5-10 minutes |
| Maintenance | Managed | You handle updates |
| Data residency | Our infrastructure | Your infrastructure |
| Cost model | Credit-based | Your server costs only |
| Rate limits | Per plan | Unlimited |

Both options expose the same API, so your integration code works with either.

## Next Steps

- [Getting Started](/docs/getting-started) — API basics and authentication
- [MCP Guide](/docs/mcp) — Set up MCP integration
- [Self-Hosting](/docs/self-hosting) — Deploy on your own servers
- [SDK Examples](/docs/sdk-examples) — Code examples in multiple languages

---

# SDK Examples
Source: https://docs.fastcrw.com/sdk-examples/

## TypeScript

```ts
const res = await fetch("https://api.fastcrw.com/v1/scrape", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com", formats: ["markdown"] }),
});

if (!res.ok) {
  throw new Error(`fastCRW error: ${res.status}`);
}

const payload = await res.json();
console.log(payload.data?.markdown);
```

This is enough for most Node.js backends, server actions, and edge handlers.

## Python

```py


res = requests.post(
    "https://api.fastcrw.com/v1/scrape",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={"url": "https://example.com", "formats": ["markdown"]},
)

res.raise_for_status()
payload = res.json()
print(payload["data"]["markdown"])
```

## Go

```go
package main


  "bytes"
  "fmt"
  "io"
  "net/http"
)

func main() {
  body := []byte(`{"url":"https://example.com","formats":["markdown"]}`)
  req, _ := http.NewRequest("POST", "https://api.fastcrw.com/v1/scrape", bytes.NewBuffer(body))
  req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
  req.Header.Set("Content-Type", "application/json")

  res, err := http.DefaultClient.Do(req)
  if err != nil {
    panic(err)
  }
  defer res.Body.Close()

  b, _ := io.ReadAll(res.Body)
  fmt.Println(string(b))
}
```

## Search

### TypeScript

```ts
const res = await fetch("https://api.fastcrw.com/v1/search", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ query: "web scraping tools 2026", limit: 5 }),
});

if (!res.ok) {
  throw new Error(`fastCRW error: ${res.status}`);
}

const { data } = await res.json();
```

### Python

```python
resp = requests.post(
    "https://api.fastcrw.com/v1/search",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"query": "web scraping tools 2026", "limit": 5},
)

resp.raise_for_status()
data = resp.json()["data"]
```

### Go

```go
body := `{"query":"web scraping tools 2026","limit":5}`
req, _ := http.NewRequest("POST", "https://api.fastcrw.com/v1/search",
    strings.NewReader(body))
req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
req.Header.Set("Content-Type", "application/json")

resp, err := http.DefaultClient.Do(req)
if err != nil {
    panic(err)
}
defer resp.Body.Close()
```

### Search + Scrape

Add `scrapeOptions` to fetch page content for each result in one call:

```ts
const res = await fetch("https://api.fastcrw.com/v1/search", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    query: "machine learning papers",
    limit: 3,
    scrapeOptions: { formats: ["markdown"] },
  }),
});
```

## Crawl Polling Pattern

Every language follows the same basic loop:

1. `POST /crawl` to start the job.
2. Read the returned crawl id.
3. `GET /crawl/:id` until the job reaches a terminal state.

That is why the examples here stay close to raw HTTP instead of pretending there is an official SDK package.

## When To Use Raw HTTP

Raw HTTP is the right default when you are:

- already inside a backend service that owns retry logic,
- calling fastCRW from a queue worker or cron job,
- or integrating through an environment where extra SDK abstraction does not buy much.

That includes server actions, background jobs, serverless functions, and internal APIs that just need a predictable request shape. The main advantage is operational clarity: what you send over the wire is exactly what fastCRW receives.

## Example: Start a Crawl and Poll It

The same shape works in every language:

```ts
const start = await fetch("https://api.fastcrw.com/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/docs", limit: 10 }),
});

const { id } = await start.json();

const status = await fetch(`https://api.fastcrw.com/v1/crawl/${id}`, {
  headers: { Authorization: "Bearer YOUR_API_KEY" },
});
```

If your app already has a standard HTTP client with auth, tracing, and retries, adding a dedicated SDK layer too early usually just hides useful operational details.

## Common Mistakes

- Do not treat `response.ok` as the full success signal. You still need to inspect `warning` and `metadata.statusCode`.
- Do not hardcode retry timing for rate limits. Read `Retry-After` from the API response.
- Do not jump straight into crawl jobs before validating a single page through [`/docs/scrape`](/docs/scrape) or [`/docs/getting-started`](/docs/getting-started).

## What To Read Next

- Use [`/docs/rate-limits`](/docs/rate-limits) before adding parallel workers.
- Use [`/docs/error-codes`](/docs/error-codes) when you need machine-readable failure handling.
- Use [`/docs/formats`](/docs/formats) when you are deciding between markdown, html, and structured extraction output.

## Production Advice

- Load the API key from an environment variable.
- Respect `Retry-After` on `429`.
- Log `warning` and `metadata.statusCode`, not just the HTTP status.
- Start with `scrape` before wiring in `crawl` or extraction.

---

# Agent Onboarding Guide
Source: https://docs.fastcrw.com/agent-onboarding/

## Overview

This guide is written for AI agents and the systems that orchestrate them. It covers the minimum you need to start making API calls, the most common patterns, and how to handle errors gracefully.

## Authentication

Every request requires an API key in the `Authorization` header:

```
Authorization: Bearer YOUR_API_KEY
```

Get a key from the [dashboard](/playground) or by registering at [/register](/register).

## Available Endpoints

| Endpoint | Method | Purpose |
| --- | --- | --- |
| `/v1/scrape` | POST | Extract content from a single URL |
| `/v1/crawl` | POST | Recursively collect pages from a domain |
| `/v1/map` | POST | Discover all reachable URLs on a domain |
| `/v1/search` | POST | Search the web and return results with content |

## Quick Start Pattern

For most agent workflows, start with this sequence:

1. **Discover** what pages exist on a target domain:

```bash
curl -X POST https://api.fastcrw.com/v1/map \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

2. **Extract** content from specific pages:

```bash
curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/page", "formats": ["markdown"]}'
```

3. **Search** when you need to find relevant pages across the web:

```bash
curl -X POST https://api.fastcrw.com/v1/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "your search query", "limit": 5}'
```

## Response Format

All endpoints return JSON with a consistent structure:

```json
{
  "success": true,
  "data": { ... }
}
```

On failure:

```json
{
  "success": false,
  "error": "description of what went wrong"
}
```

## Error Handling for Agents

Agents should handle these common scenarios:

| Status | Meaning | Agent action |
| --- | --- | --- |
| 200 | Success | Process the response |
| 400 | Bad request | Fix the request parameters |
| 401 | Invalid API key | Check authentication |
| 429 | Rate limited or out of credits | Back off and retry, or alert the user |
| 500 | Server error | Retry with exponential backoff |

## Common Agent Patterns

### Research Loop

```
search(topic) → scrape(top results) → analyze → search(refined query) → repeat
```

### Site Exploration

```
map(domain) → filter URLs → scrape(relevant pages) → synthesize
```

### Monitoring

```
scrape(url) → store result → wait → scrape(url) → compare with previous
```

## Rate Limits

Check the [rate limits documentation](/docs/rate-limits) for current limits. Agents should respect `Retry-After` headers and implement exponential backoff.

## MCP Integration

If your agent runtime supports MCP, see the [MCP guide](/docs/mcp) for a simpler integration path that avoids direct HTTP calls.

---

# Self-Hosting Guide
Source: https://docs.fastcrw.com/self-hosting/

## Quick Start

The fastest way to get CRW running locally — the install script auto-detects your OS and architecture:

```bash
curl -fsSL https://fastcrw.com/install | sh
```

This downloads the latest `crw-mcp` binary for your platform (macOS Intel/Apple Silicon, Linux x64/ARM64) and installs it to `/usr/local/bin`. You can customize the install directory:

```bash
CRW_INSTALL_DIR=~/.local/bin curl -fsSL https://fastcrw.com/install | sh
```

Or use Docker if you prefer containers:

```bash
docker run -p 3000:3000 ghcr.io/us/crw:latest
```

Either option gives you a local endpoint quickly so you can validate real targets before designing a larger deployment.

## What You Get

The self-hosted path is useful when you want to:

- keep target traffic inside your own infrastructure,
- control runtime cost directly,
- and expose scrape, crawl, and map behind your own auth, network, and observability stack.

The API shape stays familiar whether you use the managed cloud or your own deployment.

## Recommended Workflow

1. Boot the service locally or on a small VPS.
2. Validate target URLs with the `scrape`, `map`, and `crawl` routes.
3. Add LightPanda only when your workload requires browser-backed rendering.
4. Put a reverse proxy, auth, and rate limits in front of it before exposing it beyond a trusted environment.

## Early Validation Checklist

Before calling the deployment production-ready, test:

- a simple static page through `scrape`,
- a JS-heavy page with `renderJs: true`,
- a small `map` request,
- a bounded `crawl` request,
- and failure cases such as invalid selectors or target-side 403 responses.

That gives you a much clearer operational picture than only testing a single happy-path URL.

## Example Deployment Pattern

A practical first production shape is:

1. run fastCRW behind a reverse proxy,
2. keep the API private to your own network or VPN,
3. enable browser rendering only when targets actually need it,
4. add auth and rate limits at the edge,
5. then roll a small real workload through the service before broader adoption.

That is enough for many teams. You do not need a large crawler platform on day one just to validate whether the product fits your workload.

## What This Setup Is Good At

This setup is a good fit when you want to keep traffic inside your own infrastructure, control costs closely, or ship a private scraping service without managing a large crawler platform. If you need public, managed capacity instead, use the hosted product and keep the same API shape.

## When Not To Self-Host

Choose the managed product instead if:

- you want immediate capacity without operating the service,
- your team does not want to manage browser dependencies,
- or the main goal is product velocity rather than infrastructure control.

Self-hosting is valuable when ownership matters. It is not automatically the best default for every team.

## Common Mistakes

- Exposing the service publicly before adding auth, TLS, and external rate limits.
- Enabling browser-backed rendering for every target instead of only the JS-heavy ones that need it.
- Declaring success after one happy-path scrape instead of testing crawl, map, and failure behavior too.

If you are continuing toward production, read [`/docs/self-hosting-hardening`](/docs/self-hosting-hardening) next and keep [`/docs/credit-costs`](/docs/credit-costs) nearby for workload sizing.

---

# Self-Hosting Hardening
Source: https://docs.fastcrw.com/self-hosting-hardening/

## Minimum hardening baseline

- Terminate TLS in front of the API.
- Run the service as a non-root user.
- Restrict inbound access to required ports only.
- Isolate renderer sidecars from unnecessary network paths.

That baseline is the starting point, not the finish line. A self-hosted scraper talks to untrusted public pages and can sit close to valuable internal systems, so it deserves the same discipline as any other internet-facing API.

## Network and Access Control

- Put a reverse proxy or gateway in front of the service.
- Restrict who can reach the API by network, identity, or both.
- Avoid exposing internal health or admin surfaces to the public internet.
- If browser rendering is enabled, isolate the renderer from internal systems it does not need to reach.

## Runtime Isolation

Treat page fetching and browser rendering as higher-risk components than your application logic.

- run them with the least privilege possible,
- keep filesystem access narrow,
- and isolate sidecars so a renderer problem does not automatically become a broader platform problem.

## Secrets and Keys

- Keep API keys, proxy credentials, and LLM keys out of image builds.
- Inject secrets at runtime through your platform's secret store.
- Rotate keys during environment changes or incident response, not only on a fixed calendar.

## Operational guidance

- Rotate API keys during deployment cutovers.
- Keep browser-rendering dependencies on the smallest possible surface area.
- Expose `/health` only where your load balancer or monitoring needs it.
- Review warning-heavy targets separately; they often indicate anti-bot defenses rather than renderer bugs.

## Monitoring and Auditability

At minimum, watch:

- API error rate,
- warning frequency,
- crawl job duration,
- renderer availability,
- and resource spikes on the browser sidecar.

Keep enough logs to answer three questions after an incident:

1. what URL or workload triggered the issue,
2. whether it was an engine problem or a target-site problem,
3. and what data, if any, was still returned.

## Example Hardening Sequence

If you are moving from a dev VM to a real environment, the order should usually be:

1. put a reverse proxy and TLS in front,
2. add auth and external rate limiting,
3. move secrets into runtime injection,
4. restrict network access around the API and any renderer sidecar,
5. then enable monitoring and alerting on warnings, failures, and resource spikes.

That order keeps the riskiest exposure points under control early instead of treating hardening as a final cleanup step.

## When To Isolate the Renderer More Aggressively

Stronger isolation is worth it when:

- your targets are highly dynamic and require frequent JS rendering,
- the service runs close to internal systems with sensitive access,
- or many tenants or workloads share the same cluster.

In those cases, a renderer problem should not become an easy pivot into the rest of your infrastructure.

## Common Mistakes

- Leaving `/health` broadly exposed when only an internal load balancer needs it.
- Running the service with broader filesystem or network access than the scraping workload requires.
- Keeping incident logs too thin to separate target-site anti-bot issues from engine regressions.

Pair this page with [`/docs/rate-limits`](/docs/rate-limits) and [`/docs/error-codes`](/docs/error-codes) so operational hardening and runtime diagnostics are documented together.

---

# Compatibility Matrix
Source: https://docs.fastcrw.com/compatibility/

## What "Compatible" Means Here

fastCRW is built around **Firecrawl-compatible workflows**, not a blanket "drop-in replacement" claim. The goal is to preserve the core request mental model so migrations are manageable, while still documenting the places where behavior differs.

## Supported alignment

| Area | Status |
| --- | --- |
| `/v1/scrape`, `/v1/crawl`, `/v1/map` route shape | Supported |
| `limit`, `maxPages`, `max_pages` for crawl caps | Supported |
| Numeric `waitFor` for JS rendering | Supported |
| `cssSelector`, `xpath`, `chunkStrategy`, `filterMode` | Supported |

## Known differences

| Area | Current behavior |
| --- | --- |
| Screenshot output | Served by the /v2 surface as a base64 data URL (screenshot@fullPage included); not produced on native /v1 |
| `success` semantics | `success: false` when target returns HTTP 4xx/5xx with minimal content; `success: true` with `warning` when target returns error status but has real content |
| JS waiting | Numeric delay only; no selector-based wait primitive in cloud docs |
| `extract` format | Accepted as alias for `json`. Use `formats: ["json"]` with `jsonSchema` for structured extraction |
| SDKs | Raw HTTP examples only, no official language SDK package |

Treat this page as the source of truth during migrations.

## Migration Checklist

If you are moving an existing Firecrawl-style integration:

1. verify `scrape`, `crawl`, and `map` request bodies against real targets,
2. confirm how your code interprets `success`, `warning`, and target-side HTTP statuses,
3. remove dependencies on unsupported features such as browser actions,
4. and compare output quality, not just endpoint shape.

Compatibility at the request level is useful, but output semantics and operational behavior matter just as much.

## Example Evaluation Workflow

A practical migration test usually looks like this:

1. take one production URL that already works in the old integration,
2. run it through [`/docs/scrape`](/docs/scrape) with the same high-level options,
3. compare markdown quality, warnings, and target status behavior,
4. repeat the test on one JS-heavy page and one failure-prone page,
5. only then update the calling code.

That sequence catches the gap between request-shape compatibility and output-quality compatibility.

## When Compatibility Is "Good Enough"

Compatibility is good enough when your migration goal is:

- preserving the existing mental model,
- keeping endpoint names and common options familiar,
- and minimizing changes in the application layer.

It is not good enough if your current system depends on features this page already marks as unsupported or behaviorally different.

## Common Mistakes

- Assuming route-name compatibility means output semantics are identical.
- Migrating a whole workload before testing `warning` handling and failure cases.
- Ignoring unsupported capabilities such as browser actions and then discovering the gap in production.

Use this page with [`/docs/formats`](/docs/formats) and [`/docs/error-codes`](/docs/error-codes) so your migration covers both payload shape and operational behavior.

---

# Changelog
Source: https://docs.fastcrw.com/changelog/

## 2026-04-02 — Search API

### New: Search Endpoint

- `POST /api/v1/search` — search the web with optional content scraping
- Supports web, news, and image results via `sources` parameter
- Time-based filtering with `tbs` parameter (`qdr:h`, `qdr:d`, `qdr:w`, `qdr:m`, `qdr:y`)
- Category filtering: `github`, `research`, `pdf`
- Optional `scrapeOptions` to scrape each result URL in one call
- 1 credit per search + 1 per scraped result (failed scrapes refunded)
- Grouped response format when `sources` is specified

## 2026-03-11 — Engine v0.0.8

This release focused on two themes:

- making extraction behavior more reliable on real-world content,
- and making the product surface easier to understand through clearer docs and validation.

### Engine (CRW)

- **Wikipedia / MediaWiki onlyMainContent fix** — `onlyMainContent: true` now correctly extracts article text from Wikipedia pages (~49% size reduction). Previously the noise handler matched `"toc"` as a substring inside `"vector-toc-available"` on the `<html>` element, removing the entire page.
- **3-tier noise pattern matching** — noise class/id matching now uses substring (long patterns), exact-token (short/ambiguous: `toc`, `share`, `social`, `comment`, `related`), and prefix (`ad-`, `ads-`) matching to avoid false positives on real content.
- **Structural element guard** — noise handler never removes `<html>`, `<head>`, `<body>`, or `<main>` elements.
- **Re-clean after readability** — readability output is re-cleaned to strip residual noise (infobox, navbox, catlinks) inside broad containers.
- **Wikipedia-aware readability** — added `.mw-parser-output`, `#mw-content-text`, `#bodyContent` to scored selectors; selectors wrapping >90% of body are skipped.
- **JSON format validation** — `formats: ["json"]` without `jsonSchema` now returns a 400 error instead of a warning.
- **Block detection skip** — pages >50 KB skip interstitial/block detection (no more false "blocked by anti-bot" on Wikipedia).
- **Null byte protection** — URLs containing `%00` or null bytes are rejected at the validation layer.
- **Request timeout** — default bumped from 60s to 120s.
- **Dockerfile fix** — corrected `cargo build` flags, added `config.docker.toml`.

### Platform

- **Free tier 500 credits** — free tier increased from 50 to 500 credits.
- **About page** — new /about page with mission, open-source philosophy, and contact info.
- **Trust section** — stats section on landing page (AGPL-3.0, small single binary, public benchmark, 1000 one-time lifetime free credits).
- **Validation errors** — upstream 422 errors now include specific guidance about valid format names and jsonSchema requirements.
- **Header cleanup** — removed `Via` header from responses.
- **Documentation** — added docs for error codes, rate limits, credit costs, JS rendering, formats, SDK examples, MCP integration, compatibility matrix, and self-hosting hardening.

### Upgrade notes

- Re-test any extraction workflow that depends on Wikipedia or MediaWiki-style content because `onlyMainContent` behavior is now more aggressive and more accurate.
- If you were relying on permissive `json` requests without a schema, update the client now; those requests return a 400 error in this release.
- If you self-host, pull the latest container image so the Dockerfile and config changes land together.

---

## 2026-03-10

### Initial Release

- Scrape, crawl, and map endpoints — Firecrawl-compatible API shape.
- Markdown-first extraction with readability scoring.
- CSS/XPath selectors, tag include/exclude filtering.
- BM25 and cosine similarity chunk filtering.
- LLM-based structured extraction with JSON Schema validation.
- JS rendering via LightPanda CDP.
- Stealth mode with browser-realistic UA rotation.
- Credit-based billing with Stripe integration.
- Self-hosting support with single-binary deployment.

### Release framing

The first release established the core product surface: a Firecrawl-compatible scrape, crawl, and map API with markdown-first extraction, optional browser rendering, and a path to self-hosting. Later releases should be read as refinements on top of that baseline, not a new product direction.

---