By the fastCRW team · Benchmark and pricing figures verified 2026-05-18 · fastCRW launch pricing expires 2026-06-01 · Verify independently before relying on numbers.
What a Perplexity clone actually is
To build a Perplexity clone you are not building an LLM — you are building a loop around one. Perplexity-style answer engines run a three-step cycle on every question: search the live web for relevant sources, scrape the top results into clean text, and synthesize a single cited answer with an LLM. The model is the easy part; the hard part is feeding it fresh, accurate context fast enough that the page doesn't feel slow.
This tutorial builds that loop in Python on fastCRW's /v1/search endpoint, which is Firecrawl-compatible REST, so the code below works unchanged against the managed cloud at fastcrw.com or a self-hosted binary. We'll wire up search, scrape and clean the results, synthesize a cited answer, then talk honestly about latency, per-query credit cost, and what this clone won't do.
The search-scrape-synthesize loop
Concretely, one query flows like this: user question → /v1/search returns ranked URLs (optionally with content already scraped) → you trim and deduplicate that context → you pass it to an LLM with a "cite your sources" prompt → you stream the answer plus a source list back to the UI. Every box in that diagram maps to one function in the code below.
Why you need a search API, not just an LLM
An LLM alone answers from its training cutoff and hallucinates URLs. A search API gives you grounding: real, current pages with real links you can cite. That grounding is the entire value proposition of an answer engine — and it is the reason latency matters. If retrieval is slow, the whole experience is slow, no matter how fast your model streams. (For the why-grounding-matters argument in depth, see reducing LLM hallucinations with web search.)
Step 1: Wire up live web search with /v1/search
fastCRW's /v1/search hits a SearXNG sidecar and can optionally scrape each result inline via scrapeOptions, so a single call can return both the ranked URLs and their clean markdown. Each search query costs 1 credit (per the fastCRW credit table); inline scraping of results bills per scraped page on top.
Here is the search leg in raw HTTP — no SDK required:
POST https://api.fastcrw.com/v1/searchwith headerAuthorization: Bearer YOUR_KEY- Body:
{ "query": "...", "limit": 6, "scrapeOptions": { "formats": ["markdown"] } }
In Python with the standard requests library:
resp = requests.post("https://api.fastcrw.com/v1/search", headers={"Authorization": f"Bearer {KEY}"}, json={"query": question, "limit": 6, "scrapeOptions": {"formats": ["markdown"]}})
The response gives you a list of results, each with a url, title, and (because we asked for it) a markdown field of already-cleaned page content. Asking for content inline saves you a second round-trip per result.
Tuning result count and scrapeOptions for content
Result count is a direct cost/quality lever. Six results is a sane default for a general answer engine: enough diversity to triangulate facts, few enough that you stay inside the LLM context window after trimming. Bump limit for research-heavy questions; drop it for snappy lookups. Setting scrapeOptions.formats to ["markdown"] keeps the payload LLM-ready instead of raw HTML — markdown is far cheaper in tokens and far easier for the model to attribute.
Base-URL swap if you already use the Firecrawl SDK
If your codebase already calls Firecrawl's SDK, you do not rewrite anything — fastCRW is Firecrawl-compatible REST and is a drop-in after a base-URL swap. Point the SDK's api_url at your fastCRW endpoint and the same search() call routes to fastCRW. That reversibility is the whole reason to standardize on the compatible surface: you can A/B the two backends on identical traffic without forking your code. (See the search API for AI agents for the wider endpoint tour.)
Step 2: Scrape and clean the top results
If you asked for scrapeOptions in Step 1, you already have markdown for each result and can skip a separate fetch. If you didn't — or you want to scrape a few extra sources the search didn't inline — call /v1/scrape per URL to get LLM-ready markdown. fastCRW's auto renderer selects chrome → lightpanda → http, so JS-heavy pages still render without you choosing an engine by hand.
Clean ingestion is where answer quality is won or lost. On Firecrawl's own public labeled dataset — 819 labeled URLs of the 1,000-URL set, measured with diagnose_3way.py on 2026-05-08 — fastCRW posted the highest truth-recall of the three tools tested at 63.74%, ahead of Crawl4AI (59.95%) and Firecrawl (56.04%). Higher recall means more of the page's real content survives into your context, so the LLM cites complete facts instead of fragments. (Scrape-success on that run was ~92% of reachable URLs, with 0 thrown errors.)
Getting LLM-ready markdown from each source
Markdown is the right currency for an answer engine: it preserves headings, lists, and link text the model needs to attribute claims, while stripping the navigation, scripts, and ad markup that waste tokens and confuse attribution. Each scraped page carries its source url, which you'll thread through to the citation step — never let a chunk of text into the prompt without the URL it came from.
Deduplicating and trimming context
Search results overlap. Two outlets syndicating the same wire story will both rank, and feeding both inflates cost and biases the answer toward whatever got repeated. Deduplicate by canonical URL and by near-identical opening paragraphs, then trim each source to its most relevant section (a simple head-of-document slice works; embeddings-based reranking is the upgrade). Keep a running token budget so the assembled context fits your model's window with room for the answer.
Step 3: Synthesize a cited answer
Now assemble the trimmed sources into a single prompt and ask the model to answer only from the provided context, with inline citations. You have two ways to run the model.
Managed answer mode, or your own synthesis call
fastCRW's /v1/search can synthesize the answer for you with answer: true — managed answer mode. On paid plans (Hobby and up) the managed path runs a managed LLM with no key of your own, metered in credits based on usage and capped at 8,000 credits per request. LLM features such as answer synthesis require a paid plan; the FREE plan has no LLM features. If you want to choose exactly which model writes the prose, skip answer: true and call the LLM yourself in your own code, passing it the scraped sources fastCRW returned.
Managed mode is the fastest way to ship the demo — one flag, no separate synthesis code. Doing the synthesis call yourself is the move once you care which model writes the prose, or want to keep the whole loop inside your own infrastructure by self-hosting and pointing at a local model.
Prompting for inline citations
The system prompt does the heavy lifting. Number each source you pass in, then instruct the model: "Answer using only the sources below. After each claim, cite the source number in brackets like [2]. If the sources don't cover it, say so." Pass the sources as a numbered block, each prefixed with its index and URL. On the way out, map the bracket numbers back to URLs to render clickable citations — that mapping is what makes the answer feel like Perplexity rather than a chatbot.
Streaming the answer to a UI
Stream tokens to the browser as the model produces them (Server-Sent Events or a chunked fetch) and render citations as they resolve. Streaming is mostly perceptual: the answer isn't faster, but the user sees motion within a second instead of staring at a spinner — which matters because the synthesis step is the slowest part of the loop.
Step 4: Latency, caching, and cost
An answer engine lives or dies on the time-to-first-token, so it's worth knowing where the seconds actually go.
Where the seconds go in an answer loop
Three legs: search, scrape, synthesize. fastCRW search averaged 880 ms over a 100-query benchmark across 10 categories (triple-bench.ts), with a 785 ms median, and won 73 of 100 latency races against Firecrawl and Tavily. In other words, retrieval is rarely your bottleneck — the slow part of the loop is the LLM generating the answer, especially for long syntheses. Optimize there first: cap output length, stream, and pick a fast model. One honest caveat on the scrape leg: fastCRW's scrape p90 is 14,157 ms — the worst of the three tools tested — because the chrome-stealth fallback that recovers hard pages is also what produces the slow tail. For most answer-engine queries the median (p50 1,914 ms) is what you feel, but a few stubborn sources can stall a single request; set per-source timeouts and answer from what returned.
Credit cost per query, end to end
Per the fastCRW credit table, a single answer query bills roughly: 1 credit for the search call, 1 credit per result scraped (2 if a result needs the chrome renderer), and — in managed answer mode — the LLM usage metered in credits, capped at 8,000 credits. A typical six-source query with managed synthesis lands well under that cap. If you run synthesis yourself in your own code instead of answer: true, you pay only the search + scrape credits to fastCRW and the model tokens go through whatever LLM your app calls. To model your real bill against plan credit allowances, see /pricing rather than hard-coding numbers — launch pricing expires 2026-06-01.
Honest limits: what this clone won't do
Disclosure: we build fastCRW, so here is where it stops, stated plainly.
fastCRW gives you the retrieval and synthesis primitives, not a managed research agent. There is no /v1/deep-research and no /v1/agent (Spark) endpoint. The loop in this tutorial is a single search-scrape-synthesize pass — it answers one question well, but it won't autonomously decompose a question into sub-queries, follow citation trails across rounds, and re-plan the way a multi-step research agent does. fastCRW is also stateless per request, so conversational follow-ups ("and what about X?") require you to carry the prior context yourself.
When to reach for a multi-step research loop instead
If your users ask broad, multi-hop questions ("compare the last three quarters across these five competitors"), one pass won't cut it — you want an iterative loop that plans sub-questions, runs the search-scrape-synthesize cycle per sub-question, and composes a report. fastCRW supplies the per-step primitive; you own the orchestration. We walk through building exactly that in build a search answer engine. And if you're weighing fastCRW search against hosted answer APIs, the fastCRW vs Tavily/Exa/Perplexity search-and-answer comparison lays out the trade-offs with numbers.
Self-hosting the engine: binary plus SearXNG sidecar
Unlike Perplexity, you can run this whole engine yourself for $0 in software cost under AGPL-3.0 — you pay only for your server.
The scrape/crawl footprint is one ~8 MB binary
fastCRW's scrape and crawl engine is a single static Rust binary — roughly an 8 MB Docker image in one container, with no Redis and no Node.js required (a structural fact from the README, not a benchmark). That's the part that drops onto a $5 VPS without a platform team. Self-hosted, it's $0 per 1,000 scrapes; the cost floor is your VPS, not a per-call cloud meter.
Live web search needs the SearXNG sidecar too
One honest footnote: the live-web search leg of /v1/search is powered by a SearXNG sidecar, so the self-hosted answer engine is "1 container + an optional sidecar," not literally one process. The scrape/crawl/synthesize path is the single binary; the moment you want your own live web search rather than the managed endpoint, you add the SearXNG container alongside it. Budget for that when you size the box.
Sources
- Search benchmark of record —
benchmarks/triple-bench.ts(100 queries, 10 categories): avg 880 ms, median 785 ms, 73/100 latency wins - Scrape benchmark of record —
bench/server-runs/RESULT_3WAY_1000_FULL.md(diagnose_3way.py, 2026-05-08): 63.74% truth-recall of 819 labeled URLs - fastCRW repo and docs — github.com/us/crw · fastcrw.com
Related: Build a search answer engine · Search API for AI agents · Reduce LLM hallucinations with web search · fastCRW vs Tavily/Exa/Perplexity