LLM-Ready Markdown Extraction: Why Clean Beats Complete

By the fastCRW team · Engineering deep-dive · Last reviewed 2026-01-01

Disclosure: fastCRW emits LLM-ready markdown as its primary format and we build it. The extraction principles below apply to any scraping-for-AI pipeline.

The silent failure that ruins RAG

A RAG pipeline can be architecturally perfect — good chunking, a strong embedding model, a capable LLM — and still give wrong answers because the ingestion step fed it garbage. This failure is silent: nothing errors, the corpus looks populated, the demo works, and then in production the model confidently cites a cookie banner or misses the one paragraph that mattered because it was buried under nav markup. Extraction quality is the most under-instrumented variable in the whole AI-data stack, and "LLM-ready markdown" is the discipline of getting it right. The core counterintuitive principle: for LLM consumption, clean beats complete.

Why raw HTML is the wrong input

The naive approach — fetch HTML, hand it to the model or a generic html-to-text converter — fails for concrete reasons:

• Boilerplate dominates token count. Nav bars, footers, cookie banners, share widgets, related-posts rails, ad slots — on many pages this is the majority of the DOM. Feed it raw and you spend most of your token budget (and embedding signal) on content that is identical across every page and irrelevant to every query.
• Structure carries meaning the model needs. A heading hierarchy, a table, an ordered list, a code block — these encode relationships. Flatten them to a text blob and you destroy the very structure that lets the model and the chunker reason about the content.
• Invisible content pollutes retrieval. display:none blocks, JSON-LD, inline script/style, tracking pixels — none of it is content, all of it becomes noise in the embedding if you don't strip it.
• HTML is token-expensive. Tags, attributes, classes — every angle bracket is tokens the model pays for and learns nothing from. Markdown encodes the same structure at a fraction of the token cost.

What "LLM-ready" actually requires

Good extraction is a sequence of deliberate decisions, not a converter call:

1. Main-content detection. Identify the article/primary content region and discard chrome. This is the highest-leverage step — get it wrong and everything downstream is polluted.
2. Structure preservation. Map semantic HTML to semantic markdown: headings → # levels, tables → markdown tables (not flattened text), lists → lists, code → fenced blocks. The chunker later relies on these boundaries.
3. Noise removal. Strip scripts, styles, hidden nodes, SVG cruft, repeated boilerplate — without removing legitimate content that happens to sit in an unusual container.
4. Link and media handling. Decide policy deliberately: inline link text vs. reference links, image alt-text retention, whether to keep or drop pure-navigation links. Wrong policy here either bloats tokens or drops cited sources.
5. Encoding and whitespace normalization. Collapse the ragged whitespace real HTML produces so chunk boundaries are stable and tokens aren't wasted on layout artifacts.

The hard part is step 1 done fast. Heuristic main-content detection that's wrong on 10% of pages quietly corrupts 10% of a RAG corpus; a heavyweight extractor that's accurate but adds latency breaks agent use. The engineering target is accurate and sub-millisecond on the common case.

The token-economics argument (this is a cost line, not just quality)

Clean extraction is frequently sold as a quality improvement. It is also, measurably, a cost reduction, and that framing matters because it's verifiable on your own bill. Boilerplate that survives extraction is paid for three times: once when you embed it (embedding API calls), once when it occupies vector-store space and dilutes retrieval, and once when it rides along in retrieved context into every LLM call (prompt tokens, on every query, forever). A page that's 70% chrome, ingested raw, means roughly 70% of the ingestion and per-query context spend on that document is waste. Across a large corpus and high query volume, extraction quality is one of the largest controllable line items in an LLM application's bill — and unlike model choice, it has no quality downside to optimize.

Clean beats complete — the principle defended

There's an instinct to keep everything "just in case." For LLM pipelines this is wrong. The model doesn't benefit from having the footer available; it's harmed by it, because every irrelevant token competes for attention and dilutes the retrieved chunk's signal. The right bias is aggressive removal of non-content with conservative preservation of structure. "Complete" optimizes for not losing anything; "clean" optimizes for the model actually using what's there. RAG answer quality tracks the second, not the first. The only place "complete" wins is forensic/archival use cases — which are not RAG and should use a different format (raw HTML) deliberately.

Why this is an engine property, not a post-processing step

Teams often bolt an html-to-markdown library onto a generic scraper as an afterthought. That works for easy pages and degrades exactly where it matters: JS-rendered content the converter never sees, boilerplate that varies per site, tables that flatten. Extraction quality is best treated as a first-class output of the scraping engine, co-designed with the fetch/render decision (you can't extract main content you didn't render). fastCRW emits LLM-ready markdown as a primary format from the engine itself, with structured JSON as an alternative output at the same cost — the extraction is part of the engine, compiled and fast, not a fragile downstream script. It also means the same call that fetches the page returns the model-ready artifact, which is why it suits agent loops where a render-then-postprocess round trip would add latency.

Chunking starts at extraction, not after it

A widespread RAG mistake is treating chunking as a step that happens to whatever text the scraper emitted. In reality, chunk quality is largely determined by extraction, because good chunking needs structural boundaries to split on — section headings, list items, table rows, code-block edges. If extraction flattened the document into an undifferentiated text blob (the failure mode of generic html-to-text), the chunker has nothing principled to split on and falls back to fixed-size windows that slice mid-sentence and mid-table, scattering one coherent idea across chunk boundaries so retrieval can never assemble it. Conversely, structure-preserving markdown gives the chunker semantic seams: split on headings, keep a table or code block intact, never break a list item. The practical rule: a retrieval quality problem that looks like a chunking or embedding problem is, surprisingly often, an extraction problem one step upstream — you cannot chunk well on structure that extraction destroyed. Evaluating an extractor therefore means asking "does this output give a chunker clean boundaries," not just "is the text correct."

Tables, code, and the formats LLMs reason over differently

Not all content survives flattening equally, and the asymmetry matters for which use cases extraction quality is critical. Prose degrades gracefully — a flattened paragraph is still mostly usable to a model. Tables and code degrade catastrophically: a pricing table flattened to space-separated tokens loses the row/column relationships that were the information, and a code block stripped of fencing and indentation can become semantically wrong, not just ugly. For RAG over documentation, financial data, specs, or technical content — exactly the high-value corpora — table and code fidelity is frequently the difference between a usable and a misleading answer, and it's the first thing a generic converter sacrifices. This is why "preserve structure" isn't an aesthetic preference: markdown tables and fenced code blocks are the cheap, model-legible encodings of relationships that flattening annihilates, and an extractor's handling of these specifically is the highest-signal quality test for technical RAG, far more than its handling of prose.

How to evaluate extraction quality

1. Eyeball the markdown, not the HTTP 200. A successful fetch tells you nothing about extraction. Read the actual markdown for a sample across your real target diversity.
2. Boilerplate ratio. Roughly what fraction of output tokens is chrome vs. content? High ratio = a cost and quality leak.
3. Structure fidelity. Did tables survive as tables? Did heading hierarchy survive? These drive chunk quality.
4. JS-content coverage. Test pages whose main content is client-rendered; many extractors silently return the empty shell.
5. Latency of extraction itself. If it adds hundreds of ms, it's incompatible with agent loops; measure it separately from fetch.

Bottom line

LLM-ready markdown is not "convert the HTML" — it's main-content detection, structure preservation, and aggressive noise removal, done fast enough for agent loops. Get it wrong and you silently corrupt RAG answers and pay for the boilerplate three times over. Treat extraction quality as a first-class engine output and a real cost line, evaluate it by reading the markdown rather than trusting the 200, and bias toward clean over complete — because the model uses clean, not complete.

Try it

docker compose up   # markdown + JSON output, same cost, AGPL-3.0

Managed Cloud: one-time lifetime 500 free credits, no card. fastcrw.com · GitHub

Related: Scraping latency explained · Best web scraping API 2026 · Anti-bot & proxies overview