Best Chunking Strategies for RAG in 2026

By the fastCRW team · Benchmark/pricing facts verified 2026-05-29 · Verify independently before relying on any number.

The best chunking strategies for RAG start before you chunk anything

If you are choosing the best chunking strategy for your RAG pipeline, here is the uncomfortable part most guides skip: chunking is downstream of extraction. A recursive splitter, a semantic chunker, or late chunking can only work with the text you hand it. Feed it raw HTML full of nav bars, cookie banners, and broken table markup and every strategy degrades — your "semantic" boundaries land on a footer, your overlap duplicates a sidebar, and your embeddings encode boilerplate. Clean structured input is the precondition, not a nice-to-have.

This post compares seven chunking strategies — fixed-size, recursive, sentence-based, page/section-level, semantic, LLM-based, and late chunking — with code and honest trade-offs, then shows the scrape-to-chunk pipeline on top of fastCRW, a Firecrawl-compatible open-core scraper that emits clean markdown so chunk boundaries fall on real document structure. We build fastCRW, so treat the framing as interested; the chunking advice itself is tool-agnostic and you can run any of it against any extractor.

Why chunking decides RAG quality

Every chunking strategy trades the same two things against each other: context preservation versus retrieval precision. Small chunks match a narrow query precisely but lose surrounding context, so the LLM gets a fragment it cannot reason over. Large chunks keep relationships intact but dilute the embedding — a 2,000-token chunk about ten topics retrieves weakly for a query about one of them.

Three things make this irreducible to a single answer:

• Embedding model. Different models have different effective context and different sweet spots for chunk length.
• Document type. A paginated financial PDF, a flat blog post, and a code file want different boundaries.
• Query pattern. Factoid lookups favour short chunks; analytical questions need more context per chunk.

The published research disagrees with itself often enough that you should treat any "winner" as conditional and measure on your own corpus. The one robust finding across studies is that structure-aware chunking beats blind character counting — and structure only exists in the input if your extraction preserved it.

Fixed-size and recursive character splitting

Fixed-size (size-based) chunking is the simplest: pick a length, split when you hit it, repeat. Character-based counts characters; token-based counts tokens with a tokenizer like tiktoken, which matters because embedding models have token limits, not character limits. Both support overlap — the tail of one chunk repeats at the head of the next so a split sentence survives in both.

It is fast and predictable, ideal for a prototype, but it ignores structure: it will cut a table in half or split a sentence mid-clause.

Recursive character splitting is the better default. It tries a hierarchy of separators in order — paragraph break, line break, sentence, then space — and uses the highest-level one that keeps the chunk under your target. So a long paragraph splits at a sentence boundary rather than mid-word.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,           # ~10-20% overlap
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(markdown)   # markdown from your extractor

A practical default is 400–512 tokens with 10–20% overlap. Note that overlap is not free and not always worth it: more overlap means more vectors to store and embed, and some recent retrieval analyses found no measurable recall gain from overlap on certain datasets. Start small, measure, and only add overlap if your numbers improve.

Semantic and embedding-aware chunking

Semantic chunking splits on meaning rather than punctuation. It embeds each sentence, compares consecutive embeddings, and starts a new chunk where similarity drops below a threshold (commonly the 95th percentile of the gaps). The result is variable-length chunks that each hold one coherent topic.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,
)
chunks = chunker.split_text(markdown)

The honest trade-off is cost. You embed every sentence just to decide boundaries — for a 10,000-word document that is hundreds of extra embedding calls before you index anything. The reported recall gains over recursive splitting are real but modest and inconsistent: some studies show a few points, others find fixed-size chunks matching or beating semantic chunking once you account for the compute. Reach for it only when accuracy is the binding constraint and your budget tolerates embedding-per-sentence, and validate that it actually helps your queries.

Two points worth knowing: semantic chunking is only as good as the boundaries it finds in your text, and if your extractor already preserved headers and sections, a far cheaper section-level split often captures most of the same benefit.

Page-level and section-level chunking from markdown

For documents where layout carries meaning, split on the layout. Page-level chunking treats each PDF page as a chunk — sensible for financial reports, forms, and research papers where balance sheets, figures, or tables live on specific pages and breaking across them loses context. In published benchmarks across mixed document types, page-level chunking has been one of the stronger performers for paginated documents specifically; on plain text exports with arbitrary page breaks it offers nothing.

Section-level chunking is the web equivalent, and it is where clean markdown extraction pays off directly. When your scraper preserves the heading hierarchy (#, ##, ###), you can split on headings so each chunk is a whole logical section with its title attached:

import re

def split_by_heading(markdown: str):
    parts = re.split(r"(?m)^(#{1,3} .+)$", markdown)
    chunks, current = [], ""
    for part in parts:
        if re.match(r"^#{1,3} ", part or ""):
            if current.strip():
                chunks.append(current.strip())
            current = part + "\n"
        else:
            current += part or ""
    if current.strip():
        chunks.append(current.strip())
    return chunks

This is cheap (no embeddings, no LLM), it keeps tables and lists intact, and the chunk boundaries are exactly the boundaries a human author chose. It only works if the markdown is clean — which is the whole reason extraction quality sits upstream of chunking quality.

Late chunking and LLM-based chunking

Late chunking (introduced by Jina AI in 2024) inverts the usual order. Normally you split first, then embed each chunk in isolation — so a chunk saying "its population exceeds 3.85 million" has no idea "it" meant Berlin three paragraphs earlier. Late chunking runs the whole document through the transformer first, so every token embedding already carries bidirectional context, and only then applies boundaries and mean-pools the token embeddings within each span.

import requests, os

resp = requests.post(
    "https://api.jina.ai/v1/embeddings",
    headers={"Authorization": f"Bearer {os.environ['JINA_API_KEY']}"},
    json={
        "model": "jina-embeddings-v3",
        "task": "retrieval.passage",
        "late_chunking": True,
        "input": [chunk_1, chunk_2, chunk_3],  # boundaries from any strategy
    },
)
embeddings = [d["embedding"] for d in resp.json()["data"]]

Late chunking is a layer on top of any boundary strategy, not a replacement for it. It helps most on long documents with cross-references — contracts, papers where the methods depend on earlier definitions — and helps near zero on very short documents with no cross-chunk dependencies. Constraints: the model needs mean pooling and a long context window, and the whole document must fit inside that window (around 8,192 tokens for jina-embeddings-v3).

LLM-based chunking sends the document to a model and asks it to mark logical boundaries (and optionally summarise each chunk). Quality is high and context-aware, but cost and latency make it impractical at scale — it fits one-time processing of high-value content or building a gold standard to evaluate cheaper methods against. If you go this route, note that fastCRW's own LLM extraction runs on its managed LLM and cannot be pointed at a model you choose, so run your chunking LLM as a separate step in your own code.

Picking a strategy by document type

Then tune chunk size to query type: factoid queries (names, dates) do best around 256–512 tokens; analytical queries want 1024+. When content type and query type conflict, content type usually wins — test both. Before production, run 2–3 strategies on 50–100 representative documents with 20–30 real queries and measure recall and precision. Chunking strategy moves retrieval quality more than your choice of vector database does, so it is worth the experiment.

From clean scrape to chunks: a working pipeline

Every strategy above assumes clean input. Raw HTML splits badly: character-based splitting breaks <div> tags mid-element, semantic boundaries land on navigation text, and your embeddings encode menus and ads. The fix is to extract to clean markdown first, then chunk. fastCRW is a Firecrawl-compatible scraper, so if you already use the Firecrawl SDK you point it at fastCRW by swapping one base URL — the rest of your code is unchanged.

Step 1 — scrape to markdown. Using the Firecrawl-compatible SDK against fastCRW (self-hosted here on localhost:3000; for the managed cloud use https://api.fastcrw.com and your key):

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_url="http://localhost:3000", api_key="fc-...")

result = app.scrape_url("https://example.com/docs/guide", params={"formats": ["markdown"]})
markdown = result["markdown"]

Step 2 — crawl a whole site when your corpus is more than one page. fastCRW exposes the same async crawl shape (maxDepth caps at 10, maxPages at 1000):

job = app.crawl_url(
    "https://example.com/docs",
    params={"maxDepth": 3, "maxPages": 200, "scrapeOptions": {"formats": ["markdown"]}},
)
pages = [p["markdown"] for p in job["data"]]

There is no batch-scrape endpoint; for a list of URLs you iterate scrape concurrently or use crawl. Step 3 — chunk each page with the recursive splitter above (or section-level if your headings are clean), then embed and index as usual. Because the markdown preserves headers, lists, and tables, the boundaries your splitter finds are real.

Why extraction quality is the load-bearing claim here: on Firecrawl's own public scrape-content dataset, fastCRW recovered the highest truth-recall of the three tools tested — 63.74% of 819 labeled URLs (vs Crawl4AI 59.95% and Firecrawl 56.04%), measured with diagnose_3way.py on 2026-05-08 — paired with ~92% scrape-success (of reachable URLs) and 0 thrown errors over 3,000 requests. Median latency was 1914 ms (p50) — the fastest of the three. Higher truth-recall means fewer pages silently dropped from your index — and a page that never gets scraped can never be chunked well, no matter which strategy you pick.

On deployment and cost: the fastCRW engine is a single static Rust binary (~8 MB image, one container) versus a multi-container stack, so you can run extraction next to your indexer on a small box, and in self-host mode scraped content and target URLs never leave your infrastructure — which matters for regulated RAG corpora. The managed cloud is Firecrawl-compatible too; see /pricing for current tiers and github.com/us/crw for the AGPL-3.0 engine you can self-host for the cost of your own server.

For the full retrieval-and-generation side of the pipeline, see building a RAG pipeline with CRW and the scrape-to-RAG pipeline with LlamaIndex. To choose the surrounding tooling, see the best RAG tools, and for the extraction step specifically, LLM-ready markdown extraction.

Sources

• fastCRW scrape benchmark — truth-recall 63.74% of 819 labeled URLs (highest of three), p50 1914 ms (fastest), ~92% scrape-success (of reachable URLs), 0 errors over 3,000 requests (diagnose_3way.py, Firecrawl public scrape-content dataset, single run 2026-05-08). See /benchmarks.
• fastCRW engine, API surface (scrape/crawl/map/search), and self-host: github.com/us/crw (AGPL-3.0).
• Pricing: fastcrw.com/pricing.
• Late chunking method: Jina AI, "Late Chunking" (arXiv:2409.04701, 2024).

Related: RAG pipeline with CRW · Best RAG tools · LLM-ready markdown extraction · Scrape-to-RAG pipeline with LlamaIndex