Skip to content

Chunking Strategies

Every RAG system starts by cutting documents into pieces small enough to embed and retrieve individually. That decision — how you cut — turns out to matter more than most people expect: a NAACL 2025 Vectara study tested 25 chunking configurations against 48 embedding models and found that chunking configuration influenced retrieval quality as much as, or more than, the embedding model choice. Teams routinely obsess over which embedding model to use and treat chunking as an afterthought — the data says that's backwards.

The tension driving every strategy on this page is the same one: cut too small and a chunk loses the surrounding context it needs to make sense; cut too large and the embedding averages across multiple ideas, diluting the specific signal a query is searching for. Everything below is a different answer to that trade-off.

Hands-on

Every strategy on this page is runnable and wraps a real, tested reference implementation — not a from-scratch reimplementation: Chunking Strategies → in the code repo. Same sample document, ten different ways of cutting it — run one, or run --strategy all and compare.

TL;DR — quick revision
  • Default: recursive chunking, 256-512 tokens, ~10-20% overlap — cheap, structure-respecting, wins on end-to-end accuracy in 2026 benchmarks despite lower raw recall than semantic
  • Chunking config rivals embedding-model choice in impact on retrieval quality (Vectara, NAACL 2025) — don't skip tuning this
  • Semantic chunking gets higher recall (91.9% vs. recursive's 85-89%) but lower end-to-end accuracy (54% vs. 69%) — fragments can get topically pure but too small to actually answer from
  • Small-to-big / parent-document is the broadly useful upgrade once precise retrieval and sufficient generation context are in tension — which is most real systems
  • Late chunking flips the order entirely: embed the whole document first (full cross-attention), split the token embeddings after — chunks keep context from the rest of the document
  • Proposition chunking goes finer than a sentence: one atomic, self-contained fact per unit (Dense X Retrieval, Chen et al. 2023)
  • Agentic chunking costs an LLM API call per document; proposition chunking costs one local model inference call per chunk (free, no API) using the paper authors' released model — both still too slow for bulk indexing of a huge corpus
  • Adaptive chunking doesn't pick a technique for you — it runs several, scores them, and picks per document, which matters once your corpus is genuinely mixed-format

Fixed-Size Chunking

The simplest thing that could possibly work: split every N tokens or characters, with a fixed overlap between consecutive chunks so a sentence sitting exactly on a boundary still appears intact in a neighboring chunk.

Parameters: chunk_size (typically 256-512 tokens to start), overlap (10-20% of chunk size).

It respects nothing about the document's actual structure — it will happily cut a sentence, a table row, or a function definition in half purely because the character count was hit. That's also its appeal: zero embedding or LLM calls to decide anything, completely predictable chunk sizes, trivial to implement and reason about.

Use it for: simple, uniform documents where structure barely matters, fast prototyping, or as a baseline before investing in anything smarter.

from langchain_text_splitters import CharacterTextSplitter


def fixed_size_chunk(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
    """Split every `chunk_size` characters, with `overlap` characters of
    redundancy between consecutive chunks. `separator=""` disables any
    structure-awareness — this is the deliberately naive baseline."""
    splitter = CharacterTextSplitter(separator="", chunk_size=chunk_size, chunk_overlap=overlap)
    return splitter.split_text(text)

Recursive Chunking

Instead of blindly cutting at a fixed offset, try a hierarchy of separators — paragraph breaks first, then sentence breaks for any piece still too large, then word breaks as a last resort. LangChain's RecursiveCharacterTextSplitter is the reference implementation, and the code below uses it directly.

Parameters: chunk_size, overlap, and the separator hierarchy itself (defaults to ["\n\n", "\n", ". ", " ", ""] — customize per document type, e.g. add "```" for code-heavy docs).

This is the default almost everyone should start with. It's still zero-cost (no embedding/LLM calls), respects natural boundaries far better than fixed-size, and — per the 2026 benchmarks cited above — actually wins on end-to-end accuracy (69%) despite semantic chunking's higher raw retrieval recall, because it doesn't over-fragment content the way topic-shift-triggered cuts can.

Use it for: the general-purpose default for markdown, HTML, and plain text — reach for something smarter only once you have eval evidence recursive isn't enough.

from langchain_text_splitters import RecursiveCharacterTextSplitter


def recursive_chunk(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
    """Try splitting on paragraph breaks first; only fall through to a
    finer separator (sentence, then word, then raw character) for any
    piece that's still too large. The reference implementation of this
    technique — most tutorials describing "recursive chunking" mean
    exactly this class."""
    splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=overlap)
    return splitter.split_text(text)

Document/Structure-Aware Chunking

Split along the document's actual declared structure — Markdown headers, HTML tags, PDF section breaks — instead of a generic separator that merely tends to correlate with structure. MarkdownHeaderTextSplitter in LangChain is the concrete analog.

Parameters: none in the traditional sense — the document's own header levels are the parameters. Optionally cap output chunk size and further split any single section that's still too large (compose with recursive chunking).

The upside is exact: chunk boundaries land exactly where the document's author already divided it, and you get rich metadata (section title, heading path) for free — genuinely useful for filtering and citations later. The downside shows up the moment the input lacks real structure: a scanned PDF or informally-written notes give this method nothing to split on, and it silently produces one giant chunk (see the demo below — this happens on the sample PDF, which is exactly why the code repo pairs this strategy with its own Markdown sample instead).

Use it for: structured documents with clear sections — Markdown files, well-tagged HTML, PDFs with genuine heading structure.

from langchain_text_splitters import MarkdownHeaderTextSplitter


def structure_aware_chunk_markdown(markdown_text: str) -> list[dict]:
    """Split on Markdown headers, keeping the heading path as metadata —
    only works when the input actually has real header structure."""
    headers_to_split_on = [("#", "h1"), ("##", "h2"), ("###", "h3")]
    splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
    docs = splitter.split_text(markdown_text)
    return [
        {"heading_path": " > ".join(doc.metadata.values()), "text": doc.page_content}
        for doc in docs
    ]

Semantic Chunking

Embed every sentence individually, walk through them in order, and cut a new chunk boundary wherever similarity to the next sentence drops below a threshold — a proxy for "the topic just shifted."

Parameters: threshold (a cosine-similarity cutoff, typically 0.5-0.75 — lower means fewer, larger chunks; needs tuning per embedding model and corpus, there's no universal default).

This produces the most topically coherent chunks of any non-LLM method — and it measurably wins on raw retrieval recall (91.9% vs. recursive's 85-89% in 2026 benchmarks). The catch, and it's an important one for interviews: higher recall doesn't mean better answers. The same benchmarks show semantic chunking's end-to-end accuracy landing at just 54% against recursive's 69%, because topic-pure fragments can end up too small or too narrow to actually contain a complete answer — a 2026 FloTorch benchmark found semantic fragments averaging just 43 tokens. It also costs real compute — embedding every sentence before you've even embedded the chunks you'll actually retrieve.

Use it for: narratives and technical documentation where topical coherence is genuinely the retrieval bottleneck — worth the extra indexing cost specifically when eval results show it, not by default.

from chonkie import SemanticChunker


def semantic_chunk(
    text: str, threshold: float = 0.6, embedding_model: str = "all-MiniLM-L6-v2"
) -> list[str]:
    """Embed sentences, walk through in order, and cut a new chunk
    whenever similarity to the next sentence drops below `threshold` (a
    topic shift). Lower threshold = fewer, larger chunks."""
    chunker = SemanticChunker(embedding_model=embedding_model, threshold=threshold)
    chunks = chunker.chunk(text)
    return [c.text for c in chunks]

Sentence-Window Retrieval

Index single sentences individually for maximally precise matching, but at retrieval time return a window of surrounding sentences instead of just the one that matched — decoupling "what gets matched" from "what the LLM actually sees."

Parameters: window (sentences on each side of the match — typically 1-3; larger windows give more context but reintroduce the dilution problem this technique exists to avoid).

A single matched sentence is often too little for the LLM to work with — a pronoun without its antecedent, a fact stated without the qualifier one sentence earlier — but indexing bigger chunks for the sake of context hurts matching precision. This pattern gets both: precise retrieval on the sentence, sufficient context from the window.

Use it for: precise fact-lookup where the exact matching unit matters (a specific clause, a specific number) but the model still needs a sentence or two of surrounding context to interpret it correctly.

from llama_index.core import Document
from llama_index.core.node_parser import SentenceWindowNodeParser


def sentence_window_chunk(text: str, window_size: int = 2) -> list[dict]:
    """Index each sentence individually (precise matching unit), but keep
    a window of `window_size` sentences on either side for generation
    context, attached as node metadata."""
    parser = SentenceWindowNodeParser.from_defaults(window_size=window_size)
    nodes = parser.get_nodes_from_documents([Document(text=text)])
    return [{"anchor": n.text, "window_text": n.metadata["window"]} for n in nodes]

Small-to-Big / Parent-Document Retrieval

Embed and index small child chunks for precise matching, but link each one back to a larger parent chunk — when a child matches at query time, fetch and pass the parent to generation, not the child itself.

Parameters: child_size (small, e.g. 150-300 tokens — the matching unit), parent_size (larger, e.g. 1000-2000 tokens — the generation-context unit), overlap on the child tier.

This is the general form of sentence-window (child = one sentence, parent = a fixed window) — generalized so the child can be any small unit and the parent any larger one, explicitly linked via a parent_id rather than a fixed N-sentences-around rule. It directly resolves the fixed-size/semantic dilemma above: small chunks retrieve precisely; big chunks carry context; this gets both by using different chunk sizes for different jobs.

Use it for: the default pattern to reach for whenever precise retrieval and sufficient generation context are in tension — which describes most production RAG systems once they've outgrown fixed-size or recursive alone.

from llama_index.core import Document
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes


def small_to_big_chunk(text: str, parent_size: int = 1500, child_size: int = 250) -> list[dict]:
    """Embed the small child chunks (precise matching); retrieval fetches
    the linked parent chunk (full context) for generation instead —
    LlamaIndex's HierarchicalNodeParser + get_leaf_nodes is the reference
    implementation, paired with AutoMergingRetriever in a full pipeline."""
    parser = HierarchicalNodeParser.from_defaults(chunk_sizes=[parent_size, child_size])
    nodes = parser.get_nodes_from_documents([Document(text=text)])
    leaf_nodes = get_leaf_nodes(nodes)
    return [
        {
            "parent_id": leaf.parent_node.node_id if leaf.parent_node else None,
            "child_text": leaf.text,
        }
        for leaf in leaf_nodes
    ]

Late Chunking

Every strategy above embeds each chunk in isolation — the embedding model sees only that chunk's text, nothing else. Late chunking inverts the order: run the entire document through a long-context embedding model first, so every token attends to every other token via full self-attention, and only then split the resulting token-level embeddings into chunks, mean-pooling each span. Jina AI introduced the technique specifically to fix a failure late chunking's every predecessor shares: a chunk like "The company grew 3% that quarter" is meaningless in isolation once split away from the sentence naming the company — late chunking keeps that reference grounded, because the token embeddings were computed with the whole document in view before any splitting happened.

Parameters: the boundary positions themselves (produced by any other chunking method — late chunking is about how the embeddings are computed, not where the cuts go), plus the underlying model's max context length (a hard ceiling on how much of the document can share attention — 8K tokens for many current long-context embedding models).

Trade-off: requires a long-context embedding model capable of producing token-level (not just pooled-sentence) output, and a full-document forward pass is more expensive than embedding small chunks independently. Anthropic's Contextual Retrieval solves a related problem — chunks losing their surrounding context — differently: it prepends an LLM-generated summary blurb (50-100 tokens) to each chunk before embedding, one LLM call per chunk (cacheable, ~90% cost savings via prompt caching on repeat runs against the same document), cutting retrieval failures 49% alone and 67% combined with reranking. Late chunking gets a similar effect without any LLM call, at the cost of needing a long-context embedding model instead.

Use it for: documents where chunks losing their surrounding reference (pronouns, "that quarter," "the aforementioned clause") is actually hurting retrieval — financial reports, legal documents, anything with heavy cross-referencing.

from chonkie import LateChunker


def late_chunk(
    text: str, chunk_size: int = 300, embedding_model: str = "nomic-ai/modernbert-embed-base"
) -> list:
    """Embed the WHOLE document first with a long-context model, so every
    token attends to every other token — THEN pool token embeddings within
    each chunk boundary. Chunk vectors carry context from the rest of the
    document, unlike every other method here, which embeds each chunk in
    isolation."""
    chunker = LateChunker(embedding_model=embedding_model, chunk_size=chunk_size)
    chunks = chunker.chunk(text)
    return [c.embedding for c in chunks]

Proposition-Based Chunking

Go finer than a sentence: decompose a passage into propositions — atomic, self-contained factual statements, each rewritten so it makes sense with zero surrounding context (pronouns resolved, implicit subjects spelled out). This is the retrieval unit introduced by Dense X Retrieval (Chen et al., 2023, EMNLP 2024), which found proposition-level retrieval beats both passage- and sentence-level retrieval on open-domain QA benchmarks — a finer retrieval unit means a higher density of relevant information per retrieved item, and a better chance the exact right fact surfaces in the top-k instead of getting buried inside a longer, only-partially-relevant chunk. The paper's authors released a model fine-tuned specifically for this decomposition — chentong00/propositionizer-wiki-flan-t5-large, a 780M-parameter T5 model — which runs entirely locally with no API cost, rather than requiring a prompted call to a large general-purpose LLM.

Parameters: input format is fixed by how the model was trained — Title: {title}. Section: {section}. Content: {content} — so the lever is really just chunking your source text into reasonably-sized passages before feeding each one through.

Trade-off: one model inference call per source chunk (free/local with the released model, or an API call if you instead prompt a general LLM for this), and — as the notes on Dense X Retrieval point out — decomposing to this granularity can weaken broader contextual continuity, since closely related facts that used to sit in one chunk now get retrieved (or not) independently of each other.

Use it for: dense factual documents where a single query needs to land on one precise fact, not a paragraph containing it — knowledge bases, structured reference material, anywhere Recall@k on exact facts is the metric that matters.

import json

from transformers import AutoTokenizer, T5ForConditionalGeneration


def proposition_chunk(content: str, title: str = "", section: str = "") -> list[str]:
    """Decompose a passage into propositions — atomic, self-contained
    factual statements — using the actual model released by the Dense X
    Retrieval authors (Chen et al., 2023), fine-tuned specifically for
    this task. Input format ("Title: ... Section: ... Content: ...") and
    JSON-list output are both fixed by how the model was trained."""
    model_name = "chentong00/propositionizer-wiki-flan-t5-large"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = T5ForConditionalGeneration.from_pretrained(model_name)

    input_text = f"Title: {title}. Section: {section}. Content: {content}"
    input_ids = tokenizer(input_text, return_tensors="pt", truncation=True).input_ids
    output_ids = model.generate(input_ids, max_new_tokens=512)
    output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)

    try:
        return json.loads(output_text)
    except json.JSONDecodeError:
        return [output_text]  # model occasionally returns plain text instead of a JSON list

Agentic Chunking

Skip the rule entirely — prompt an LLM to read the document and decide chunk boundaries directly, using genuine reading comprehension rather than a mechanical separator or similarity threshold.

Parameters: none numeric; the prompt is the entire mechanism (e.g., "keep a clause together with its stated exception, even though the exception discusses a different sub-topic").

Trade-off: the most expensive strategy on this page — an LLM call (or several, for a long document processed in windows) purely to decide where to cut, before any embedding for retrieval has even happened. It can incorporate domain/legal/logical coherence no mechanical rule can — recognizing that a clause and its exception belong together even though a similarity-threshold approach would see two different sub-topics and split them.

Use it for: high-value, complex documents where getting boundaries exactly right justifies real cost — contracts, medical records, regulatory filings. Not for bulk indexing of a large, lower-stakes corpus.

import os

from chonkie import SlumberChunker
from chonkie.genie import BaseGenie


class AnthropicGenie(BaseGenie):
    """Chonkie's SlumberChunker calls .generate() internally, iteratively,
    to decide where each chunk should end — this adapter is the only
    piece needed to point that at Anthropic (Chonkie ships OpenAI/Gemini/
    Groq/Azure genies built in, just not Anthropic)."""

    def __init__(self, model: str = "claude-sonnet-4-5"):
        import anthropic

        self.client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
        self.model = model

    def generate(self, prompt: str) -> str:
        response = self.client.messages.create(
            model=self.model, max_tokens=1024, messages=[{"role": "user", "content": prompt}]
        )
        return response.content[0].text


def agentic_chunk(text: str, chunk_size: int = 400) -> list[str]:
    """Let an LLM decide chunk boundaries directly, using its own reading
    comprehension rather than a mechanical rule. The chunking algorithm
    itself is Chonkie's SlumberChunker; AnthropicGenie above just wires
    in which LLM answers its internal "where should this chunk end"
    questions."""
    chunker = SlumberChunker(genie=AnthropicGenie(), chunk_size=chunk_size, verbose=False)
    chunks = chunker.chunk(text)
    return [c.text for c in chunks]

Adaptive Chunking

This one isn't a chunking method — it's a strategy selector. In a genuinely mixed document collection (clean Markdown docs, messy OCR'd contracts, choppy Slack exports), no single method above wins for every document type. Adaptive chunking runs several candidate strategies against a document, scores each for quality (a coherence score — average embedding similarity between sentences within each resulting chunk is one simple proxy), and picks whichever wins for that specific document, rather than a person hardcoding "use method X for type A, method Y for type B" by hand.

Parameters: the candidate strategy set to try (and each of their parameters), plus the scoring function itself — coherence is the simplest signal; production systems often also score boundary respect (did the cut land on a natural break?) and size validity.

Trade-off: real overhead — running and scoring multiple strategies per document at indexing time instead of picking one upfront. Worth it specifically because most real enterprise document collections are heterogeneous, making this more broadly applicable than it sounds at first.

Use it for: genuinely mixed-format corpora — which describes most real-world document repositories once you look closely.

import re

import numpy as np
from chonkie import SemanticChunker
from langchain_text_splitters import CharacterTextSplitter, RecursiveCharacterTextSplitter
from sentence_transformers import SentenceTransformer


def _cosine_sim(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))


def _coherence_score(chunks: list[str], embed_model: SentenceTransformer) -> float:
    """Average pairwise sentence-similarity within each chunk — a proxy
    for "do these chunks actually hold together topically?" Higher is
    better."""
    scores = []
    for chunk in chunks:
        sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", chunk) if s.strip()]
        if len(sentences) < 2:
            continue
        embeddings = embed_model.encode(sentences, convert_to_numpy=True, show_progress_bar=False)
        pairwise = [
            _cosine_sim(embeddings[i], embeddings[j])
            for i in range(len(embeddings))
            for j in range(i + 1, len(embeddings))
        ]
        if pairwise:
            scores.append(sum(pairwise) / len(pairwise))
    return sum(scores) / len(scores) if scores else 0.0


def adaptive_chunk(text: str) -> tuple[str, list[str]]:
    """Run multiple candidate strategies — each one a tested library
    implementation, not a custom one — score each for coherence, and
    return whichever wins for THIS document, rather than hardcoding one
    strategy for every document type in a mixed corpus. The selection
    logic itself is necessarily custom (no single library owns this
    meta-technique); what it selects between is not."""
    embed_model = SentenceTransformer("all-MiniLM-L6-v2")
    candidates = {
        "fixed_size": CharacterTextSplitter(
            separator="", chunk_size=512, chunk_overlap=50
        ).split_text(text),
        "recursive": RecursiveCharacterTextSplitter(
            chunk_size=512, chunk_overlap=50
        ).split_text(text),
        "semantic": [
            c.text
            for c in SemanticChunker(embedding_model="all-MiniLM-L6-v2", threshold=0.6).chunk(text)
        ],
    }
    scored = {name: _coherence_score(chunks, embed_model) for name, chunks in candidates.items()}
    best_name = max(scored, key=scored.get)
    return best_name, candidates[best_name]

Comparison table

Strategy Mechanism Cost Best for
Fixed-size Split every N tokens, fixed overlap None Simple/uniform docs, fast prototyping
Recursive Paragraph → sentence → word fallback None General-purpose default — start here
Structure-aware Split on the document's own headers/tags Parsing only Markdown, tagged HTML, structured PDFs
Semantic Embed sentences, cut on similarity drop Medium (embed every sentence) Topical coherence is the actual bottleneck
Sentence-window Index sentences, retrieve a window around each Low-medium Precise fact lookup needing some surrounding context
Small-to-big Embed small children, generate from linked parent Medium (2-tier storage) Precision vs. context tension — most systems, eventually
Late chunking Embed whole doc first, pool token spans after Medium-high (long-context forward pass) Cross-referencing documents, chunks losing meaning in isolation
Proposition LLM extracts atomic self-contained facts High (LLM call per chunk) Dense factual QA needing exact-fact precision
Agentic LLM decides boundaries directly Highest (LLM call per document) High-value complex documents — contracts, medical records
Adaptive Run multiple strategies, score, pick winner Medium-high (multiple strategies run) Genuinely heterogeneous document collections

Don't hand-roll a splitter — these are solved problems

Every technique on this page is old and well-understood enough that a mature, widely-used library already implements it correctly — chunk boundaries, overlap semantics, and edge cases included. The code in this page's companion repo deliberately wraps real implementations rather than reimplementing the algorithms: langchain-text-splitters for fixed-size, recursive, and Markdown-header splitting; Chonkie — a purpose-built, lightweight chunking library (pip install chonkie) — for semantic, late, and LLM-guided (SlumberChunker) chunking; LlamaIndex's HierarchicalNodeParser and SentenceWindowNodeParser for small-to-big and sentence-window; and, for proposition chunking, the actual fine-tuned model the Dense X Retrieval authors released, rather than an improvised prompt. Only the strategy-selection logic in adaptive chunking is inherently custom — no library owns "run several chunkers and pick the best," since that's an orchestration choice, not a chunking algorithm.

Scenario Check

Ten chunking strategies is a lot to keep straight — these questions test whether you can match a symptom to its fix, the way a system-design interview actually probes this.

Sources & further reading