Chunking Strategies for RAG

You feed a 50-page contract into a RAG system and ask it, "What is the termination clause?" The retriever hands back the cover page. Why? Because the chunker cut the document into 512-token pieces, and the termination clause sits 20 pages in, split across a page break, with no local keyword tying it to the query.

The fix isn't a better embedding model. It's chunking: how big, how much overlap, where to split, how much surrounding context to carry along.

Feb 2026 benchmarks turn up some genuine surprises:

• Vectara's 2026 study: recursive 512-token chunking beat semantic chunking, 69% to 54% accuracy.
• SPLADE + Mistral-8B on Natural Questions: overlap provided zero measurable benefit.
• Context cliff: response quality drops sharply around 2,500 tokens of context.

The "obvious" answer, semantic chunking with 20% overlap and 1000-token chunks, is often wrong. This lesson builds intuition for seven strategies and tells you when to reach for each one.

Chunking is the step that splits a document into smaller pieces, chunks, before you embed and index them for retrieval. Get the split wrong and nothing downstream, not a bigger embedding model, not a reranker, recovers the context that got cut off.

Fixed chunking. Split every N characters or tokens. It's the simplest baseline you can build, and it shows: chunks break mid-sentence with no regard for meaning. Good compression, bad coherence.

Recursive. LangChain's RecursiveCharacterTextSplitter tries splitting on \n\n first, then \n, then ., then space, falling back cleanly whenever a level doesn't produce a clean cut. It's the 2026 default for good reason.

Semantic. Embed each sentence, compute the cosine similarity between adjacent sentences, and split where that similarity drops below a threshold. It preserves topic coherence better than any fixed rule can, but it's slower, and without a floor it can produce tiny 40-token fragments that hurt retrieval more than they help.

Sentence. Split on sentence boundaries, either one sentence per chunk or a window of N sentences. Up to roughly 5k tokens, it matches semantic chunking's results at a fraction of the compute cost.

Parent-document. Store small child chunks for retrieval and the larger parent chunk for context, then retrieve by child and return the parent. It degrades gracefully: even a mediocre child chunk still points to a reasonable parent.

Late chunking (2024). Embed the whole document at the token level first, then pool those token embeddings into chunk embeddings afterward. Because the embedding step sees the whole document, cross-chunk context survives the split. It needs a long-context embedder (BGE-M3, Jina v3) and more compute than the alternatives.

Contextual retrieval (Anthropic, 2024). Prepend each chunk with an LLM-generated summary of where it sits in the document ("This chunk is section 3.2 of the termination clauses..."). Anthropic's own benchmark found a 35-50% retrieval improvement from this alone, though it's expensive to index: every chunk costs an extra LLM call before it ever reaches the vector store.

The rule that beats every default

Match chunk size to query type:

That's NVIDIA's 2026 benchmark speaking, and the rule underneath it is simple: a chunk needs to be big enough to hold the answer plus its local context, and small enough that the retriever's top-K results stay focused on the answer instead of drowning it in surrounding noise.

Step 1: fixed and recursive chunking

def chunk_fixed(text, size=512, overlap=0):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]
 
 
def chunk_recursive(text, size=512, seps=("\n\n", "\n", ". ", " ")):
    if len(text) <= size:
        return [text]
    for sep in seps:
        if sep not in text:
            continue
        parts = text.split(sep)
        chunks = []
        buf = ""
        for p in parts:
            if len(p) > size:
                if buf:
                    chunks.append(buf)
                    buf = ""
                chunks.extend(chunk_recursive(p, size=size, seps=seps[1:] or (" ",)))
                continue
            candidate = buf + sep + p if buf else p
            if len(candidate) <= size:
                buf = candidate
            else:
                if buf:
                    chunks.append(buf)
                buf = p
        if buf:
            chunks.append(buf)
        return [c for c in chunks if c.strip()]
    return chunk_fixed(text, size)

Step 2: semantic chunking

def chunk_semantic(text, encoder, threshold=0.6, min_chars=200, max_chars=2048):
    sentences = split_sentences(text)
    if not sentences:
        return []
    embs = encoder.encode(sentences, normalize_embeddings=True)
    chunks = [[sentences[0]]]
    for i in range(1, len(sentences)):
        sim = float(embs[i] @ embs[i - 1])
        current_len = sum(len(s) for s in chunks[-1])
        if sim < threshold and current_len >= min_chars:
            chunks.append([sentences[i]])
        else:
            chunks[-1].append(sentences[i])
 
    result = []
    for group in chunks:
        text_group = " ".join(group)
        if len(text_group) > max_chars:
            result.extend(chunk_recursive(text_group, size=max_chars))
        else:
            result.append(text_group)
    return result

Tune threshold per domain: set it too high and you get fragments, too low and everything collapses into one giant chunk.

Step 3: parent-document

def chunk_parent_child(text, parent_size=2048, child_size=256):
    parents = chunk_recursive(text, size=parent_size)
    mapping = []
    for p_idx, parent in enumerate(parents):
        children = chunk_recursive(parent, size=child_size)
        for child in children:
            mapping.append({"child": child, "parent_idx": p_idx, "parent": parent})
    return mapping
 
 
def retrieve_parent(child_query, mapping, encoder, top_k=3):
    child_embs = encoder.encode([m["child"] for m in mapping], normalize_embeddings=True)
    q_emb = encoder.encode([child_query], normalize_embeddings=True)[0]
    scores = child_embs @ q_emb
    top = np.argsort(-scores)[:top_k]
    seen, parents = set(), []
    for i in top:
        if mapping[i]["parent_idx"] not in seen:
            parents.append(mapping[i]["parent"])
            seen.add(mapping[i]["parent_idx"])
    return parents

Key insight: dedupe parents. Multiple children often map to the same parent, and returning all of them wastes context for no benefit.

Step 4: contextual retrieval (Anthropic pattern)

def contextualize_chunks(document, chunks, llm):
    context_prompts = [
        f"""<document>{document}</document>
Here is the chunk to situate: <chunk>{c}</chunk>
Write 50-100 words placing this chunk in the document's context."""
        for c in chunks
    ]
    contexts = llm.batch(context_prompts)
    return [f"{ctx}\n\n{c}" for ctx, c in zip(contexts, chunks)]

Index the contextualized chunks, and retrieval benefits from that extra surrounding signal at query time.

Step 5: evaluate

def recall_at_k(queries, corpus_chunks, encoder, k=5):
    chunk_embs = encoder.encode(corpus_chunks, normalize_embeddings=True)
    hits = 0
    for q_text, gold_idxs in queries:
        q_emb = encoder.encode([q_text], normalize_embeddings=True)[0]
        top = np.argsort(-(chunk_embs @ q_emb))[:k]
        if any(i in gold_idxs for i in top):
            hits += 1
    return hits / len(queries)

Always benchmark it yourself. The "best" strategy for your corpus won't necessarily match any blog post.

• Chunking evaluated only on factoid queries. Multi-hop queries reward a completely different strategy. Stratify your eval set by query type, or you'll crown the wrong winner.
• Semantic chunking without a minimum size. Left unchecked, it produces 40-token fragments that hurt retrieval more than they help. Always enforce min_tokens.
• Overlap as cargo cult. 2026 studies find overlap often provides zero benefit while still inflating the index (20% overlap means about 25% more chunks to embed and store). Measure it, don't assume it.
• No min/max enforcement. Chunks of 5 tokens and chunks of 5,000 tokens both break retrieval. Clamp both ends.
• Cross-doc chunking. Never let a chunk span two documents. Chunk per document, then merge.

The 2026 default stack:

Start with recursive chunking at 512 tokens, measure recall@5 on a 50-query eval set, and tune from there.

Save as outputs/skill-chunker.md:

---
name: chunker
description: Pick a chunking strategy, size, and overlap for a given corpus and query distribution.
version: 1.0.0
phase: 5
lesson: 23
tags: [nlp, rag, chunking]
---
 
Given a corpus (document types, avg length, domain) and query distribution (factoid / analytical / multi-hop), output:
 
1. Strategy. Recursive / sentence / semantic / parent-document / late / contextual. Reason.
2. Chunk size. Token count. Reason tied to query type.
3. Overlap. Default 0; justify if >0.
4. Min/max enforcement. `min_tokens`, `max_tokens` guards.
5. Evaluation plan. Recall@5 on 50-query stratified eval set (factoid, analytical, multi-hop).
 
Refuse any chunking strategy without min/max chunk size enforcement. Refuse overlap above 20% without an ablation showing it helps. Flag semantic chunking recommendations without a min-token floor.

1. Easy. Chunk one 20-page document with fixed(512, 0), recursive(512, 0), and recursive(512, 100). Compare chunk counts and boundary quality.
2. Medium. Build a 30-query eval set over 5 documents. Measure recall@5 for recursive, semantic, and parent-document. Which wins? Does it match the blog posts?
3. Hard. Implement contextual retrieval. Measure MRR improvement over baseline recursive. Report index cost (LLM calls) vs accuracy gain.

• Yepes et al. / LangChain: Recursive Character Splitting docs: the default in production.
• Vectara (2024, NAACL 2025). Chunking configurations analysis: the earlier peer-reviewed study (chunking matters as much as embedding choice). The 69% vs 54% figure in The Problem comes from a separate Feb 2026 benchmark, summarized in the Premai guide below, not from this paper.
• Jina AI: Late Chunking in Long-Context Embedding Models (2024): the late chunking paper.
• Anthropic: Contextual Retrieval: 35-50% retrieval improvement with LLM-generated context prefixes.
• NVIDIA 2026 chunk-size benchmark: Premai summary: chunk size by query type.