Chunking Strategies, Compared

Chunking is cutting a source document into pieces small enough for an embedding model to take and large enough that each piece carries a self-contained idea. Every RAG pipeline starts there, and where you cut is not a hyperparameter. It is the upper bound on what the retriever can ever return.

Take a query like "what does the budget abort threshold look like". It can only succeed if the chunk holding the abort threshold is reachable. Suppose a fixed-window splitter separated the threshold value from its surrounding context. The embedding lands in a different cluster, the BM25 score drops, the rerankers see noise, and the LLM generates a wrong answer. The 2024 paper "LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs" measured a 35 percent absolute swing in retrieval recall purely from the chunking choice. Follow-up work in 2025 on contextual chunk headers narrowed the gap but did not close it.

Here you build five strategies side by side, run them against a fixture corpus with gold-labeled answer spans, and read the recall numbers yourself.

Fixed-window

Fixed-window chunking cuts every N characters. It is the brute-force baseline. You can add overlap, so a sentence cut at position N appears whole inside the chunk that starts at position N - overlap. It is fast and deterministic, and it is terrible at boundaries. I'd use it as a control, not a default.

Sentence

Sentence chunking splits on sentence boundaries, using a regex or a simple state machine, and packs one or more sentences into a chunk up to a target character budget. That ends mid-word cuts, but it still cuts mid-paragraph and mid-section. Many early RAG pipelines defaulted to it, and it is a reasonable choice for prose with no other structure.

Recursive split

Recursive split is a hierarchy strategy, popularized by 2023-era libraries. It tries the strongest separator first (double newline, paragraph), falls back to the next (single newline), then to sentences, then to spaces (word boundaries), and stops recursing once a chunk fits the budget. Because it adapts per region, it does well on documents with inconsistent structure.

Semantic clustering

Semantic clustering embeds every sentence and groups contiguous sentences that share a topic centroid. A cut happens when the running similarity to the centroid drops below a threshold, so the boundaries follow meaning rather than character counts. A hard character cap (max_chars, 900 in the code) also forces a cut when a chunk grows too long. It is slower to build and depends on the embedding model, but it holds up on documents that switch topics inside a paragraph.

Structural markdown headers

Structural chunking applies to documents with explicit structure (markdown, reStructuredText, RFC-style numbered sections) and cuts at heading boundaries. Each chunk is the heading plus everything underneath it, down to the next heading of any level. You get the smallest chunks per topic, but only when the corpus is well-formed.

How recall@k measures the boundary choice

Recall@k is the fraction of queries where any of the top-k retrieved chunks overlaps the gold answer span. A gold-labeled query carries the exact character offsets of that span inside the source document. After chunking, you check whether any of the top-k chunks the retriever returned overlaps it. If one does, recall@k for that query is 1. If none does, it is 0. Average across the query set, run the same evaluation for each strategy, and the spread shows which boundary policy survives the corpus you have.

code/main.py implements:

• fixed_window(doc_id, text, size, overlap) - the baseline.
• sentence_chunks(doc_id, text, target) - simple sentence packer.
• recursive_split(doc_id, text, target, separators) - hierarchical recursion.
• semantic_chunks(doc_id, text, similarity_threshold, max_chars) - centroid-based clustering on top of a deterministic mock embedding.
• structural_markdown(doc_id, text) - header-aware splitter.
• mock_embed(text, dim) - a hash-based embedding so the loop runs offline.
• DenseIndex - the same shape used in Phase 19 Track B's hybrid retrieval lesson.
• eval_recall(chunker, fixture, ks) - the comparison loop.
• A main() that runs every strategy on the fixture corpus and prints a recall@k table.

Run it:

python3 code/main.py

The output is a small table with one row per strategy and one column per k, followed by chunk counts. On the fixture (three short documents, nine queries), every strategy reaches 1.00 at recall@3 and recall@5, so the separation shows up at recall@1. Recursive and structural score 1.00, fixed-window 0.89, sentence 0.78, and semantic is lowest at 0.67. Semantic also emits the most chunks (20, against 6 to 9 for the others), because the hash-based mock embedding makes neighboring sentences look dissimilar and triggers many cuts. A corpus this small cannot crown a winner, so read it as a demonstration of the harness, then run your own corpus through it.

Orphan sentences. Sentence packing can produce chunks that miss the topic sentence, and the embedding then points at the wrong cluster.

Mid-symbol cuts. Fixed-window inside code or YAML splits an identifier in half, and the two halves embed to noise.

Header-only chunks. Structural markdown can emit a chunk containing nothing but ## Title. Filter those out, or attach the next chunk's first paragraph.

Semantic drift. Semantic clustering under-cuts when the corpus is uniformly on topic. A 5000-character chunk packs many specific answers into one diffuse embedding. Pair semantic with a hard character cap (the code defaults to max_chars=900).

Stale embeddings. Semantic clustering uses an embedding model, so changing the model changes the chunks. Pin the chunk model separately from the retrieval model, or rebuild the index together.

Two properties decide the default chunker for a new corpus: the document type and the paragraph length.

When in doubt, pick recursive split. It is the strongest single-strategy baseline.

Production patterns:

• Run the eval before you ship a new pipeline, and don't trust the strategy your library defaults to.
• Re-run the eval whenever the embedding model or the corpus mix changes, since the winner is corpus-dependent.
• Persist the strategy name in each chunk's metadata so you can attribute regressions later.

The Track F end-to-end RAG system in lesson 69 uses the chunker you select here as its first stage. The eval harness in lesson 68 reads recall@k from the same shape that eval_recall returns here. Pick the strategy that wins on your corpus and feed it forward.

1. Add a sixth strategy: token-window using tiktoken instead of character counts. Compare it against fixed-window on the same fixture.
2. Inject a 30 percent fraction of code blocks into the prose fixture and re-run the table. Explain why every strategy except structural markdown loses recall.
3. Replace the deterministic embedding with the one from your project's real provider. Measure the semantic-clustering recall delta, and report whether the spread between strategies widens or narrows.
4. Add a summary field per chunk: a one-sentence centroid description. Re-run the eval with the summary appended to the chunk body, and measure the recall lift.

• LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs (arXiv 2406.15319)
• Anthropic, Contextual Retrieval
• LlamaIndex, Chunking strategies for production RAG
• Phase 11 lesson 06: RAG fundamentals
• Phase 11 lesson 07: advanced RAG
• Phase 19 lesson 65: hybrid retrieval that ranks the chunks produced here
• Phase 19 lesson 68: the eval harness that scores the strategy choice in production