RAG chunking strategies: where the answer gets cut

What you're seeing

The strip is the whole document with the words replaced by cells, so you can see its shape at once: eight sections, each with an outlined heading row. The alternating red bands are chunks, the bright vertical lines are the boundaries between them, and hatched cells belong to two chunks at once because of overlap. The one white ring is the gold answer for the current question. Retrieved chunks glow and carry a rank badge. When a boundary passes through the ring, it pulses.

It loads on question 1 with the SMALL preset: fixed 40-word chunks, no overlap, top-2 retrieval. The answer to "how long does the stove take to boil a liter of water" is one sentence, 23 words long, and the boundary between chunk 1 and chunk 2 lands after "brings one liter of water to a". Both halves are retrieved, so the panel on the right shows the model being handed a sentence that stops mid-thought in one chunk and "rolling boil in about four minutes" with no subject in the other. The verdict is PARTIAL. At this setting, 6 of the 8 answers have no single chunk that holds them, and only 2 come back whole.

The four stocks across the top are the pipeline: DOCUMENT (914 words) flows into CHUNKS, the retriever picks k of them, and CONTEXT is the number of words a model would receive for this question. The scatter plot at the bottom is the experiment trail. Every configuration you try adds one dot at its average words sent per question (x) and the number of answers fully retrieved out of 8 (y). The curve that appears is the cost-versus-recall tradeoff for this document.

A few things to try:

• Press SMALL + OVERLAP. The hatched cells appear, question 8 flips to HIT, and the cost stays at 80 words. Questions 1, 2, and 3 stay PARTIAL, because overlap only rescues answers shorter than the overlap.
• Press SENTENCE 60. The cuts move to sentence ends and recall jumps to 6 of 8 for about 99 words per question.
• Press STRUCTURE 120. Every section becomes one chunk with its heading on top. All 8 answers come back whole for about 229 words per question.
• Press HUGE. Three 300-word chunks plus a tail. Cost climbs to 600 words per question, and recall is still only 6 of 8: question 7 is cut at word 600, and question 4 loses the ranking to two chunks that mention "stove" and "fuel" more often.
• Drag the size slider slowly from 20 to 300 with FIXED selected. Recall jumps around rather than climbing smoothly, because each size moves every boundary at once.
• Hover or tap any cell to see which chunk it fell into and which section it came from.

How it actually works

Nothing in the demo is scripted. The handbook is a fixed string, each question stores its gold answer as a pair of character offsets into that string, and every number on screen falls out of three chunkers, one ranking function, and one comparison of offsets.

The three chunkers

• Fixed starts a new chunk every N words. With overlap, the last M words of each chunk are repeated at the start of the next, so the step between chunk starts is N minus M. It knows nothing about sentences, paragraphs, or headings, which is why its cuts land anywhere.
• Sentence adds whole sentences to a chunk until the next sentence would push it past N words, then closes the chunk. Overlap carries whole trailing sentences into the next chunk, up to M words. A single sentence longer than N becomes its own oversized chunk, because cutting inside a sentence is the thing this strategy refuses to do.
• Structure splits at headings first. If a section fits in N words it becomes one chunk. If not, it splits at paragraphs, and only then at sentences. Every chunk cut from a section keeps the heading as its first line, and that repeat counts toward both N and the context cost.

Three sanity checks run on load, as console.assert calls you can see in the page source: every word belongs to at least one chunk for all three chunkers, and to exactly one chunk when overlap is zero.

Retrieval

Retrieval is BM25 with k1 = 1.2 and b = 0.75 over whatever chunks currently exist, after lowercasing and a short stopword list. Each chunk is scored by how often it contains each query word, how rare that word is across chunks, and how long the chunk is relative to the average. The top k chunks with a nonzero score are "retrieved". A chunk that shares no word with the question is never retrieved, which is how real keyword indexes behave too. A second, plain implementation of the formula runs on load and the two are asserted equal to nine decimal places.

One word is one token in the demo, and the UI says so. Real tokenizers would make every count a little larger, but the shape of every result stays the same.

Verdicts

Each question's verdict is computed from offsets only. HIT means one retrieved chunk contains the whole gold span. PARTIAL means retrieved chunks contain part of the span and no single chunk contains all of it. MISS means no retrieved chunk touches the span at all. The SPLIT stat counts gold spans that no chunk in the index holds in full, retrieved or not, which is the damage the chunker did before the retriever ever ran.

The sim stops there on purpose. No model writes an answer, because the point is what the model is handed. If the handed text stops at "brings one liter of water to a", the best model in the world cannot answer the question from it.

What is chunk overlap?

Chunk overlap is the number of words or tokens that the end of one chunk shares with the start of the next. With 40-word chunks and 10 words of overlap, chunk 2 starts 30 words into the document and repeats the last 10 words of chunk 1. Each word in the overlap region now lives in two chunks, so a fact that straddles the old boundary has a chunk that holds it whole.

The demo shows both what overlap buys and what it does not. SMALL + OVERLAP repeats 10 words per boundary. Question 8, a 16-word answer that SMALL cut, becomes a HIT. Question 1, a 23-word answer, stays cut because 23 is more than the 10-word repair window, and the three-step procedure in question 2 (58 words) can never fit inside a 40-word chunk no matter how much overlap you add. Overlap also costs index size: SMALL has 23 chunks, SMALL + OVERLAP has 31, and every retrieved chunk is still 40 words, so the per-question context does not grow, but the index does.

Common defaults are modest. As of November 4, 2026, LangChain's base TextSplitter defaults to 4,000 characters per chunk with 200 characters of overlap, and LlamaIndex's SentenceSplitter defaults to 1,024 tokens with 200 tokens of overlap. Both are starting points, not recommendations, and both measure in different units than this demo.

Fixed vs semantic chunking

Fixed-size chunking cuts by count and nothing else. It is fast, predictable, and easy to budget, and it is the strategy that produces the cut through question 1. Every other strategy exists to move the cuts to places where the text was already divided by its author.

Sentence chunking moves cuts to sentence boundaries. Recursive or structure-aware chunking tries the biggest natural separators first (headings, then blank lines, then sentences) and only falls back to smaller ones when a piece is still too big; LangChain's RecursiveCharacterTextSplitter is the common implementation. The demo's Structure chunker is the same idea with one addition: it writes the section heading into every chunk it makes, so a chunk that says "If the control valve sticks" also says "Troubleshooting". Anthropic's Contextual Retrieval post (September 19, 2024) takes that further by having a model write 50 to 100 tokens of document-specific context onto each chunk, and reports that this cut the top-20 retrieval failure rate from 5.7% to 3.7% for embedding search, and to 2.9% when a contextual BM25 index was fused in.

Semantic chunking is the option the demo does not simulate. It embeds each sentence, walks the document, and places a cut wherever the similarity between neighboring sentences drops below a threshold, so topic shifts become boundaries even when there is no heading. It is the most expensive strategy, because it calls an embedding model once per sentence before the index exists, and it is sensitive to the threshold. Chroma's July 2024 technical report Evaluating Chunking Strategies for Retrieval found that chunking choice moved recall by up to 9 percentage points across strategies, that a well-parameterized recursive splitter performs well in practice while its default separators produce very short chunks that scored poorly, and that their own cluster-based semantic chunker was consistently strong.

Note what changes and what does not. Switching from BM25 to embeddings changes the scores, so a chunk can win on meaning instead of shared words. It does not change the boundaries. A sentence cut in half embeds as half a sentence. The semantic versus keyword search demo shows the scoring side of that story on chunks that already exist.

What is the best chunk size for RAG?

There is no size that is best everywhere, and anyone quoting one without a benchmark is guessing. The trail in the demo is the honest answer: recall and cost both climb with chunk size, but not smoothly and not together. On this document, 120-word structure chunks reach 8 of 8 for 229 words per question, while 300-word fixed chunks cost 600 words per question and still miss two.

What you can say in general:

• A chunk has to be at least as long as the answers you expect. If your users ask for procedures, a chunk that holds one sentence will never return a procedure whole.
• Smaller chunks rank more precisely, because a chunk about one thing scores high for that one thing. Larger chunks rank more vaguely and send more words per hit, which raises cost and feeds the lost in the middle effect once the context gets long.
• The budget is the context window minus the prompt minus the room for the answer, divided by top-k. Work backward from that, and read the context window explainer if the units are unfamiliar.
• Structure beats size. A 120-word section with its heading usually retrieves better than a 120-word window that starts mid-paragraph, at the same cost.

Measure instead of guessing. Build a small set of questions with known answer spans, as this demo does with eight, run every configuration you are considering, and plot context against recall. The right chunk size is the knee of your own curve.

Why does my RAG miss answers?

When a retrieval-augmented generation system returns a confident wrong answer or "I don't know" for a question the documents clearly cover, the failure usually sits in one of four places, and the demo reproduces three of them.

1. The answer was cut. A boundary split it and only one half was retrieved. This is question 1 at SMALL. Fix: larger or structure-aware chunks, or overlap for short answers.
2. The right chunk exists but lost the ranking. The chunk that holds the whole answer scored lower than chunks that merely repeat the query's words. This is question 4 at HUGE, where two chunks that say "stove" and "fuel" more often outrank the one that says what fuel the stove uses. Fix: smaller chunks, a hybrid index, or a reranker on a bigger candidate set.
3. No chunk shares a word with the question. The user says "light" and the document says "ignite". BM25 scores everything 0, and the demo shows MISS with a note. Fix: embedding search, query rewriting, or synonyms in the index.
4. The chunk was retrieved and the model still missed it. Too many chunks, the answer buried in the middle of a long context, or a model that paraphrased wrong. The demo stops before this step, which is the point: fix retrieval first, because nothing downstream can recover text that was never sent.

The full pipeline, from embedding to answer, is in the retrieval-augmented generation explainer. This post is the step before it.

How to chunk documents for RAG

1. Keep the document's structure when you parse it. Headings, paragraphs, lists, and tables are free boundary hints, and most loaders throw them away.
2. Split at the largest natural boundary that fits your size limit, and only fall back to sentences when a section is too big. Never split inside a sentence unless the sentence is longer than the chunk.
3. Prepend context to each chunk: the document title, the section heading, or a generated summary. The Structure chunker in the demo does the cheapest version of this.
4. Add a little overlap for short facts that sit on boundaries, and accept that overlap cannot help answers longer than a chunk.
5. Build an evaluation set of questions with gold spans before you tune anything, and plot context cost against recall the way the trail does.
6. Revisit the choice when your documents change. A chunker tuned for API docs will cut a legal contract in the wrong places.

What the demo simplifies

• One word is one token. Real tokenizers split long words and punctuation, so every count would be somewhat larger.
• Retrieval is BM25 only. An embedding index or a hybrid one would change which chunks win, not where the boundaries fall. Vector indexes themselves are covered in the HNSW demo.
• The document is 914 words with eight clean headings. Real corpora have tables, code, footers, and pages that were scanned sideways, and every one of those is a chunking hazard the demo does not show.
• Gold spans are exact. In practice, judging whether a retrieved chunk "contains the answer" is itself a hard problem.
• Semantic chunking is described, not simulated. It needs an embedding model in the loop, and the demo runs with no model at all.

Frequently asked questions

Key takeaways

• Retrieval returns chunks, not answers. If the answer straddles a boundary, the model is handed a sentence that stops mid-thought, and nothing downstream can recover the missing half.
• Chunk size is a cost knob as much as a quality knob. In the demo, 300-word chunks send 7.5 times the words of 40-word chunks per question and still miss more answers than 120-word structure chunks.
• Overlap repairs short answers that sit on a boundary and does nothing for answers longer than a chunk. A 58-word procedure never fits inside a 40-word chunk, overlap or not.
• Splitting at headings and paragraphs keeps text that was written together in the same chunk, and repeating the heading inside each chunk gives the ranker the section's subject for free.
• Big chunks blur ranking as well as costing more. A 300-word chunk about everything can lose to a 300-word chunk that repeats the query's words, which is why HUGE misses a question that STRUCTURE 120 gets.
• No chunk size is best everywhere. Build eight questions with gold spans, try your configurations, and read the knee of your own curve.