Lost in the middle: where context windows go blind

The interactive below puts a single needle fact inside a simulated token ribbon and lets you run seeded recall trials, sweep the needle through every position, and toggle mitigation strategies. Move the needle to the middle, run 100 trials, then compare that recall rate against the same needle placed at the start or end.

What you're seeing

The main canvas shows a horizontal token ribbon representing the full context window. One bright red cell is the needle fact, every darker cell is a distractor. Above the ribbon a heatmap accumulates recall rates by position across trials, and a U-shaped trend curve builds from those results. The right panel shows the current simulated model answer, whether it matched the needle, and a confidence bar.

The HUD tracks context size in tokens, needle depth as a percentage of context, trial count, recall rate at the current position, distractor count, and the position penalty versus the best edge performance. Every recall value is labeled SIMULATED, and the caveat panel notes that the pattern is conceptual, varies by model and eval, and does not mean middle tokens are invisible to the model.

A few things to try:

• Place the needle at 50% depth and run 100 trials. Then move to start and run another 100. The heatmap and penalty stat make the edge advantage visible immediately.
• Hit Play on the position sweep. The needle moves through every position preset automatically and the trend curve fills in, showing the full U shape in one pass.
• Scale the context length from 8K to 128K with the same needle position. Watch the middle penalty grow as the context lengthens, because the needle is now further from both edges in relative terms.
• Raise distractor density and switch query specificity from Exact to Multi-hop. Each change deepens the dip independently, and together they can push mid-position recall near zero.
• Toggle the strategy chips: Move evidence, Repeat summary, and Retrieve top chunks. Each flattens the U by a different route, and the panel notes the tradeoff that comes with each one.

How position-dependent recall actually works

When an LLM processes a long input, every token can theoretically attend to every other token. In practice, empirical needle-in-a-haystack evaluations have shown that models tend to weight tokens near the start and end of the context more than tokens buried in the middle, a pattern first described and named in research by Liu et al. (2023). The exact cause is debated, but training data, positional encoding schemes, and the way attention interacts with very long sequences all likely contribute.

The lab models this as a Bernoulli recall probability that peaks near position 0% and 100% and dips in between, scaled by context length, distractor count, and query specificity. This is a teaching abstraction, not a measurement of any real model, and the sim labels it as such. What the abstraction captures correctly is the qualitative shape and the directional effect of each variable: longer contexts tend to worsen the dip, more distractors tend to worsen it, and vaguer queries tend to worsen it.

One important nuance: a middle miss does not usually look like an explicit failure. In the sim you will notice that failed trials often return a plausible-sounding distractor value rather than a blank or an "I don't know." This matches how real position failures manifest in LLM outputs, which makes them easy to misread as hallucinations rather than positioning problems.

What actually flattens the U

The three mitigations in the lab each address the problem from a different angle.

Move evidence up front places the needle fact near the beginning of the context before querying. This works well when you know in advance which fact the model will need, but it requires knowing that in advance. If you are filling a context with many documents and do not know which one the query will hit, you cannot reliably move the right one to the front.

Repeat summary appends a condensed version of the key evidence at the end of the context. The model now sees the fact near a strong position (the end) as well as wherever it originally sat. This is a second chance rather than a fix: the original middle copy is still less likely to be recalled, and the summary copy may introduce slight paraphrasing that interacts with query specificity.

Retrieve top chunks trades the positional problem for a retriever-accuracy problem. Instead of putting all documents in a single long context, a retriever (like the RAG pattern described in retrieval-augmented generation) selects the most relevant chunks and places them near the top of a shorter context. If the retriever finds the right chunk, the model sees the fact at a strong position in a shorter window. If the retriever misses, the mitigation fails entirely, a different failure mode than a soft positional dip.

Newer models have narrowed the dip significantly compared to early evaluations, and the magnitude of the effect varies by model, task, and eval setup. For a broader view of how the model processes the context it does attend to, see the context window explainer.

Frequently asked questions

Key takeaways

• Position is a real lever, not a formatting detail. The same fact with the same query can succeed near the edges of the context and fail in the middle, so where evidence sits in the prompt changes whether the model uses it.
• A middle miss rarely announces itself. In the lab, most failed trials confidently return a nearby distractor fact instead of saying not found, which is why position problems often get misdiagnosed as hallucinations.
• Everything that makes the task harder deepens the dip. Longer contexts, more distractor facts, and vaguer queries each grow the middle penalty, so a fact that survives mid-position at 8K can start failing at 128K.
• Mitigations route around the curve instead of fixing it. Moving evidence to the top only works when you already know which fact matters, a repeated summary is a second chance rather than a guarantee, and retrieval trades the position problem for a retriever-accuracy problem.
• The U shape is a tendency, not a law. It varies with model, task, length, and eval setup, newer models narrow the dip, and a middle miss does not mean the model cannot see middle tokens.