Context window: what an LLM sees, and what it forgets

The interactive below slides two context windows through the same document at the same speed. The top lane is a 2,000-token window, the bottom is 32,000. Press play, try the preset documents, or paste your own text and watch which tokens scroll off the back of each window as the playhead advances.

New tab

Demo

Context window: 2k vs 32k, sliding through your text

Two LLM context windows, 2k and 32k tokens, slide through the same document side by side so you can see exactly which tokens fall off the back of the smaller one.

Built with Canvas 2D and vanilla JS. GPT-4 (cl100k_base) pre-tokenization, bundled inline, no model run at runtime. Render and interaction by Pro Trailblazer.

What does the demo show?

Two stacked lanes scroll the same text from right to left. The top lane is the of a 2,000-token model; the bottom lane is a 32,000-token one. Each small block is a , not a word. As the playhead advances, new tokens enter on the right and old ones slide toward the left edge, where the smaller window starts dropping them.

The strip across the top is the whole document, with two highlighted bands showing how much of it each window currently covers. The smaller band makes the gap obvious: on a long input, the 2k window only ever sees a thin slice. The strip along the bottom plots tokens lost over time for both windows.

The readouts track POS (the playhead's position in tokens), 2K_LOST and 32K_LOST (how many tokens have fallen off the back of each window so far), and FLUX (tokens per second at the current speed).

A few things to try in the demo:

Load the long contract preset and press play. The 2K_LOST counter starts climbing within seconds while 32K_LOST stays at zero. That gap is the whole point.
Paste your own text. A long email thread or a book chapter makes the difference visceral. Watch the token count as you paste: it climbs faster than the word count does.
Drag the scrubber back and forth. Scrubbing pauses autoplay and lets you line up the exact moment a specific sentence drops out of the 2k window.
Bump the speed to 4x. On a long document the small window churns through its entire capacity again and again while the large one barely fills.

How does a context window actually work?

A context window is the maximum number of tokens a model can attend to in a single forward pass. It is a hard architectural limit, fixed when the model is trained, not a preference you can talk it past. Everything inside the window is fair game for the next prediction. Everything that has scrolled past the edge is gone, and the model has no memory of it at all.

The window is measured in tokens, not words or characters. This demo uses the GPT-4 cl100k_base pre-tokenizer, which splits text into subword pieces and punctuation. As a rough rule, 1,000 tokens is about 750 English words, so a 2,000-token window holds maybe three or four pages. Code, non-English text, and unusual formatting tokenize less efficiently and fill the window faster.

When a conversation or document runs longer than the window, the runtime slides the window forward, dropping the oldest tokens to make room for new ones. That is what the lanes show. In a chat app it is why a long session eventually 'forgets' how it started: those early tokens were evicted to fit the latest turn.

A larger window helps, but it is not perfect memory either. Models tend to attend most strongly to the beginning and end of the window and least to the middle, an effect often called 'lost in the middle.' A fact sitting in the center of a nearly full 32k window can be present but effectively ignored.

When the window is the bottleneck, the fix is rarely to prompt harder. It is to put the right tokens back in front of the model. Retrieval-augmented generation does this by fetching only the most relevant chunks at query time, and a running summary can compress old turns into a few tokens. The window itself is held in GPU memory as the KV cache, which is why bigger windows cost more to serve.

Frequently asked questions

What is a context window in an LLM?

A context window is the fixed maximum number of tokens a model can read at once, counting your prompt, any system instructions, the retrieved or pasted text, and the response it is generating. Anything beyond that limit cannot influence the output. It is set by the model's architecture, so a 32k model simply cannot consider a 40k-token input without dropping part of it.

How many words fit in a context window?

Roughly 750 English words per 1,000 tokens, so a 2,000-token window holds about 1,500 words and a 32,000-token window about 24,000. The exact ratio depends on the text: code, punctuation-heavy content, and non-English languages use more tokens per word, so they fill the window faster.

What happens when text exceeds the context window?

The runtime slides the window forward and drops the oldest tokens to make room, so the model loses access to whatever scrolled off the back. It does not summarize or compress on its own; those tokens are simply gone from its view. This is the usual reason a long chat session seems to forget how it began.

Does a bigger context window always give better answers?

Not necessarily. Beyond fitting more text, models often recall information at the start and end of the window better than the middle, an effect called 'lost in the middle.' A focused, well-retrieved 2k of context can beat a padded 32k window, which is why retrieval and good prompt ordering still matter even on long-context models.

Key takeaways

A context window is a hard limit, not a soft preference. Once a token slides past the edge the model cannot reach it, so there is no fading memory, just an abrupt cutoff.
Token count is not word count. The cl100k_base tokenizer splits on subwords and punctuation, so 2,000 tokens is roughly 1,500 English words, and code or non-English text fills the window faster.
A bigger window does not mean even recall across all of it. Models tend to attend most to the start and end of the window, so a fact buried in the middle of a full 32k window can be present but effectively ignored.
Most 'the model forgot what I said' moments are context-window evictions, not reasoning failures. The fix is usually to put the dropped tokens back in front of the model with a summary or retrieval, not to re-prompt harder.