Self-attention: how a token decides which tokens to read
Last updated: July 18, 2026
The interactive below runs real scaled dot-product self-attention over a sentence you can edit. Tap any word to make it the query, and watch weighted arcs fan out to every other word, thick where the attention is strong and faint where it is weak. Add words from the bank, switch between one head and two, and open the panels to see the raw numbers behind each weight.
Demo
Self-attention across a sentence
Tap any word to make it the query and watch scaled dot-product attention score, scale, and softmax a weight for every other word. Toggle a second head to see position tracked separately from meaning.
What you're seeing
The sentence starts as "the cat sat on the warm mat," drawn as a row of tappable chips. Each chip is one . Tap one to make it the query, the word currently doing the looking. Downward arcs stretch from the query to every other token, and the thickness and brightness of each arc is that token's attention weight. A pulse of light travels the strongest arc, so the current focus is always visible.
The readout at the top tracks four numbers: which token is the query, which token it attends to most strongly, the size of that peak weight, and the entropy of the whole distribution in bits. Low entropy means the query is locked onto one or two words. High entropy means its attention is spread thin across many.
You are not limited to the default sentence. Type your own, or tap from the bank of 48 words that carry hand-built meaning vectors. Words outside that bank still work, but they show up as dashed chips with a question mark, because their meaning vector is a neutral placeholder. Their arcs are still drawn, they just are not meaningful.
A few things to try:
- Tap different words as the query. Make "mat" the query, then "cat," and watch which words each one leans on.
- Edit the sentence. Type your own or pull words from the bank, then re-tap a query to see the weights recompute.
- Turn on the second head. Switch from one head to two and compare what each one attends to.
- Open The numbers. Expand it to see the raw dot products, the same scores after dividing by the square root of the dimension, and the softmax weights.
- Open Show the math. Watch the live values get substituted into softmax(QKᵀ/√d)V one step at a time.
How self-attention actually works
gives every token three vectors derived from its : a query, a key, and a value (the ). The query is what this token is looking for, the key is what each token offers, and the value is the information it hands over if it is attended to.
The score between the query and one key is their dot product, a single number for how well they match. The query is dotted against every key in the sentence, producing one raw score per token. Those scores are then divided by the square root of the key dimension, which keeps them from growing too large as the vectors get wider.
A turns the scaled scores into weights that are all positive and sum to 1. The output for the query token is the weighted sum of every token's value vector, using those weights. So the new representation of "it" is literally a blend of the other words, weighted by how much it attends to each. Every token is processed this way at once, in parallel.
Why run two heads?
One set of query, key, and value vectors can only capture one kind of relationship. In the demo, the first head is semantic: it scores words by meaning, so "cat" and "mat" attend to each other and "warm" leans toward "mat." The second head is positional: it is a previous-token head, where each word looks one step back regardless of meaning.
is exactly this idea scaled up. A real model runs many heads in parallel, each with its own query, key, and value projections, and concatenates their outputs. One head can track subject-verb agreement while another tracks position and a third tracks something we do not have a clean name for.
Notice that the positional head needs a position signal to work at all. Attention on its own is order-blind: shuffle the words and the raw content scores come out the same. That is why models add position information to the tokens before attention ever runs.
Self-attention is one instance of a more general pattern. To see the same score, normalize, allocate loop show up in search ranking and social feeds, read how attention works across search, feeds, and LLMs. And to see where these weights end up, the blended output feeds next-token prediction.
Frequently asked questions
What is self-attention in simple terms?
It is the step where each word in a sentence looks at every other word and builds a new version of itself, blended from the words it finds relevant. A model uses it to work out things like which noun a pronoun refers to, or which adjective modifies which object.
What are the query, key, and value?
Three vectors each token produces from its embedding. The query is what a token is looking for, the key is what a token offers to matches, and the value is the content it contributes when attended to. Scores come from queries dotted with keys, and the output is a weighted sum of values.
Why divide the scores by the square root of the dimension?
As vectors get wider, dot products grow larger on average, which pushes softmax toward putting almost all its weight on a single token and flattening the gradients. Dividing by the square root of the key dimension keeps the scores in a range where softmax stays smooth and training remains stable.
What is multi-head attention?
Running several attention operations in parallel, each with its own query, key, and value projections, then combining their outputs. Each head can specialize, one on meaning and another on position, so the model captures several kinds of relationship at once.
How does self-attention relate to the context window and KV cache?
The keys and values every token computes are what a model caches during generation so it does not recompute them for each new token. That cache grows with the length of the context window, which is a big part of why long contexts cost more memory and time.
Key takeaways
- Attention is a weighted average, not a lookup. Every token contributes to every other token, just by different amounts, and the weights for one token always sum to one.
- Dividing the scores by the square root of the key dimension is not cosmetic. Without it, large dot products push softmax into a corner where one weight is nearly 1 and the gradients flatten out, so training stalls.
- A single head can only track one kind of relationship at a time. Real models run many heads in parallel so one can follow meaning while another follows position.
- Attention itself is order-blind. The raw mechanism would score the same words the same way in any order, which is why a separate position signal has to be added before the scores are ever computed.
- The keys and values a token produces here are exactly what a KV cache stores during generation, which is why longer contexts cost more memory and time.