Temperature vs top-k vs top-p: what each knob actually does

What you're seeing

All four rows show the same 20 candidate for the next word, sorted from most to least likely. The top row is the raw , the scores the model produces before anything turns them into probabilities. The second row is those scores after temperature and a . The third row is what survives top-k, and the fourth is what survives top-p. The strip at the bottom collects the tokens that actually get sampled.

The number beside each row is the effective vocabulary: how many tokens can still be picked at that stage. It reads 20, 20, then k, then the final survivor count. Rows two and three never cut anything, which is the first lesson: temperature changes the odds, but every token is still in the running.

In the third row a hatched shutter slides in from the right and covers everything past the k-th token. In the fourth row a shaded band fills from the left while a dashed line climbs with the running total of probability. When that line crosses p, the shutter drops. The survivors are then rescaled so they add up to 1 again, which is why the bars in lower rows can be taller than the same bars above them.

A few things to try:

Set top-k to 5 and switch between contexts. On "The capital of France is", the fifth token holds about 0.9% of the probability, so k = 5 keeps junk. On "My favorite color is", the same k = 5 cuts 43% of the probability, including perfectly good colors.
Turn top-k off (slide it to 20) and set top-p to 0.9. Now the survivor count adapts: 2 on the confident prompt, 11 on the open-ended one.
Press Sample 100 at T = 0.1, then at T = 1.5. With both cuts off, the low temperature produced 4 distinct tokens out of 100 and the high temperature produced 19.

The stats in the top right describe the final row: ENTROPY is how spread out it is in bits, SURVIVORS is the final token count, N counts samples, and LAST is the most recent pick.

What is the difference between temperature and top-p?

divides every logit by T before the softmax. Below 1, the gaps between scores grow and the leader pulls further ahead. Above 1, the gaps shrink and the distribution flattens. On the open-ended prompt in the demo, the top token holds 14% at T = 1 and 74% at T = 0.1. No token is ever removed, though. Even at T = 2 the 20th token still has a nonzero chance.

p_i = exp(z_i / T) / sum_j exp(z_j / T)

(also called nucleus sampling) is a cut. It sorts the tokens, adds up their probabilities from the top, and stops at the first token where the running total reaches p. Everything after that is discarded and the survivors are renormalized. Temperature decides how the probability is spread. Top-p decides where the line is drawn.

Because top-p runs on whatever temperature produced, the two interact. On the open-ended prompt with p = 0.9, the survivor count is 8 at T = 0.5, 11 at T = 1, and 14 at T = 1.5. Same p, different nucleus. For a deeper look at the first knob on its own, see how temperature reshapes the next-token distribution.

What is the difference between top-k and top-p?

keeps a fixed number of tokens: the k most likely, no matter how much probability they hold. Top-p keeps a fixed share of the probability, no matter how many tokens that takes. That one difference decides how each behaves when the model's confidence changes.

The demo panel labeled 'Same knobs, two contexts' runs your current settings on both contexts at once. A fixed k gives the same headcount in both columns. The weakest token that k lets through tells the story: about 0.9% on the confident prompt, which is noise, and 8.7% on the open-ended one, which is a reasonable answer. Top-p alone keeps 2 and 11.

This is why top-p became the more common default. A confident model gets a short list and an uncertain one gets a long list, without anyone retuning a number. Top-k is still useful as a hard cap on how many tokens the sampler considers. The top-k and top-p deep dive covers both cutoffs on their own, plus , a newer cutoff that scales with the top token's probability.

What order are temperature, top-k, and top-p applied?

There is no single standard. The demo uses the order in Hugging Face Transformers, which builds its sampler as temperature, then top-k, then top-p (then min-p). Many other stacks do the same.

llama.cpp does not. Its default sampler chain, as of October 2026, runs top-k, top-p, and min-p first and applies temperature last, and it lets you reorder the chain with --samplers. In that order, temperature cannot change which tokens survive the cut. On the open-ended prompt, top-p 0.9 keeps 11 tokens at every temperature, where the Transformers order keeps 14 at T = 1.5.

Transformers default:  logits -> temperature -> top-k -> top-p -> sample
llama.cpp default:     logits -> top-k -> top-p -> min-p -> temperature -> sample

Hosted APIs add another wrinkle: they decide which knobs you get. OpenAI's API, for example, has no top-k parameter, and as of October 2026 its reasoning models reject custom temperature and top-p values. If you move a config between tools, check what each one actually exposes and in what order it applies them.

What are the best temperature and top-p settings?

There is no universal best, but there are sane starting points:

Code, extraction, structured output: low temperature (0 to 0.3), or . You want the single most likely answer.
Chat and factual answers: temperature around 0.7 with top-p around 0.9. Enough variety to sound natural, with the long tail cut off.
Brainstorming and fiction: temperature 1.0 to 1.2 with top-p 0.92 to 0.95.

Change one knob at a time. Raising temperature flattens the curve, which lets more tokens into the top-p nucleus, so raising both stacks the effect twice. If outputs ramble or drift off topic, lower one of them before touching the other. Drift at high temperature is closely related to the failure shown in the hallucination simulation, where small sampling errors compound over a long answer.

How the demo computes it

The 20 logits for each context are hand-set examples, chosen so the probabilities look like a real model's: one dominant answer for the capital of France, a long gentle slope for favorite colors. Everything after that is computed live in the page: the softmax at your temperature, the top-k cut, the running sum for top-p, the renormalization, and every sample, drawn from a seeded random number generator so two loads play out identically. The page checks that every stage sums to 1.

Probability bars use a square-root scale so a 1% token is still visible next to an 80% one. The dashed cumulative line in the fourth row uses a plain linear scale from 0 to 1. To see where these logits come from in the first place, read how LLMs generate text.

Frequently asked questions

Should I use temperature or top-p?

Usually both, but tune one. Temperature sets how adventurous the model is across the board. Top-p sets a safety net that trims the unlikely tail. A common pattern is to pick a top-p near 0.9 and adjust temperature for the task.

Does temperature 0 make top-p irrelevant?

In practice, yes. Temperature 0 is treated as greedy decoding: the single most likely token wins every time, so there is nothing left for a cutoff to trim.

What does top-p 1.0 mean?

It turns top-p off. The running total only reaches 1 after every token, so nothing gets cut. In the demo, p = 1.0 shows OFF and the fourth row passes the third row through unchanged.

Can I use top-k and top-p together?

Yes, where the tool exposes both. Top-k caps the headcount first, then top-p trims inside that cap. The demo's default (k = 8, p = 0.9) shows the stack: on the open-ended prompt, k cuts 20 tokens to 8 and p trims those to 7.

Why do two tools give different results with the same settings?

They may apply the steps in a different order, use different defaults for the knobs you did not set, or not support a knob at all. llama.cpp's default chain applies temperature after the cuts, while Hugging Face Transformers applies it before.

Key takeaways

Temperature never removes a token. It only moves probability between tokens. Top-k and top-p are the steps that actually take tokens off the table.
A fixed top-k ignores the shape of the distribution. In the demo, k = 5 keeps a 0.9% long shot on a confident prompt and throws away 43% of the probability on an open-ended one.
Top-p adapts on its own. The same p = 0.9 keeps 2 tokens when the model is sure and 11 when it is not.
Order matters, and it is not universal. Hugging Face Transformers applies temperature before the cuts, while llama.cpp's default chain applies it after, so the same three numbers can produce different survivor sets in different tools.
Tune one knob at a time. Raising temperature and top-p together compounds the effect, which is how most rambling, off-topic sampling configs happen.