What is speculative decoding? Draft fast, verify once

What you're seeing

Each block is one of the in the sentence. The dim blocks on the left are the prompt, "The rocket launch was delayed". Solid red blocks are tokens the big model has committed. Translucent blocks are guesses from the draft model that nobody has checked yet. The text bar near the bottom shows the same thing as a sentence: committed text in white and pending guesses in gray.

A round starts with the small sphere dropping guesses one after another, quickly. With the default Draft length K of 4, it drops four. Then a box closes around all four guesses and the big sphere pulses: that is one of the big model, scoring every guess in parallel. When the pass finishes, the guesses resolve left to right. Accepted ones turn solid. The first rejected one shatters, everything after it fades out, and a fresh solid block appears in its place. That replacement is the big model's own token, and it comes out of the same pass that did the checking. If all four guesses pass, the big model still adds one bonus token, so a round always moves the sentence forward by at least one token.

The histogram at the bottom left counts how many guesses survived each round. At the defaults the tallest bar sits on the right edge, where all four guesses survived. The second tallest is usually on the far left: roughly one round in five loses even the first guess and moves forward by just the big model's correction. The four stats are measured from the run, not scripted: TOK/S is committed tokens per second over the last few seconds, ACCEPT is the share of drafted tokens that were kept, and SPEEDUP is TOK/S divided by the big model's one-token-per-pass rate. In Big model only mode it reads 1.0x by construction.

Things to try: drag Draft quality down to 0.40 and watch the histogram slide left and the speedup melt away. Push Draft length K to 8 with quality at 0.95 and the sentence fills in a few big strides. Use Side or Top to change the camera.

How does speculative decoding work?

A language model writes one token at a time, and each token needs a full pass through the model (the autoregressive generation post walks through that loop and the that makes it affordable). The surprising part is what limits the speed of that pass. For a big model serving one conversation, the slow step is reading tens of gigabytes of weights out of GPU memory for every token. The arithmetic itself is small and the chip mostly waits on memory. Because the weights are already being read, scoring five positions in one pass costs barely more than scoring one.

Speculative decoding exploits that. It pairs the big model (the target) with a much smaller model (the draft) that shares its vocabulary. Each round goes like this:

Draft. The small model generates K tokens the normal way, one at a time. It is cheap, so this is fast.
Verify. The big model runs one forward pass over the prompt plus all K guesses. Thanks to causal attention, that single pass yields the big model's own prediction at every one of the K positions, plus one more after the last guess.
Accept a prefix. Walk the guesses left to right and keep each one the big model agrees with. Stop at the first disagreement.
Correct. At the first disagreement, use the big model's own token instead. If there was no disagreement, take the extra token from position K + 1 as a bonus.
Repeat from the new end of the sentence.

Every round costs one big-model pass plus K cheap draft steps, and produces between 1 and K + 1 tokens. Plain decoding produces exactly 1 token per big-model pass. That ratio is the whole speedup.

Why the output never changes

The headline promise is that speculative decoding is lossless: the text comes from the same probability distribution as the big model alone. How that works depends on how the model picks tokens.

With , the big model always takes its single most likely token. Then the rule is simple: a guess is accepted only if it is exactly the token the big model would have picked. Every committed token is the big model's choice, so the output is identical, word for word.

With sampling (any above zero), exact matching isn't enough, because the big model doesn't have one "right" token, it has a distribution. The algorithm in the original papers handles this with a modified form of rejection sampling. Let P be the big model's probability for a token at a position and Q the draft model's probability for the same token:

accept the drafted token t with probability   min(1, P(t) / Q(t))

on rejection, sample the replacement from     normalize(max(0, P - Q))

If the draft was underconfident about a token (Q below P), the guess is always kept. If the draft was overconfident (Q above P), it is kept only part of the time. The replacement is drawn from the probability mass the draft under-covered. Put together, each committed token is distributed exactly as if the big model had sampled it directly. That is the proof in Leviathan, Kalman, and Matias (2023) and in the concurrent DeepMind paper by Chen et al. (2023), which calls the same idea speculative sampling.

How much faster is speculative decoding?

It depends on two numbers: how often the big model accepts a guess (call it the acceptance rate, a) and how cheap a draft step is compared to a big-model pass (c). If each guess survives independently with probability a, the expected number of tokens per round with a draft of length K is:

tokens per round = (1 - a^(K+1)) / (1 - a)
time per round   = K * c + 1          (in units of one big-model pass)
speedup          = tokens per round / time per round

The demo uses c of about 0.08 (a 60 ms draft step against a 750 ms verify pass at speed 1). Running it for three minutes per setting and counting committed tokens gave these results:

Big model only: 1.3 tokens per second, 1.0x. This is the baseline every other row is measured against.
K = 1, quality 0.80: 1.6x. 1.8 tokens per round. Even one guess helps.
K = 2, quality 0.80: 2.1x.
K = 4, quality 0.80 (the default): 2.5x, with 3.3 tokens per round and 62% of drafted tokens kept.
K = 6, quality 0.80: 2.6x.
K = 8, quality 0.80: 2.7x, and only 48% of drafted tokens kept, because more of each long draft lands after a miss.
K = 4, quality 0.95: 3.3x. K = 8, quality 0.95: 4.2x.

Those top-end figures are optimistic on purpose: the sentence is short and predictable, and the demo charges nothing for the bigger verify pass. Real systems land in the lower half of the demo's range. The original speculative decoding paper reported a 2x to 3x speedup on an 11-billion-parameter T5 model with identical outputs, and the DeepMind paper reported 2x to 2.5x on the 70-billion-parameter Chinchilla model.

Notice how little the longer drafts added. Doubling K from 4 to 8 at quality 0.80 only moved the speedup from 2.5x to 2.7x, while the share of drafted tokens kept fell from 62% to 48%. Each extra guess costs a draft step, and it survives only if every guess before it survived too. Past a point, longer drafts mostly produce tokens that get thrown away.

Draft model vs target model: what makes a good draft?

The draft model's only job is to imitate the big one on the next few tokens. It doesn't need to be good in general. It needs to agree with the target, and it has to be fast enough that K draft steps stay small next to one big pass. In practice that means a much smaller model from the same family, trained on similar data with the same tokenizer, so the two models split text into the same tokens.

The demo makes the failure case easy to see. At quality 0.40, only 14% of drafted tokens survive, most rounds end with the very first guess rejected, and the speedup drops to 1.2x. At K = 8 and quality 0.40 it is 1.1x: the big model does the same work as before, plus it now waits on a draft model whose work is almost all discarded. Speculative decoding never makes the answer worse, but a bad draft makes it barely faster. Push the acceptance rate toward zero in the formula above and the speedup falls below 1x: every round still pays for K draft steps and still yields only one token.

Predictability matters as much as the draft. In the demo's sentence, glue words like "the" and "and" are accepted more often than content words like "scrubbed", which is also what real drafts experience: boilerplate, code, and text that repeats the prompt are easy to guess, and creative writing at a high temperature is not. That's why acceptance and speedup vary from one prompt to the next.

Separate draft models aren't the only option. As of October 2026, the vLLM speculative decoding docs list several ways to produce the guesses: a separate draft model, EAGLE-style heads trained on top of the target model, multi-token prediction heads, and n-gram matching that copies likely continuations straight from the prompt. All of them feed the same verify step.

When speculative decoding does not help

The trick depends on the big model having spare compute during each pass. A server handling many requests at once already fills that spare compute with other users' tokens, which is what continuous batching is for. At large batch sizes the verify pass stops being nearly free, and the gain from speculation shrinks. Speculative decoding is at its best when latency for a single stream matters most, such as one user waiting on a long answer, a local model on your own GPU, or an agent waiting on each step.

It also costs memory, since the draft model and its own cache have to fit next to the big model. And it doesn't change the total amount of reasoning a model does. If you want better answers by spending more compute at inference, that is a different idea, covered in inference-time compute. Speculative decoding gets the same answer sooner.

What the demo simplifies

The sentence is hand-authored. Each slot has one target token, one plausible wrong guess, and a predictability weight. No model runs in your browser.
The chance that a guess matches is draft quality scaled by a per-word predictability weight (capped at 98%), drawn from a seeded random generator, so every load plays out the same way.
The verify pass costs the same 750 ms whether it checks 1 guess or 8. On real hardware it grows a little with K.
The left-to-right verdict animation is a replay of a result the verify pass already computed, so its time is excluded from TOK/S, as is the pause at the end of the sentence.

Frequently asked questions

Does speculative decoding change the model's output?

No. With greedy decoding the output is identical token for token. With sampling, the modified rejection rule (accept with probability min(1, P/Q), resample from the leftover probability on rejection) makes the output follow exactly the big model's distribution. Any difference comes from floating-point details, not from the method.

What is a draft model?

A small, fast model that guesses the next few tokens for a larger target model to check. It must use the same tokenizer as the target, and it is useful only to the extent that it predicts what the target would say.

How many tokens should the draft model guess?

It depends on the acceptance rate. One miss discards every later guess, so long drafts pay off only when the draft agrees with the target most of the time. In the demo at quality 0.80, K = 6 gave 2.6x and K = 8 gave 2.7x.

Does speculative decoding work with temperature and top-p sampling?

Yes. The acceptance rule works against whatever final distribution the target samples from, so it applies after temperature and truncation are applied to both models. Higher temperatures usually lower the acceptance rate, because a flatter distribution is harder to guess, which shrinks the speedup. See how temperature reshapes the distribution.

Why does a chatbot still stream one word at a time if it uses speculative decoding?

Speculative decoding changes how fast tokens are produced, not the order they come out in. Accepted guesses arrive in small bursts, one burst per verify pass, and the app still sends text to your screen in order as it arrives. Why ChatGPT types one word at a time covers what the streaming you see does and does not tell you.

Key takeaways

Generating a token is limited by reading the model's weights from memory, not by arithmetic, so checking five guessed tokens in one pass costs about the same as generating one. Speculative decoding spends that spare compute.
The big model never gives up control. It keeps the longest run of guesses it agrees with, replaces the first bad guess with its own token, and the rest of the draft is thrown away.
With the real acceptance rule, min(1, P/Q) plus a resample from the leftover probability, the output distribution is mathematically identical to the big model running alone. The speedup is free in quality terms, and only costs memory and engineering.
The whole gain rides on how often the draft agrees with the big model. In the demo, dropping draft quality from 0.80 to 0.40 takes the speedup from 2.5x to 1.2x, and nothing about the big model changed.
Longer drafts are not free. Each extra guess costs draft time and has a lower chance of surviving, because one miss discards everything after it, so the best draft length depends on how predictable the text is.