What Is Speculative Decoding and Why Does It Speed Up LLMs?

If you have ever watched a large language model stream its answer word by word, you have felt the fundamental bottleneck of modern text generation. The model is fast at math but slow at delivery, and that slowness has a specific cause. Every single token it produces requires a full forward pass through billions of parameters, and each pass must wait for the one before it to finish. You cannot generate token fifty until you have generated token forty-nine, because token forty-nine is part of the input that decides token fifty. This is what people mean when they call autoregressive generation "sequential." It is a chain, and you can only add one link at a time.

Speculative decoding is one of the cleverest answers the field has found to this problem. It does not make your GPU faster, it does not shrink the model, and, done correctly, it does not change a single character of the output. Instead, it changes the *order* in which work happens so that the expensive model spends more of its time verifying cheap guesses in parallel and less of its time generating tokens one at a time. In this article we will build up the idea from first principles, walk through exactly how the draft model and verification step interact, look at real code, and talk about where it helps, where it hurts, and how to reason about it as an engineer.

Why Token Generation Is Slow In The First Place

To understand the fix, you have to understand the pain precisely. A large decoder-only transformer generates text one token at a time. Given a prompt, it runs a forward pass and produces a probability distribution over the entire vocabulary for the next token. You sample from that distribution, append the chosen token to the sequence, and run the forward pass again. Repeat until you hit a stop condition.

The critical detail is that a forward pass over a large model is memory-bandwidth bound, not compute bound, during generation. When you are producing a single token, the GPU has to load all of the model's weights out of high-bandwidth memory to do the multiply-accumulate work, but the amount of actual arithmetic per token is tiny relative to how much data got moved. The GPU's compute units sit mostly idle while they wait for weights to stream in. This is the key insight that makes speculative decoding possible: a forward pass that processes one token and a forward pass that processes eight tokens cost *almost the same wall-clock time*, because in both cases the dominant expense is loading the weights, not the math applied to them.

Think about what that means. If you could somehow hand the big model eight candidate tokens at once and ask "are these the tokens you would have produced?" you could check all eight in roughly the time it normally takes to produce one. The problem, of course, is that you do not know the eight tokens in advance. If you knew them, you would not need the model. That is the gap speculative decoding fills.

The Core Idea: Guess Cheaply, Verify Expensively

Speculative decoding introduces a second, much smaller model called the draft model (sometimes called the assistant model or the approximation model). The draft model is trained on similar data and shares the same tokenizer as the large target model, but it is a fraction of the size, so running it is cheap and fast.

The strategy works in two phases that repeat in a loop:

1. Drafting. The small draft model generates a short run of candidate tokens autoregressively. Because it is small, producing, say, four or five tokens is quick. These are guesses about what the big model would probably say next.
2. Verification. The large target model runs a single forward pass over the original context plus all the drafted tokens at once. Because of the memory-bandwidth reality we just described, checking five tokens in one pass costs about the same as generating one token normally. In that one pass, the target model tells us, for each drafted position, what probability *it* would have assigned. We then accept the drafted tokens that agree with the target model and reject the rest.

The magic is in the acceptance rule. When the guesses are good, you get several tokens confirmed for the price of one big forward pass. When the guesses are bad, you fall back to roughly the speed of normal generation. Crucially, and this is the part that surprises people, the tokens you end up emitting are statistically identical to what the target model would have produced on its own. You are not trading quality for speed. You are trading wasted parallel capacity for speed.

How Verification Preserves The Output Distribution

This is the heart of the technique, and it is worth slowing down. A naive version of "guess and check" would say: accept a drafted token only if it exactly matches the target model's most likely token. That would work for greedy decoding, but it would break sampling, because it would bias the distribution toward high-probability tokens and quietly change the character of the output. The authors of the original speculative sampling work solved this with a rejection-sampling scheme that provably preserves the target distribution.

Here is the logic for a single drafted token. Let p(x) be the probability the target model assigns to token x at this position, and let q(x) be the probability the draft model assigned when it guessed token x.

• If p(x) is greater than or equal to q(x), the target model likes this token at least as much as the draft did, so you accept it unconditionally.
• If p(x) is less than q(x), the draft was overconfident. You accept it with probability p(x) / q(x), and otherwise you reject it.
• When you reject, you do not just stop. You resample the replacement token from an adjusted distribution, the normalized positive part of p(x) - q(x), which corrects for the bias introduced by having consulted the draft in the first place.

You process the drafted tokens left to right. The first time a token is rejected, you discard it and everything after it, emit the resampled token in its place, and start the next drafting round from there. If every drafted token is accepted, you get a bonus: the same target forward pass already computed the distribution for the position *after* the last drafted token, so you can sample one extra "free" token from it. That means in the best case a draft of k tokens can yield k + 1 accepted tokens from a single target pass.

The mathematical result is clean and important to internalize: the sequence of tokens produced by this procedure has exactly the same probability distribution as sampling directly from the target model. Speedup is free of quality cost, assuming your implementation is correct. When people say speculative decoding is "lossless," this rejection-sampling proof is what they are pointing at.

A Concrete Walkthrough

Let us make it tangible with a tiny example. Suppose the context so far is "The capital of France is" and we want the next tokens.

The draft model runs first and proposes four tokens:

draft proposes:  " Paris" "," " which" " is"

Now the target model runs one forward pass over "The capital of France is Paris, which is" and returns its own probability for each position. We walk left to right applying the acceptance rule:

• Position 1, " Paris": target strongly agrees, p >= q, accept.
• Position 2, ",": target agrees, accept.
• Position 3, " which": target actually preferred " the" here. The draft's q for " which" was high but the target's p for it is low, so p / q is small. We roll the dice and reject.

At the rejection, we throw away " which" and the " is" that followed it. We resample from the corrected distribution and get, say, " the". Our emitted sequence for this round is:

accepted:  " Paris" "," 
resampled: " the"

So from one draft round and one target pass we advanced three tokens: two accepted plus one corrected resample. The next round starts drafting after " the". If the draft is well matched to the target on this kind of text, most rounds will accept most tokens, and the average number of tokens per target pass, the acceptance rate, climbs well above one. That average is the single number that most directly governs your speedup.

What Determines The Speedup

The wall-clock improvement from speculative decoding is not a fixed constant. It depends on a handful of interacting factors, and understanding them is what separates someone who *uses* the technique from someone who can *tune* it.

• Acceptance rate. The more drafted tokens the target accepts on average, the fewer expensive target passes you need per unit of output. This is dominated by how well the draft model's distribution matches the target's on your actual traffic. A draft that is a distilled sibling of the target accepts far more than an unrelated small model.
• Draft length (the lookahead `k`). Drafting more tokens per round means more potential tokens per target pass, but it also means more draft compute and a higher chance that a late token gets rejected and wastes the ones after it. There is a sweet spot, often somewhere around four to eight tokens, and it shifts with the workload.
• Draft model cost. The draft runs autoregressively too, so if it is too large it eats the savings. The ideal draft is small enough to be nearly free yet accurate enough to be accepted often. This tension is the central design tradeoff.
• Batch size. Speculative decoding shines at low batch sizes and low latency, exactly the interactive single-user regime where the target model is most memory-bandwidth bound and its compute units are most idle. At very high batch sizes the target model is already keeping its compute units busy, the idle capacity that speculation exploits shrinks, and the benefit narrows.
• Domain predictability. Highly structured or repetitive text, code, boilerplate, formatted output, is easy to predict, so acceptance rates soar. Free-flowing, surprising prose is harder, and acceptance drops.

A useful mental model: speculative decoding converts *spare parallel compute* into *lower latency*. If you have spare parallel compute (small batch, memory-bound target) and a good predictor (a well-matched draft), you win big. If you do not, the technique gracefully degrades toward ordinary generation speed rather than making things worse.

Trying It Yourself With Hugging Face

You do not need to implement the rejection sampler by hand to experiment with this. The Hugging Face transformers library exposes assisted generation, which is speculative decoding under the hood, through a single argument. You load a large target model and a small draft model that shares the same tokenizer, then pass the draft as the assistant.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

target_name = "meta-llama/Llama-2-13b-hf"
draft_name = "meta-llama/Llama-2-7b-hf"  # smaller sibling, same tokenizer family

tokenizer = AutoTokenizer.from_pretrained(target_name)
target_model = AutoModelForCausalLM.from_pretrained(
    target_name, torch_dtype=torch.float16, device_map="auto"
)
draft_model = AutoModelForCausalLM.from_pretrained(
    draft_name, torch_dtype=torch.float16, device_map="auto"
)

prompt = "Explain why the sky appears blue in a single paragraph."
inputs = tokenizer(prompt, return_tensors="pt").to(target_model.device)

# assistant_model turns on speculative (assisted) decoding
output = target_model.generate(
    **inputs,
    assistant_model=draft_model,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

The only line that matters for our purposes is assistant_model=draft_model. Remove it and you get ordinary autoregressive decoding. Add it and the library runs the draft-then-verify loop for you, applying the acceptance rule internally so the output distribution is preserved. The two models must share a compatible tokenizer, because the drafted token ids have to mean the same thing to both models when the target verifies them.

If you want to feel the difference, time a generation with and without the assistant model on the same prompt and the same sampling settings. On predictable prompts you will typically see a meaningful drop in latency, and the decoded text will be drawn from the same distribution either way.

Variants: Beyond A Separate Draft Model

The classic setup uses two separate models, but the field has produced several variations on the theme, all sharing the guess-and-verify skeleton.

1. Self-speculation and early-exit drafting. Instead of a separate model, you use a cheaper version of the target itself as the drafter, for example by running only a subset of its layers to produce guesses that the full model then verifies. This removes the need to find and host a matching draft model.
2. Medusa-style extra heads. You bolt several lightweight prediction heads onto the target model so it proposes multiple future tokens at once from a single backbone pass, then verify those candidates in a tree. There is no separate draft model to serve at all.
3. N-gram and prompt-lookup drafting. For tasks with heavy repetition, such as summarization or code editing where much of the output echoes the input, you can draft by literally copying likely continuations from the prompt or from an n-gram table. It costs almost nothing and, on the right workload, accepts remarkably often.
4. Tree-based speculation. Rather than a single linear draft, you propose a *tree* of possible continuations and verify many branches in one target pass, keeping whichever path the target accepts furthest along. This raises the expected number of accepted tokens per pass at the cost of more verification bookkeeping.

Each variant is a different answer to the same two questions: how do we produce cheap guesses, and how do we verify them in parallel without corrupting the output distribution. Once you internalize that framing, every new paper in this space slots neatly into place.

When Speculative Decoding Is Worth It

Speculative decoding is not a universal free lunch, and part of being a good engineer is knowing when to reach for it.

• Great fit: interactive, latency-sensitive, single-stream or low-batch serving, where a user is watching tokens appear and every hundred milliseconds matters. Chat assistants, coding copilots, and streaming completions are the classic wins.
• Great fit: predictable or structured output, code, JSON, templated text, or anything where a small draft can guess correctly most of the time and acceptance rates stay high.
• Weaker fit: very high-throughput batch serving where the target model is already compute bound and its parallel capacity is saturated. Here the spare capacity that speculation feeds on has already been consumed by the batch, so the latency benefit shrinks.
• Weaker fit: situations where you cannot obtain a well-matched draft model. A poorly aligned draft has a low acceptance rate, and the wasted draft compute plus frequent rejections can erode most of the gains.
• Operational cost: you now have two models to load, version, and keep in memory. The draft consumes VRAM and adds serving complexity, which is a real tradeoff even when the latency math works out.

The decision usually comes down to two questions. First, is my serving regime memory-bandwidth bound, meaning low batch and latency sensitive? Second, can I get a draft whose predictions the target accepts often on my real traffic? If both answers are yes, speculative decoding is one of the highest-leverage optimizations available, because it buys latency with no quality cost and no retraining of the target.

The Bigger Picture For AI Engineers

What makes speculative decoding such a satisfying technique to study is that it is not a hack. It rests on a precise understanding of *why* inference is slow, memory bandwidth dominating single-token passes, and a precise piece of mathematics, rejection sampling that provably preserves the target distribution, that turns that understanding into free speed. It is systems thinking and probability theory meeting in a single loop.

That combination is exactly what modern AI engineering rewards. The people who ship fast, reliable LLM products are rarely the ones who only know how to call an API. They are the ones who understand what happens inside the box: how attention and the key-value cache shape memory use, why batching changes the compute profile, how quantization and speculative decoding and continuous batching each attack a different part of the latency budget, and how to measure whether an optimization actually helped on their own traffic instead of trusting a benchmark from a blog post.

If reading this made you want to understand the rest of that stack, the parts of inference and serving that turn a research model into a product, that curiosity is worth following. Our AI Engineering Roadmap course on teachyou.ai is built to take you exactly there. It walks through the transformer internals that make techniques like this possible, the serving and optimization patterns that production teams actually use, and the hands-on practice of measuring and improving real systems, so that the next time you read a paper like the one behind speculative decoding, it reads less like magic and more like something you could have designed yourself. Speed, correctness, and a clear mental model of why both hold: that is the mindset the roadmap is designed to build, and speculative decoding is a perfect example of where it leads.