Why do LLMs generate one token at a time?

A language model writes by choosing small pieces of text called tokens. A token can be a word, part of a word, or punctuation. The model assigns probabilities to possible next tokens, then a selection rule chooses one.

Imagine the text ends with The sky is. If the model chooses blue, its next prediction uses The sky is blue. Choosing clear would give it a different input. The text already available at each step is called the prefix.

That dependency explains the one-token-at-a-time pattern: the next prediction depends on which output token was actually chosen. Speculative decoding tries to advance several tokens per expensive model call by checking a cheaper model's proposed continuation.

Why does the previous token matter?

Generation that conditions each prediction on the already chosen output is called autoregressive generation. Choosing a token changes the input to the following prediction.

A GPU can process many known prompt tokens together because their identities are already available. During generation, those future inputs do not exist yet. Scoring both possible futures after blue and clear would create branching work; the model still needs to choose which path to follow.

Training examples already include their continuation tokens. Their positions can be processed together, while a rule prevents each position from using later tokens. This rule, called a causal mask, preserves the prediction task. The same distinction allows parallel verification, following the decoder construction in the Transformer paper.

This describes conventional autoregressive generation; other generative architectures can work differently.

How can a model check several tokens together?

Call the model whose output we want the target. A cheaper helper, the drafter, proposes a short continuation. Verification means asking the target to score that known candidate sequence and deciding which prefix to keep.

As the previous post explains, processing a known prompt is called prefill; extending it is called decode. A draft gives the target another known sequence to process.

Suppose the drafter proposes four tokens. The target scores the first after the real prefix, the second after that prefix plus the first proposal, and so on. Those inputs are available together, and the causal mask keeps each position consistent with its proposed history.

The target has evaluated one proposed path. It has not independently chosen four future tokens. This is the mechanism described by Chen and colleagues.

For greedy decoding, the selection rule always chooses the target's highest-probability token. We can keep proposals until the first disagreement:

1draft proposes:      A B C D2target agrees with:  A B3first disagreement:      X instead of C4output advances by:  A B X

D was proposed after C. Rejecting C invalidates that proposed suffix, so we discard it. If all four proposals match, the target can supply a bonus token, advancing by five. End-of-sequence rules and output limits can stop a round earlier.

This greedy example compares token choices. Random sampling needs a different rule to preserve the target's probabilities.

What changes when token selection is random?

With random sampling, the selection rule draws a token according to its probability rather than always choosing the largest one. Consider this synthetic vocabulary of just two symbols at the same prefix:

Here, p(A) = 0.6 means the target would choose A 60% of the time. The drafter's q(A) = 0.8 overproduces A. Accepting every draft would therefore change the output distribution.

The standard exact sampler draws a proposal x from the draft distribution q, then accepts it with probability:

1accept(x) = min(1, p(x) / q(x))

For A, acceptance is 0.6 / 0.8 = 0.75. For B, it is 1.00, because the ratio exceeds one. The accepted probability mass is 0.8 × 0.75 = 0.6 for A and 0.2 × 1 = 0.2 for B.

The remaining rejection mass is 0.2. At the first rejection, the algorithm draws a replacement from the target's underrepresented possibilities, then discards the remaining proposals. This replacement distribution is the normalized positive part of p - q:

1r(x) = max(p(x) - q(x), 0) / sum_y max(p(y) - q(y), 0)

In our example, only B is underrepresented. The replacement puts all rejected mass on B, restoring the target's final probabilities to 0.6 and 0.4.

The residual distribution is used only on rejection. If every proposal is accepted, the algorithm draws a bonus token from the target. Together, acceptance and replacement recover the target distribution, as shown in the original speculative-decoding algorithm.

The accompanying exact-arithmetic check verifies this small example. It runs no model and measures no inference speed.

This guarantee applies to the distributions actually being sampled. Changes such as adjusting randomness with temperature, limiting eligible tokens, repetition penalties, or output constraints must be handled consistently. Preserving a distribution does not promise identical random output from the same seed. Floating-point and batching effects still matter.

When does checking a draft save time?

Ordinary decode can spend much of its time moving the model's learned parameters, or weights, from GPU memory. Scoring several proposed tokens together can do more arithmetic per expensive data access. Chen and colleagues' conditional-scoring analysis explains why this can help at small serving batches. Verification cost still depends on draft length and system load.

For an actual system, I would compare:

1speculative time per committed token2  = (drafting + verification + scheduling/state work)3    / committed output tokens

A committed token is one retained in the answer. A round that commits three tokens in the time ordinary decoding needs for two helps. Committing three in the time it needs for four is slower, regardless of acceptance rate.

Longer drafts offer more possible progress, but a later proposal counts only if every earlier one survives. Ignoring stopping rules, a draft of length K has expected committed count:

11 + sum, for j = 1..K, of P(first j draft tokens are accepted)

The extra one is the correction or bonus token. Replacing those probabilities with powers of one average acceptance rate is a simplifying assumption; an observed average does not establish it.

What happens to state after a rejection?

The engine must also manage the records it computed while checking the draft.

Models that use attention to look back at earlier tokens save key and value records, usually called the K/V cache, to reuse work on later predictions. Verification can write these records for the proposed suffix. After rejection, only records belonging to the accepted path remain valid. The speculative tail must be ignored, overwritten, or recycled. Grammar rules, repetition penalties, and stop conditions must also follow the accepted history.

Sampling the figure's replacement X and computing its own K/V are separate events. Its records may be computed on the following model pass, so the output-token count and cached-position count can differ.

A separate attention-based draft model has its own K/V, computed with its own weights. Matching token positions do not make those records interchangeable with the target's. Integrated predictors that reuse target features or predict multiple tokens have their own sharing arrangements; not every drafter is a separate model.

Hybrid models can also carry a running summary called recurrent state, which needs additional rollback. Fireworks' October 2026 Kimi K3 report describes ReplaySSM reconstructing accepted recurrent-state updates from a committed checkpoint. Rejected proposals cannot remain in the running state.

Why does DeepSeek change the policy with load?

DeepSeek's V3 report described a multi-token prediction module that could support speculative decoding. It provides historical context; later serving designs go further. DeepSeek-V3 technical report, sections 2.2 and 5.4.2

The July 2026 DSpark paper combines parallel proposal generation with a lightweight sequential component. It describes selecting proposal prefixes using confidence and a measured hardware throughput profile, and deployment under live DeepSeek-V4 traffic. Verifying the full candidate block is not always worthwhile.

That selection also affects correctness. Choosing whether to verify an earlier proposal using knowledge of later candidate realizations can bias the output distribution. The paper's causal admission rules address this. Scheduling decisions can change sampling behavior.

DeepSeek's DeepInfer deployment Q&A says production drafts five tokens and chooses verification prefixes using confidence and hardware profiles. Its Open-PerfectBlend EP64/DP64 table estimates +5.4% throughput at 60 concurrent sequences per rank, but −1.5% at 16, for dynamic verification versus fixed-five verification. These are DeepSeek's workload-specific estimates, not comparisons with ordinary non-speculative decoding or results from this lab.

The useful policy changes with the operating point.

How does the rest of the service affect the trade?

At higher concurrency, ordinary decode can already share weight reads across many requests. Speculative verification then competes for compute and batch capacity. Mixture-of-experts (MoE) models also route work among specialized model components, adding dispatch and communication costs. Variable verification lengths affect allocation, kernels, and GPU execution plans such as CUDA graphs.

The service objective matters too. Shortening one response while lowering sustainable requests per second can suit a latency-sensitive endpoint and hurt a heavily loaded shared service. Compare both against an explicit latency requirement and workload.

Production use is public. Together's December 2025 account describes target-specific draft training and serving adaptations alongside hardware and kernel changes. Modal's September 2026 update discloses custom DFlash speculators for specific Kimi and Qwen managed endpoints, using different serving engines. Neither establishes the stack behind every model from those providers.

vLLM and SGLang expose separate drafters, model-native predictors, and feature-based methods. Support depends on the method, model, hardware, parallelism, and scheduler configuration. vLLM distinguishes standard rejection sampling from explicitly lossy verification options. An available flag still needs a deployment test.

I have not found sufficient public evidence to identify the current speculative implementation behind an arbitrary OpenAI, Anthropic, or Gemini request. Foundational papers do not establish those services' present production stacks.

What would I test next?

For this lab, I would choose one open model, one supported proposer, and fixed prompts. Run ordinary decoding, then replay that workload with speculation at several concurrency levels.

Record time to first token, time between committed tokens, completed requests per second at a latency limit, draft and verification work, accepted-prefix lengths, peak memory, and cost. Include easy continuations and frequent disagreements. Test rollback, stop sequences, output limits, and constrained decoding separately.

For greedy decoding, compare token sequences. For sampled decoding, test the sampler's distributional behavior; matching a few answers is not a correctness proof. Keep numerical differences visible.

The synthetic probability check establishes the arithmetic. A serving experiment would show whether a particular policy helps a particular workload. Neither substitutes for the other.

Longer sessions raise another question: what should the model keep in context, even when a much larger window is available?