Does speculative decoding require retraining the target?

The original approach can use an existing target without retraining, although particular systems may train or adapt their drafting components.

Does it improve factual accuracy?

Not inherently. Exact distribution-preserving decoding retains the target's output distribution, including its limitations.

Will every draft token be accepted?

No. Acceptance depends on the relationship between draft and target probabilities at the current prefix, and rejected prefixes invalidate later speculative continuations.

AI Glossary / Generative AI and language / Advanced

Speculative decoding

Definition

Speculative decoding is an inference technique that accelerates autoregressive generation by proposing several tokens with a cheaper draft process and checking them with a target model. Accepted proposals let the system advance more than one token per target-model iteration. In exact speculative sampling, a specific acceptance-and-correction rule preserves the target output distribution. Speed gains depend on proposal quality, implementation, and hardware; they are not guaranteed for every workload.

What it means

Ordinary autoregressive generation repeatedly runs the target model to produce the next token. This creates a sequence of dependent steps. A faster draft model can guess several likely next tokens before the target checks them.

The target does not merely approve text because it sounds reasonable. In the exact algorithm, acceptance is governed by the draft and target probabilities, with a correction when a draft token is rejected.

Preserving a distribution is different from reproducing the identical sampled sentence under every random seed. It means that the probabilities of possible outputs match the specified target decoding distribution under the algorithm's assumptions. Greedy and stochastic decoding require appropriate treatment.

How it works

Starting from an accepted prefix, the draft model proposes a short continuation. The target evaluates the proposed positions in parallel, using the relevant prefixes to obtain its probabilities.

The system accepts a consecutive prefix of proposed tokens according to the acceptance rule. At the first rejection, later draft tokens based on that rejected prefix are discarded, and a corrected token is sampled. If all proposals are accepted, the target can provide an additional token.

The process repeats. Exact speculative sampling uses acceptance probability min(1, p(x)/q(x)) for a proposed token and, after rejection, a distribution proportional to the positive part of p minus q. Here p is the target distribution and q the draft distribution for the same prefix and decoding setup.

Three practical examples

1. A text-serving system uses a smaller compatible draft model to propose tokens for a larger target, then measures latency on its actual requests.

2. A summarization service evaluates whether repetitive, predictable portions of its outputs produce enough accepted draft tokens to justify the extra draft work.

3. A deployment team compares speculative and ordinary decoding across batch sizes, checking both resource use and whether the implementation preserves the intended output behavior.

Think of it this way

Think of a junior assistant preparing a short draft while a senior reviewer checks several proposed steps together. When the draft is often acceptable, the reviewer can move through the work faster.

The analogy has limits. Exact speculative decoding relies on probability calculations, not subjective editorial approval. The target model also remains capable of its original mistakes: faster decoding does not make its facts more reliable or its reasoning more accurate by itself.

Worked example

Consider a single-position toy vocabulary containing A and B. The draft distribution is q(A) = 0.6 and q(B) = 0.4. The target distribution is p(A) = 0.3 and p(B) = 0.7.

If the draft proposes A, accept it with probability 0.3/0.6 = 0.5. If it proposes B, accept with probability one because 0.7/0.4 exceeds one. The overall probability of accepting A directly is 0.6 × 0.5 = 0.3, and of accepting B directly is 0.4.

The remaining rejection probability is 0.3. The positive part of p − q is zero for A and 0.3 for B, so correction always returns B. Final probabilities are therefore 0.3 for A and 0.4 + 0.3 = 0.7 for B, matching the target. Multi-token speculation applies this logic with prefix-dependent distributions.

Illustrative case study

Fictional case: Poplar Serve pilots speculative decoding for a report assistant. Short, predictable completions show promising latency, but long prompts and larger batches change the balance between draft overhead and target computation.

The team measures end-to-end latency and throughput separately, records accepted draft lengths, and compares the same target model and decoding configuration. It also checks memory requirements for keeping the extra model available.

The decision is workload-specific. A successful pilot for one request mix does not become a universal promise that every application will run twice as fast or that the target model's quality has improved.

Common misconceptions and mistakes

Speculative decoding is not simply replacing the target model with a smaller model. The target still checks proposals and determines the preserved distribution in the exact method.

It is also not ordinary best-of-many generation, which chooses among completed candidates. The proposal verification happens during token generation.

Another mistake is claiming exactness for any approximate acceptance heuristic. The mathematical guarantee depends on the specified algorithm and compatible probability handling. Finally, fewer serial target iterations need not mean fewer total operations or lower cost under every hardware and batching configuration.

Business use

The technique can reduce response latency when draft tokens are cheap and frequently accepted, and when the target benefits from parallel verification. It is primarily a serving optimization.

Assess it using the intended traffic mix, output lengths, batching policy, and hardware. Track draft overhead, acceptance behavior, memory, latency, and throughput. Preserve the target configuration when comparing quality or output distributions. A business case should be based on measured operational savings, with the exact algorithm or approximation clearly documented.

Check your understanding

A draft favors A more than the target does. Can the system accept every proposed A and still claim the toy target distribution is preserved?

Answer: Not under the stated setup. Some A proposals must be rejected and corrected so the final A frequency falls from the draft's 0.6 to the target's 0.3.

Related terms

Rejection sampling
Greedy decoding
Probability distribution
Autoregressive decoding
draft model
target model
speculative sampling
inference latency
throughput
batching

Learn next

Rework the two-token probability example with a different draft distribution and verify the final probabilities. Then examine a documented implementation's acceptance rule and benchmark it against ordinary decoding on a fixed workload. Separate distributional correctness from measured speed and resource consumption.

Sources

Sources: Leviathan, Kalman, and Matias, Fast Inference from Transformers via Speculative Decoding, https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf — original overview, acceptance rule, correction distribution, and algorithm read. Publication record: https://proceedings.mlr.press/v202/leviathan23a.html. The two-outcome calculation and fictional serving case are original illustrations; the paper's reported speedups are not treated as universal guarantees.

Previous: Specular-highlight removal Next: Speculative multithreading