Speculative Decoding
Reading
- Fast Inference from Transformers via Speculative Decoding — Leviathan, Kalman, Matias (Google Research)
- Looking back at speculative decoding — Leviathan, Kalman, Matias (Google Research blog; you might want to read this first)
About this week
This week, we’re diving into speculative decoding, one of the key techniques behind modern high-throughput LLM inference. The core problem is simple: autoregressive decoding is inherently sequential—generating K tokens normally requires K serial runs of the model. Speculative decoding instead uses a smaller, faster draft model to propose several tokens ahead, then verifies those proposals in parallel with the larger target model. Crucially, the algorithm can accelerate generation without changing the target model’s output distribution.
This also gives us a chance to go deeper on speculative decoding after it came up during our vLLM week. Rather than treating it as another inference optimization in the stack, we’ll look at the actual algorithm: how draft tokens are proposed and accepted, why rejected tokens require a corrected distribution, and how the speedup depends on the draft model’s acceptance rate and relative cost.
In particular, the paper is a nice bridge between the systems questions we discussed with vLLM—latency, memory bandwidth, and parallelism—and the probabilistic details required to make speculative execution exact.
Join us at CASI for discussion at 8 pm, and (optional) quiet reading from 7 pm.
