Definition
Speculative decoding is an LLM inference acceleration technique in which a small, cheap draft model proposes the next several tokens and the large target model verifies all of them in a single parallel forward pass. Verification in parallel is much cheaper than generating token by token, so accepted tokens arrive at a fraction of the normal cost; at the first rejection, generation falls back to the large model's own choice.
The acceptance-based scheme guarantees the output distribution is identical to what the large model would have produced alone — the speedup costs nothing in quality, only extra compute for the draft model. Realized speedup depends on the acceptance rate: a draft model well-matched to the target (often a distilled sibling) accepts most proposals on predictable text and fewer on unpredictable text.
How it comes up in interviews
A favorite "how would you make inference faster without changing the output?" answer — the provable-equivalence property is the point interviewers listen for, along with what governs the speedup (acceptance rate) and the relationship to distillation (the draft is often a distilled version of the target).