Definition
Speculative decoding is a technique for speeding up text generation by pairing a large, accurate model with a smaller, much faster one that guesses several tokens ahead on its behalf. The small "draft" model proposes a short run of upcoming tokens, and the large "target" model then checks all of them in a single pass, accepting the ones that match what it would have generated itself and discarding the rest from the point where they diverge. The result is the same output the large model would have produced generating token by token, just arrived at with fewer expensive passes through it.
The reason speculative decoding exists is that standard generation is slow in a specific, frustrating way: producing each new token normally requires a full forward pass through the entire model, and for a large model that pass is expensive even though it is only computing one token's worth of output at a time. Hardware built for parallel computation ends up underused in this pattern, since generating one token barely keeps it busy. Speculative decoding gets around this by verifying several candidate tokens in the time it would normally take to compute just one, putting that idle parallel capacity to work.
What makes speculative decoding different from simply swapping in a smaller, faster model is that it does not change what comes out at all. The large target model still has final say over every token, checking each guess from the draft model against what it would have produced on its own and rejecting anything that does not match. A naive shortcut that just used the small model's output directly would be faster but would also produce a noticeably worse answer. Speculative decoding is built specifically to preserve the exact output of the large model while skipping some of the expensive computation it would otherwise need.
By 2026, speculative decoding is a standard optimization inside most serious model-serving stacks rather than a research curiosity. It shows up under different names and with different flavors, sometimes pairing two separate models of different sizes, sometimes using a lightweight component attached to the same model to speculate about its own future tokens, but the core idea of guess-then-verify is now common wherever teams care about squeezing more speed out of large model deployments without changing the quality of what gets generated. Teams adopting it generally treat it as an infrastructure decision rather than a modeling one, since it changes how fast an existing model serves, not which model gets chosen in the first place.
This page covers how speculative decoding actually works step by step, how it compares to standard token-by-token decoding, what separates it from model distillation, which also pairs a small model with a large one but for a very different purpose, and where the technique earns its keep. The idea worth holding onto is that speculative decoding is a pure speed optimization: done correctly, it changes how fast an answer arrives, not what the answer actually is. Keep that framing in mind through the rest of this page, since every comparison and every practice below traces back to that single guarantee.
Key Takeaways
- Speculative decoding pairs a small, fast draft model with a large target model, letting the draft guess several tokens ahead for the target to verify at once.
- It exists because generating one token at a time through a large model wastes the parallel capacity of modern hardware, and verifying several tokens together puts that capacity to work.
- It preserves the exact output the large model would have produced alone, rejecting any guessed token that does not match, unlike a shortcut that just uses the small model's output directly.
- By 2026 speculative decoding is a standard optimization in most serious model-serving stacks, not a niche research technique.
- It is a speed optimization, not a quality tradeoff; the large model still has final say over every token that ends up in the output.
How Speculative Decoding Works
The process starts with the small draft model generating a short sequence of candidate tokens, often somewhere between two and eight, continuing from wherever the conversation or document currently stands. This step is cheap because the draft model is small, so producing a handful of guesses this way costs relatively little compared to running the same number of steps through the large model. Choosing how many tokens to guess ahead in this step is itself a tunable setting, and it interacts directly with how well the draft model tends to predict this particular kind of content.
The large target model then processes all of those candidate tokens in a single forward pass rather than one at a time, which is possible because checking whether a sequence of tokens matches what the model would have generated can be done in parallel, unlike generating them one after another in the first place. This single pass produces, for each candidate position, what the large model would actually have predicted there. This is the step that actually delivers the speedup, since checking many tokens together costs roughly the same as checking just one on the kind of hardware these models typically run on.
The target model's predictions are then compared against the draft model's guesses token by token, starting from the beginning of the candidate sequence. Every guess that matches gets accepted as final output, for free, without the large model having had to generate it step by step. The moment a guess does not match, everything from that point onward is discarded, and the large model's own prediction at that position is used instead. Even a single accepted token is still a small win, since it came essentially for free compared to generating it the normal way.
The process then repeats from wherever the sequence left off, whether that means many tokens were accepted at once or the very first guess was wrong and only one token moved forward. On average, across a real workload, a decent chunk of guesses tend to match, which is what produces the overall speedup, even though any individual round might see only one token accepted instead of several. Across a long enough response, this back-and-forth between confident stretches of matching guesses and occasional resets is what determines the final speedup a workload actually sees.
Speculative Decoding Compared to Standard Decoding
Standard decoding, sometimes called autoregressive decoding, generates exactly one token per full forward pass through the model, then feeds that token back in to generate the next one, and repeats until the response is finished. It is simple, well understood, and needs nothing extra beyond the one model doing the generating, which is why it remains the default approach in plenty of settings. Anyone building a small internal tool or a low-traffic prototype has little reason to reach for anything more complicated than this.
Speculative decoding adds real complexity in exchange for speed. It requires a second, smaller model that has to be reasonably well aligned with the large model's behavior, plus the machinery to generate candidates, verify them, and handle the accept-or-reject logic correctly. None of that is free to build or maintain, and a poorly chosen draft model can produce guesses that rarely match, in which case the extra machinery buys little to no speedup. Teams sometimes underestimate this setup cost going in, then find the actual engineering work involves as much effort tuning the draft model as it does anything else.
The upside, when it works well, can be substantial: fewer expensive passes through the large model for the same amount of generated text, which translates into faster responses and lower compute cost per token generated. How much faster depends heavily on how often the draft model's guesses actually match the target model's real predictions, since a draft model that guesses badly forces frequent rejections and erases much of the benefit. Providers that offer speculative decoding as a built-in serving option have often already done a fair amount of this tuning work on your behalf.
Standard decoding remains the simpler, safer default when you are not squeezing for every bit of latency, since it has one fewer moving part that can go wrong and no need to source or maintain a compatible draft model. Speculative decoding earns its complexity mainly at scale, where shaving latency and compute cost off of a huge volume of generation requests adds up to a real difference. Treat the decision the same way you would treat any other performance optimization: worth the effort once the scale justifies it, and not before.
What Makes Speculative Decoding Different From Model Distillation
Speculative decoding and model distillation both involve a small model and a large model working together, which is exactly why the two get confused. Distillation trains a smaller model to imitate a larger one's behavior, producing a standalone model that can be deployed entirely on its own, with the large model no longer needed once training is done. Both techniques also tend to get pitched by vendors using similar language about speed and efficiency, which does not help anyone trying to tell them apart quickly.
Speculative decoding never trains anything and never produces a standalone replacement. The small draft model and large target model both run together at generation time, every single time, with the large model always making the final call on what actually gets output. Remove the large model from a speculative decoding setup and you no longer have the same system at all, just a smaller, weaker model on its own. That dependency is the clearest test for telling the two apart: ask whether the smaller model can stand entirely on its own, and if it cannot, you are almost certainly looking at speculative decoding rather than distillation.
The purposes differ just as much as the mechanics. Distillation is aimed at replacing a large model with something cheaper to run everywhere going forward, accepting some quality loss as the price of a smaller footprint. Speculative decoding is aimed at making the large model's own exact output arrive faster, with no intended quality loss at all, since the large model still verifies and approves everything that gets returned. One approach is about permanently shrinking a footprint, the other is about temporarily borrowing speed while keeping full quality intact.
The two techniques are not mutually exclusive and are sometimes combined: a distilled model can serve as the draft model inside a speculative decoding setup, since distillation is actually a common way to produce a draft model that closely tracks the large model's behavior. But that combination only works because the two ideas are doing different jobs, one producing a smaller model, the other speeding up a bigger one that stays firmly in charge. Recognizing that overlap helps explain why teams sometimes describe their setup using both terms in the same sentence without actually contradicting themselves.
Where Speculative Decoding Fits and Where It Does Not
Speculative decoding fits well anywhere a large model is being served at real scale and latency or compute cost per token genuinely matters, which describes most production chat and coding assistants running on frontier-sized models. The speedup compounds across huge volumes of generated tokens, so even a moderate per-token improvement adds up to a meaningful difference in total serving cost and response time. Teams operating at this scale are usually already tracking cost per token closely enough to notice and appreciate a real improvement here.
It also fits well when a reasonably good draft model is already available or cheap to produce, whether that is a smaller version in the same model family or a distilled model trained specifically for this purpose. The technique depends entirely on the draft model guessing well often enough to be worth the overhead, so having a solid draft model on hand is close to a precondition for getting real value out of it. Building a draft model from nothing specifically for this purpose is a real investment, so it pays to check what is already available before starting from scratch.
It fits poorly for small models that are already fast enough on their own, since the whole point of speculative decoding is to buy back speed lost to a large model's cost, and there is little to gain from speeding up something that already responds quickly. Adding the extra machinery in that setting is complexity without a real payoff. There is a certain irony in bolting an optimization onto something that was never the bottleneck to begin with, and it is worth resisting that urge.
It also fits poorly when no adequate draft model exists and building one is not worth the effort for the workload's scale, since a mismatched draft model that rarely guesses correctly can add overhead without delivering a real speedup. In those cases, standard decoding, or a different optimization entirely, is usually the more sensible path. Forcing the technique in anyway, just because it worked well somewhere else, tends to produce disappointing results that make the whole approach look worse than it actually is.
How to Use Speculative Decoding Well
Choose a draft model that is genuinely well aligned with the target model's behavior, ideally from the same model family or explicitly trained to track it, since the speedup depends almost entirely on how often the draft's guesses actually match the target's real predictions. A mismatched pairing can end up slower overall once you account for the extra step of running the draft model in the first place. Revisit this choice periodically too, since newer, better-aligned draft models sometimes become available well after the initial setup was built and never get reconsidered.
Measure the acceptance rate, the fraction of draft tokens the target model actually confirms, in your specific workload rather than trusting a general benchmark number, since acceptance rates vary a lot depending on the kind of text being generated. Code completion and repetitive text tend to see high acceptance, while creative or highly variable writing tends to see lower acceptance and therefore a smaller real-world speedup. A single global number can also hide a lot of variation, so breaking it down by task or content type tends to reveal where the technique is actually earning its keep.
Tune how many tokens the draft model guesses ahead per round rather than assuming more is always better. Guessing further ahead raises the potential payoff when everything matches but also raises the wasted work when a guess is wrong early in the sequence, so the right number depends on your measured acceptance rate rather than a fixed default borrowed from someone else's workload. Treat this as an ordinary tuning parameter to sweep across a representative sample of real traffic rather than something to guess at once and leave alone.
Verify early and often that the technique is actually preserving the target model's exact output rather than silently drifting toward the draft model's behavior, since a bug in the verification logic can quietly let mismatched tokens through. Speculative decoding is only worth the complexity if it is genuinely a free speedup, and losing that guarantee turns it into a mysterious quality regression instead. Build this verification directly into your testing process rather than treating it as a one-time check performed only when the feature first ships.
Keep the infrastructure cost of running two models in mind when estimating overall savings, since the draft model still consumes memory and compute even though it is small, and that cost needs to be netted against the speedup to know the true benefit. On some workloads and hardware, the net gain is smaller than the headline speedup on the large model alone would suggest. Run the actual numbers for your own hardware and traffic rather than assuming a headline benchmark from someone else's setup will transfer cleanly to yours.
Best Practices
- Pick a draft model closely aligned with the target model's behavior, since the speedup depends on how often its guesses actually match.
- Measure acceptance rate on your own workload rather than relying on a general benchmark, since it varies a lot by content type.
- Tune the number of tokens guessed per round based on your measured acceptance rate instead of a borrowed default.
- Verify that the technique preserves the target model's exact output, since a bug in the verification step can quietly change quality.
- Account for the draft model's own memory and compute cost when estimating the true net speedup.
Common Misconceptions
- Speculative decoding is not a way to trade quality for speed; the target model verifies every token, so the final output should match what it would have produced alone.
- Speculative decoding is not the same as model distillation; distillation produces a standalone smaller model, while the draft model here only works paired with the target model.
- A faster draft model does not automatically mean a faster overall system; a draft model that rarely guesses correctly can add overhead without a real speedup.
- Speculative decoding does not help much for models that are already small and fast, since there is little slow generation left to speed up.
- It is not a training technique; nothing gets trained or updated during speculative decoding, it only affects how existing models generate tokens.
