hidekazu-konishi.com

Speculative Decoding and Draft Models on AWS - Acceptance Rates, Which Methods Preserve the Output Distribution, and What the Managed Draft Model Takes Away

First Published: 2026-08-24
Last Updated: 2026-08-24

Decoding is slow. The wait after submitting a prompt until the first token appears is bearable, but from there the tokens crawl out one at a time. Look it up and you find speculative decoding, a technique that promises a speedup from adding one small model. And every source says the same thing about it: the quality is not compromised.

This article was written to unpack that last sentence.

Speculative decoding itself isn't complex. It involves having a small, fast draft model predict several tokens ahead, then having a larger, slower target model verify those predictions all at once. If the draft model is correct, multiple tokens are confirmed in a single call. If it's incorrect, the system discards the prediction from that point and the target model writes the output itself. That's the core of the mechanism.

The hard part is that several methods travel under the same name and promise guarantees at different levels. One proves as a theorem that the output probability distribution does not change at all. Others state outright in their own papers that they do not require the distributions to match, but still assert that the quality of the generated output is maintained. Both are called speculative decoding, and both are introduced as methods that do not compromise quality.

Three conclusions are worth stating up front.

First, what gets guaranteed follows from the rule that verifies the draft, not from how the draft gets made. Whether the draft model is built using a separate model, by adding a small head to the target model, or by reading the target model's own intermediate representations - this affects the speed, but not the level of the guarantee. What matters is the rule that decides whether a proposed token is accepted or rejected. Methods that utilize rejection sampling preserve the output distribution. Methods that use a threshold do not. For one and the same way of making the draft, either verification rule may be available.

Second, even with speculative decoding, you cannot guarantee that the output comes back bitwise identical to what it would have been otherwise. In fact, it moves further away from that possibility. When speculative decoding is enabled, the forward pass of the target model goes from one position to k+1 positions. A change in the shape changes the numbers. As discussed in the previous article in this series, you cannot rely on a regression test that demands bitwise identical output after introducing speculative decoding.

Third, the acceptance rate and the batch size decide whether it gets faster at all. Speculation is a technique that increases the total arithmetic, and it only pays while compute capacity that would otherwise sit idle can absorb that increase. As the number of concurrently processed requests increases, those available resources diminish, and beyond a certain point, speculation begins to negatively impact performance. Therefore, the inference engine includes a setting that allows it to reduce the draft length to zero when it becomes overloaded.

All specifications and figures presented in this article have been verified against primary sources. The verification date is August 23, 2026. The sources split by the type of claim. What a method guarantees comes from the arXiv originals, what the inference engine does comes from the official vLLM documentation, and what is available on AWS comes from the official AWS documentation. How the sources split is itself part of the subject here, and Chapter 5 takes it up directly.

Table of Contents

1. 1. Turning On Speculative Decoding Changes What You Can Claim
2. 2. Two Words to Cut and One Vocabulary to Carry Over
3. 3. What Happens Inside One Iteration
4. 4. What the Primary Sources Say Each Verification Rule Guarantees
5. 5. Quality and Distribution Are Promises at Different Levels
6. 6. The Acceptance Rate Decides the Speedup
7. 7. Where the Draft Comes From
8. 8. Speculation Buys Serial Steps With Arithmetic
9. 9. Three Paths on AWS
10. 10. Two Mechanisms Under One Feature Name on Amazon SageMaker AI
11. 11. What the Managed Draft Model Takes Away
12. 12. What the AWS Neuron Documents Actually Say
13. 13. How to Check It After You Turn It On
14. 14. A Checklist for Choosing
15. 15. Failure Modes and Anti-Patterns
16. 16. Frequently Asked Questions
17. 17. Summary
18. 18. References

1. Turning On Speculative Decoding Changes What You Can Claim

1.1 Intended Audience

This article is intended for readers who are currently serving large language models themselves and are considering speculative decoding to address slow decoding speeds.

You likely already understand that adding draft models can improve speed. However, there are three key areas you may be less familiar with. First, what factors determine how much faster the decoding will be. Second, what the phrase about quality not being compromised actually guarantees. And third, which of the paths on AWS to take, and what you give up in exchange.

This article is aimed at those involved in designing inference infrastructure, and those responsible for deciding between Amazon SageMaker AI, AWS Neuron, and self-hosting. You do not need to be able to write kernel code; this article will not include any kernel implementation code. Furthermore, it will not cover the training procedures for draft models.

1.2 What This Article Does Not Cover

The phenomenon itself, where the output changes with a fluctuating batch size even though nothing was modified, is out of scope here. That topic is covered in the previous article in this series, Reproducible LLM Inference. This article takes that as given and covers what happens when you deliberately add something yourself.

Reducing the precision of the weights is not covered either. Quantization is covered in a separate article, LLM Weight Quantization on AWS. This article will not delve into the process of quantizing draft models.

The performance features on the managed inference endpoint side are not covered either. Information on latency-optimized inference within Amazon Bedrock can be found in Amazon Bedrock Inference Throughput and Latency Optimization. The Amazon Bedrock user guide states the following about that feature:

The Latency Optimized Inference feature is in preview release for Amazon Bedrock and is subject to change.

Which acceleration techniques the managed side uses internally is not published. Therefore, the framework this article uses, which is to read the level of the guarantee off the verification rule, cannot be applied to the managed side. That it cannot be applied is itself what drives the conclusion in Chapter 13.

Whether to self-host at all is not argued here either. That decision is addressed in Self-Managed LLM Inference on Amazon EKS. This article assumes that the reader has already chosen to self-host and begins by addressing the next decisions they will need to make.

How to run the job that measures whether quality dropped after speculation went in is not covered either. Amazon Bedrock Model Evaluation Practical Guide provides that information. This article writes only what should be compared and delegates the rest.

1.3 What the Existing Articles Already Hold

This website already contains articles that mention speculative decoding. However, each one mentions it only as a configuration item, without explaining what it is. This article aims to fill that gap.

Existing Article	What That Article Holds	Relationship to This Article
Reproducible LLM Inference	Why the same input does not return the same numbers. Defines the three-level vocabulary.	This article does not redefine that vocabulary. It refers to it and uses it. It also assumes as established the causal relationship where batch size affects numerical values.
Disaggregated Prefill and Decode for LLM Serving on AWS	Separation of prefill and decode, KV transfer, and compatibility hashing. Lists the speculative method and the draft model configuration among the items the compatibility hash requires to match.	This article writes only what those items mean. It does not delve into handshake processes or transfer modes.
Self-Managed LLM Inference on Amazon EKS	Discussion of whether to self-host and serving configurations. Mentions speculative decoding in one line, as one of the features AWS Neuron carries.	This article is the one that expands that line. It does not discuss whether or not to self-host.
Amazon Bedrock Inference Throughput and Latency Optimization	Performance features on the managed side.	This article does not describe how to use these features; it only provides cross-links.
Amazon Bedrock Model Evaluation Practical Guide	How to run evaluation jobs.	This article delegates the task of determining what should be compared.
LLM Inference Resilience Patterns on AWS	Retries, circuit breakers, and stream recovery.	This article does not cover fault handling.
AWS Custom Silicon History and Timeline and AWS History and Timeline regarding Amazon SageMaker	A timeline of when various products were released.	This article does not create a timeline.
Open-Weights LLM Release History and Timeline	The lineage of open-weight models.	This article does not compare the capabilities of different models.

1.4 Verification Date and Primary Sources

This article assigns different primary sources to each type of technical claim. The verification date for all sources is August 23, 2026.

Type of Claim	Primary Source
What a method guarantees	The arXiv original, with the version stated
What an inference engine does	The official vLLM documentation
What is available on AWS	The Amazon SageMaker AI Developer Guide, the AWS Neuron documentation, and AWS What's New

This article is not a setup guide. The exact flags to enable are in the official documentation, and they can change. This article explains why those flags are necessary and provides a framework for you to determine, based on your specific situation, whether or not you should enable them.

2. Two Words to Cut and One Vocabulary to Carry Over

This chapter is short. Skip it, though, and you will read the rest of the article with a different thing in mind at three separate points. Two of these terms will be defined within this article. The remaining term is borrowed from the previous article and will be used without further definition.

2.1 The Word acceptance Carries Two Meanings

In this article, acceptance means the target model accepting a token proposed by the draft model. The acceptance rate is the proportion of proposed tokens that were accepted, and the acceptance length is the number of tokens committed in a single verification step.

It is an entirely different thing from the acceptance of acceptance testing in software engineering. From here on, acceptance without further qualification means the first one.

2.2 The Word lossless Carries Two Meanings Too

The literature on speculative decoding frequently uses the term lossless. In this context, lossless does not refer to compression. It means that the output probability distribution remains unchanged.

In the context of data compression, lossless refers to the ability to perfectly reconstruct the original byte sequence. The same term is also used when referring to reducing the size of a model's weights. In this article, lossless refers to the distribution, not the size.

And the most important point to remember is this: When you read materials that use the term lossless, you must verify what it refers to in each specific document. Chapters 4 and 5 are designed to help you do that.

2.3 The Three-Level Vocabulary Carries Over Unchanged

The previous article, Reproducible LLM Inference, names the three levels at which two systems can be called the same. This article uses those levels without redefining them.

Level	Name	Claim
Level 1	bitwise identity	The same byte sequence is returned for the same request. String comparisons match.
Level 2	distribution equivalence	The output is drawn from the same probability distribution. Individual outputs may differ.
Level 3	quality parity	The results are indistinguishable when aggregated. The distribution may differ.

These three levels are nested. If Level 1 is satisfied, then Level 2 is also satisfied. If Level 2 is satisfied, then Level 3 is also satisfied. The reverse is not necessarily true.

This article deals with Levels 2 and 3. The previous article dealt with Level 1, and this one takes up which of the claims survive and which disappear once speculative decoding is turned on.

3. What Happens Inside One Iteration

3.1 How Many Times the Models Get Called

A single iteration of speculative decoding proceeds as follows:

First, the draft model generates k candidate tokens. That part is serial. The draft model itself generates autoregressively, one token at a time, so producing k tokens takes k calls to it.

Next, the target model is called once. At this point, instead of a single position, it receives k+1 positions. The target model then calculates, in parallel, the probability distribution for the next token at each position, based on the sequence up to that point.

Then the verification rule runs. It walks the draft proposals from the front and stops at the first rejection. Whatever prefix it accepted becomes committed, and at the rejected position the target model puts a token of its own. If it accepted everything, the target model can draw one more token from the distribution it computed for the last position.

The Amazon SageMaker AI Developer Guide describes this process as follows:

The draft model generates candidate tokens, which are then validated by the larger but slower
target model. At each iteration, the draft model generates multiple candidate tokens. The
target model verifies the tokens, and if it finds that a particular token is not acceptable, it
rejects the token and regenerates it. So, the target model both verifies tokens and generates a
small amount of them.

3.2 Why Verification Is Cheaper Than Generation

Generating a single position's output and verifying k+1 positions can take roughly the same amount of time. This is why speculative decoding works. The reason is that during decoding, memory bandwidth rather than computation limits the target model's forward pass.

To generate a single token, the model needs to read all of its weights from memory. The actual computations performed with those weights, when processing a single position, are minimal. Because reading the weights only needs to happen once, even when processing k+1 positions, the time taken doesn't increase significantly as the number of positions increases. The AWS Neuron design document for the vLLM Neuron framework puts it this way:

In standard autoregressive decoding, the target model generates one token per forward pass.
Each pass is memory-bandwidth-bound on accelerators - the model weights must be loaded from HBM
for every single token. Speculative decoding amortizes this cost by verifying multiple draft
tokens in a single target forward pass.

It's important to remember the underlying assumption here: As long as memory bandwidth is the limit, adding positions barely adds time. This assumption breaks down the moment the limiting factor shifts from memory bandwidth to computation. Chapter 8 discusses this in detail.

3.3 What Happens at the Rejected Position

Rejection is not harmless. Every draft after the rejected position is discarded. If the rejection lands on the third draft, the fourth and everything past it disappears, even though the target model has already finished computing them.

The draft model does not get back the time it spent generating them, and the target model does not get back the arithmetic it spent verifying them. A lower acceptance rate means a greater number of drafts are discarded.

3.4 The Saving Comes From Fewer Serial Steps

Speculative decoding reduces the number of times the target model is called sequentially to produce a single response. In standard decoding, the target model is called once for each output token. With speculation, a single call can determine an average of multiple tokens.

Two things do not go down. One is the time per call to the target model, which in fact rises slightly as k grows. The other is the total arithmetic, which certainly rises.

These two are what define the character of speculative decoding. It buys serial steps with arithmetic, so it only pays where arithmetic is going spare.

4. What the Primary Sources Say Each Verification Rule Guarantees

This chapter is central to this article.

As the previous chapters showed, the structure of a single iteration is the same regardless of the method. The draft proposes, the target verifies in parallel, and the accepted prefix is committed. What differs between the methods is the rules that determine whether or not to accept. And it is that rule that fixes the level of the guarantee.

4.1 Matching Under Greedy Decoding

The simplest rule is to accept a draft proposal only when the proposed token matches the target's most probable token. It applies when greedy decoding is in use, and it is called a greedy match.

Under this rule, the sequence of tokens that is finalized will be the same sequence that would result from greedy decoding without any speculation. A draft proposal only gets through when it matches the target's most probable token, and where it does not, the target puts its own most probable token there.

The official vLLM documentation includes this as a verification item.

Greedy Sampling Equality: Confirms that greedy sampling with speculative decoding matches
greedy sampling without it. This verifies that vLLM's speculative decoding framework, when
integrated with the vLLM forward pass and the vLLM rejection sampler, provides a lossless
guarantee.

The AWS Neuron documentation, specifically the design document for the vLLM Neuron framework, states the same thing about greedy decoding just as strongly. This document focuses on Trainium2 and Trainium3. Section 12.4 takes up the scope.

This guarantees the output distribution is identical to the target model alone (for greedy
decoding, outputs are bit-exact).

This is the highest-level claim of three. At the same time, there is a condition on the range in which this claim holds. Section 4.7 takes up that condition.

Hydra's arXiv:2402.05109, in its first version, gives its reason for skipping quality evaluation altogether in experiments that use the greedy verification rule.

Unless otherwise specified, experiments are conducted using the greedy verification criterion;
since there is no stochasticity in the greedy sampling procedure, we don't report the quality
of generations as they are identical to the base model.

4.2 What Modified Rejection Sampling Guarantees

When not using greedy decoding - that is, when sampling probabilistically with a temperature greater than zero - the rules become more complex.

The original paper by Leviathan et al. (arXiv:2211.17192) proposes a procedure that compares the distribution of the draft model with the distribution of the target model, accepting samples based on the ratio of these distributions and, if rejected, drawing new samples from an adjusted distribution. The paper provides a proof in the appendix demonstrating that this procedure preserves the distribution.

We will now show that for any distributions p(x) and q(x), the tokens sampled via speculative
sampling from p(x) and q(x) are distributed identically to those sampled from p(x) alone.

The claim in the body of the paper is stated just as strongly.

the output distribution is guaranteed to stay the same

Chen et al. at DeepMind (arXiv:2302.01318) call the same procedure modified rejection sampling and explicitly bound the scope of the guarantee.

This is combined with a novel modified rejection sampling scheme which preserves the
distribution of the target model within hardware numerics.

Speculative sampling does not require making any modifications to the target language model's
parameters or architecture, is provably lossless within numerics

The qualifier within hardware numerics is decisive for this article. Section 4.7 takes it up.

Everything so far is a claim at Level 2 of the three, distribution equivalence. It does not say that individual outputs match. It says that they are drawn from the same distribution.

4.3 Lenience - The Opt-Out the Original Paper Provides

There are options for relaxing distribution preservation, even without changing the core method. The paper by Leviathan et al. carries an appendix section called lenience, which describes deliberately loosening distribution preservation to buy speed.

A strong property of Algorithm 1 is that the output distribution is guaranteed to remain
unchanged. That said, if we're willing to allow some changes, with nice guarantees, we can get
further inference speed improvement.

Specifically, the paper describes multiplying the probability on the draft side by a factor less than 1 before comparing it to the target side. The smaller the factor, the more likely the draft will be accepted, increasing the acceptance rate and thus improving speed. However, this does alter the output distribution. The paper also notes that there are limits to how much this distribution can change.

And the paper explicitly records that none of the measurements in its main body use this relaxation at all.

Note that the results in this paper except for this section use the strictest version of
Algorithm 1 and don't allow lenience of any kind.

The significance of this statement is considerable. The same algorithm, presented in the same paper, can either preserve the distribution or not, depending on the configuration. Therefore, simply verifying which method is being used is not sufficient to determine whether the distribution is being preserved. It is necessary to examine the configuration as well.

4.4 Typical Acceptance Does Not Require the Distributions to Match

The Medusa paper (arXiv:2401.10774) proposes an alternative rule, starting from empirical observations that using rejection sampling during probabilistic sampling can lead to a decrease in acceptance rates. The proposed rule, called typical acceptance, accepts a token when the probability the target model assigns to it clears a threshold. The threshold is the smaller of two values: a fixed value and a value dependent on entropy.

The paper explicitly states that this rule does not require distribution alignment.

We ascertain that it is typically unnecessary to match the distribution of the original model.
Thus, we propose employing a typical acceptance scheme to select plausible candidates rather
than using rejection sampling.

In the related work section of the same paper, the difference from existing truncation sampling methods is put this way.

However, we diverge because we do not insist on an exact correspondence between the output and
language model distribution. This deviation allows us to facilitate more diverse yet
high-quality outputs, achieving greater efficiency without compromising the integrity of the
generated text.

A third party has named this rule outright. Hydra (arXiv:2402.05109), in its second version, refers to this rule as follows.

the non-greedy, non-distribution-preserving typical acceptance verification criterion
introduced by Cai et al. 2024

There are two key points to consider.

First, not requiring the distributions to match is not the same thing as the quality degrading. What the Medusa paper claims is the first, not the second. It in fact argues that the quality of the output holds up.

Second, whether or not the quality can be maintained is a matter of empirical evaluation, and the results of those evaluations are not uniform. Hydra's second version, using this rule, measured the quality of the generated output with a judge model and reported the following.

While neither Medusa nor Hydra is able to achieve the same quality as random sampling from the
base model for any of the posterior thresholds considered

The paper further reports that only Hydra++, with a posterior threshold set to 0.15, achieved a comparable level of output quality to direct sampling from the original model.

Therefore, whether or not using typical acceptance results in output quality equivalent to the original model depends on the quality of the draft. Under one and the same verification rule, the quality matched where the heads were good and fell short where they were not. This is a worked example of Level 3, quality parity, being something you measure rather than something you are promised.

4.5 Which Rule the EAGLE Family Uses

The EAGLE papers state repeatedly that they use no relaxation.

The initial paper, arXiv:2401.15077, states:

EAGLE also employs this sampling approach, ensuring that the distribution of the generated text
remains unchanged for both the greedy and non-greedy settings.

In contrast, EAGLE does not employ any relaxations and maintains the output distribution of the LLM unchanged.

The same paper clearly articulates the logic that if distribution preservation holds, quality evaluation is unnecessary.

Acceleration of EAGLE theoretically guarantees the preservation of the original LLMs' output
distribution. Consequently, evaluating the quality of EAGLE's generated results is both
unnecessary and meaningless.

This one sentence explains why the three levels form a ladder. If Level 2 holds, Level 3 comes along with it, so there is nothing to measure. If Level 2 does not hold, Level 3 cannot be known without measuring. The Hydra measurement in Section 4.4 is exactly that measurement.

EAGLE-2, arXiv:2406.16858, takes a similar stance.

EAGLE-2 neither fine-tunes the original LLM nor relaxes acceptance conditions, making it a
lossless acceleration method. Therefore, we do not evaluate the generation quality

The EAGLE papers also state, every time, why they leave Medusa out of the comparison.

Methods like Medusa relax acceptance conditions under non-greedy settings, which do not
guarantee lossless acceleration.

Regarding EAGLE-3, arXiv:2503.01840, careful consideration is needed when interpreting the text. That paper describes speculative sampling in general as a lossless LLM acceleration technique and restricts its comparisons to methods that use no relaxation. However, the collation behind this article found no statement in the text that asserts EAGLE-3's own distribution preservation the way the first paper does. The changes in EAGLE-3 involve eliminating feature prediction and instead directly predicting tokens, as well as fusing multiple layers of features - a design modification on the draft side, rather than a change in the verification rules.

4.6 Four Rules Placed Against the Three Levels

The statements from the primary sources so far map onto the three-level vocabulary as follows.

The cells take one of three values. What the primary source states as a guarantee is written Stated as a guarantee. What it does not state is written Not stated. What has been shown by measurement but is not stated as a guarantee is written Measured, not stated.

Verification rule	Primary source	bitwise identity	distribution equivalence	quality parity
greedy match	vLLM official documentation, AWS Neuron documentation	Stated as a guarantee	Stated as a guarantee	Stated as a guarantee
Modified rejection sampling	arXiv:2211.17192, arXiv:2302.01318, arXiv:2401.15077	Not stated	Stated as a guarantee	Stated as a guarantee
Rejection sampling with lenience	Appendix of arXiv:2211.17192	Not stated	Not stated	Not stated
Typical acceptance	arXiv:2401.10774, arXiv:2402.05109 (Version 2)	Not stated	Not stated	Measured, not stated

Not stated does not mean that the property breaks. It means the primary source does not state it. Collapse that distinction and the article ends up claiming more than its sources do.

The most crucial aspect of interpreting the table is the entry Measured, not stated in the bottom right corner. Whether typical acceptance ends up with quality equivalent to the original model is not something the primary source guarantees. It was measured in an experiment. And as Section 4.4 showed, the measurement split by configuration.

4.7 The Top Level Is an Algorithmic Claim, Not a Production Assumption

There is a condition attached to the first row that has not been stated yet.

The two statements quoted in Section 4.1 both claim the top level for greedy match. That claim holds only while the numbers the target model computes are the same whether or not speculation is on. That premise does not hold.

When speculation is enabled, the number of positions passed in the forward pass of the target model changes from one to k+1. As established in the previous article Reproducible LLM Inference, the inference engine's kernel changes its splitting strategy for a reduction when the shape it processes changes. A change in the split changes the order of the additions, the rounding changes, and the logits change.

Chen et al.'s paper directly addresses this point as part of its experimental design.

Even with greedy sampling, a single token deviating due to numerics could result in two
sequences diverging wildly. Since pseudo-random seeds are processed differently between ArS and
SpS, and because the different computation graphs lead to different numerics, we cannot not
expect identical outputs. However, we expect the samples to come from the same distribution
within numerics and we empirically verify this by evaluating these benchmarks.

The original text explicitly states that you should not expect the same output when comparing speculative and non-speculative runs. What you can expect is only that the outputs come from the same distribution, within hardware numerics.

The official vLLM documentation also states the same principle, broken down into three separate points.

Theoretical Losslessness - Speculative decoding sampling is theoretically lossless up to the
precision limits of hardware numerics. Floating-point errors might cause slight variations in
output distributions

The same page names batch size explicitly among the things that can move the output distribution.

Batch Size and Numerical Stability: Changes in batch size may cause variations in logprobs and
output probabilities, potentially due to non-deterministic behavior in batched operations or
numerical instability.

The first row of the table in Section 4.6 is therefore a claim at the level of the algorithm. At the level of the implementation, both the primary sources and the engine documentation attach caveats. This article handles the distinction as follows. Claims at the level of the algorithm are quoted as the primary sources state them, and production serving, where the batch size moves with the load, does not assume the top level. Speculative decoding is not a way to get bitwise identical output back.

5. Quality and Distribution Are Promises at Different Levels

5.1 The Words the AWS Documents Use

The Amazon SageMaker AI Developer Guide describes speculative decoding as follows:

Speculative decoding is a technique to speed up the decoding process of large LLMs. It
optimizes models for latency without compromising the quality of the generated text.

The AWS What's New announcement that introduced EAGLE-based speculative decoding uses the same word.

maintaining the same output quality while dramatically increasing throughput

Both say quality. That is the word for Level 3, and the promise it makes is that the two systems are indistinguishable once the metrics are aggregated.

In contrast, the documentation for AWS Neuron, specifically the design document for the vLLM Neuron framework, uses a different term.

Speculative decoding is a technique for accelerating autoregressive LLM inference without
changing the output distribution.

This one says distribution. That is Level 2. And for greedy decoding, the same document goes all the way to Level 1.

This guarantees the output distribution is identical to the target model alone (for greedy
decoding, outputs are bit-exact).

Across the AWS documents, the level of the promise spans all three. The passages quoted above are what the collation behind this article turned up. They are not an exhaustive survey of what AWS publishes.

5.2 The vLLM Documentation Splits It Into Three

The official vLLM documentation takes this problem head on, in a section called Lossless guarantees of Speculative Decoding. That section breaks the claim of losslessness into three.

The first is the theoretical level. It asserts that the algorithm preserves the distribution, with an explicit qualifier that this holds only up to the precision of the hardware numerics.

The second is the implementation level. It claims that two tests verify the vLLM implementation against the algorithm, one for the convergence of the rejection sampler and one for equality under greedy sampling.

The third is the operational level, and this is the only one stated in the negative.

vLLM Logprob Stability - vLLM does not currently guarantee stable token log probabilities
(logprobs). This can result in different outputs for the same request across runs.

The same page separates what it guarantees from what it does not, at that granularity. This way of reading is exactly what this article is arguing for.

5.3 Each One Is Correct About What It Describes

One point deserves care here. None of this says that the AWS documents are wrong.

The Amazon SageMaker AI Developer Guide introduces a managed inference optimization feature. What a reader wants to know is whether the text this feature produces gets worse, not whether the probability distribution is mathematically preserved. quality is the right granularity of answer to that question.

The AWS Neuron design document explains how one particular implementation behaves, for engineers. Writing distribution there, and bit-exact for greedy decoding, is the right granularity too.

The problem is that two descriptions written at different granularities get summarized in the same words and circulate that way. Take home only the sentence that quality is not compromised, and a reader will read it as Level 1 or Level 2 without noticing. Then they build an exact-match regression test, it fails, and they have no idea why.

So what this article is saying is not that AWS is wrong. It is that quality, distribution, and bitwise are promises at different levels, and which one is being promised has to be checked document by document.

5.4 How to Judge a Document for Yourself

Which level a document is claiming can be worked out in the following order.

First, identify the subject. Who is making the claim? Is it the service provider, the designer of the method, or the developer of the implementation? The three write at different granularities.

Second, check the word. quality is Level 3. distribution is Level 2. identical and bit-exact are Level 1.

Third, read the qualifier to the end. If a qualifier such as within hardware numerics or up to the precision limits of hardware numerics is attached, the claim sits at the level of the algorithm and not at the level of the implementation.

Fourth, go down to the verification rule. Rather than which method the document uses, look at which rule decides acceptance. The table in Section 4.6 is that mapping.

6. The Acceptance Rate Decides the Speedup

6.1 What the Acceptance Rate Is

The acceptance rate is the proportion of the tokens proposed by the draft model that pass verification. It is often denoted by the symbol α.

The paper by Leviathan et al. demonstrates that α directly reflects the similarity between the distributions of the draft model and the target model. If the two distributions perfectly align, α will be 1; if they do not overlap at all, α will be 0. In essence, the acceptance rate is not an adjustable parameter, but rather a result of how the draft model is selected.

6.2 Expected Tokens per Iteration

Given a draft length of k, the number of tokens finalized in a single iteration ranges from a minimum of 1 to a maximum of k+1. The minimum value of 1 occurs when the initial draft is rejected; even then the target places one token of its own, so no iteration ends up empty-handed.

While the original paper uses γ to represent draft length, this article uses k for consistency with other materials. The Leviathan paper, under the assumption that acceptance at each position is independent, derives the expected number of tokens per iteration as (1 - α^(k+1)) / (1 - α). The same paper gives, as a theorem, the wall-clock improvement factor (1 - α^(k+1)) / ((1 - α)(kc + 1)), where c is the cost of one draft model call relative to one target model call. From this, the paper concludes that if α > c, then there exists a value of k that results in improvement.

This formula expresses a straightforward concept. If the draft model is sufficiently inexpensive and accurate, it provides a benefit. However, if it is inexpensive but inaccurate, or accurate but expensive, it does not provide a benefit.

6.3 Longer Drafts Stop Paying

Increasing k raises the theoretical maximum number of tokens that can be confirmed in a single iteration. However, in practice, performance degrades once a certain point is reached.

The reason for this is evident in the formula. The expected accepted length grows as a power of α, so each extra unit of k returns less than the last. Conversely, the number of times the draft model is called increases proportionally with k. The optimal value of k is the point where the diminishing returns and the proportional increase intersect.

Chen et al.'s paper empirically demonstrates this.

Note that as K increases, the overall speedup plateaus or even regresses, with XSum being
optimal at K=3. The variance consistently increases with K.

A frequently overlooked aspect is that variance consistently increases with k. While the average may improve, the tail of slower responses grows longer. If you are making decisions based on response time percentiles, you should not rely solely on the average.

6.4 What Gets Thrown Away on Rejection

When the acceptance rate declines, two things are wasted.

First, there is the time the draft model spent. The time it used to generate the drafts after the rejected position is gone.

Second, there's the computation performed on the target model. Because the target model has already computed all k+1 positions in parallel, the work it did on the rejected positions was done only to be thrown away.

This second point leads to Chapter 8. It's acceptable to discard computations only when there is excess computational capacity available.

6.5 The Acceptance Rate Moves With the Domain

Acceptance rates, even when using the same draft model and target model combination, can change based on the nature of the input.

Chen et al. devote a section to this, reporting that the acceptance rate differs between summarization and code generation, and that code generation showed the larger acceleration. Intuitively, the draft is more likely to be right where the syntax pins the next token down to almost one choice.

Therefore, acceleration rates measured on one workload should not be applied to another workload. It is necessary to re-evaluate using input that closely resembles real-world traffic. This is also why Amazon SageMaker AI lets you train the draft side on your own data. Chapter 10 takes that up.

7. Where the Draft Comes From

The previous chapters established that the verification rule decides the level of the guarantee and the acceptance rate decides the speed. This chapter covers the choice that moves the acceptance rate, which is how the draft gets made.

7.1 Running a Separate Model

The simplest approach is to use a smaller model from the same series as a draft model. It ends up as a separate process from the target, with its own weights and its own compiled artifact.

The key advantage of this method is that it requires no modifications to the target model. The disadvantage is that it necessitates an additional model, which consumes memory and increases startup time.

There is a direction to how the size is chosen. Recall the condition α > c from Section 6.2. A larger draft raises c, so it is only worth it if the gain in α outruns that. Chen et al. pair a 4 billion parameter draft model with a 70 billion parameter target model.

7.2 Using a Model With a Different Vocabulary

By default, the draft and target models have to share the same vocabulary. The proposed token IDs get compared against the distribution on the target side, so the comparison falls apart if the IDs mean different things.

vLLM provides a mechanism to relax this requirement.

By default, vLLM requires the draft and target models to share the same vocabulary. Setting
use_heterogeneous_vocab: true enables the Token-Level Intersection (TLI) algorithm, which
allows draft models from a different model family with a different tokenizer.

However, there is a condition. The documentation states that this setting requires greedy sampling on the draft side, and that probabilistic draft sampling is not supported yet. Turning one setting on takes another setting off the table, and the same shape shows up again in Chapter 11.

7.3 Adding Heads to the Target Model

Instead of incorporating another model, Medusa's approach involves attaching multiple small prediction heads to the final layer of the target model. The first head predicts the token after next, the second the one after that, and so on, all in parallel.

The advantage is that the forward pass for drafting effectively goes away. By running the target model once, the outputs from the heads are obtained as a byproduct.

The disadvantage is that the heads have to be trained. And there is a second one. The early heads predict independently of one another, so the i-th head does not know what heads 1 through i-1 predicted. Hydra makes that point and shows that making the heads sequentially dependent lengthens the acceptance length.

7.4 Reusing the Target Model's Features

EAGLE neither adds heads nor stands up a separate model. It feeds the target model's own internal representations into the draft side.

The first EAGLE takes features from the target model's upper layers and predicts forward autoregressively at the feature level. EAGLE-3 modifies this design.

we introduce EAGLE-3, which abandons feature prediction in favor of direct token prediction and
replaces reliance on top-layer features with multi-layer feature fusion via a technique named
training-time test.

The AWS Neuron design document for the vLLM Neuron framework describes the EAGLE-3 draft model this way:

A single decoder layer (not a full copy of the target model)

This approach differs from the previous two in that the draft model looks inside the target model. Draft and target are tightly coupled, so replacing the target means rebuilding the draft as well.

7.5 When the Model Already Has It Built In

Some models inherently possess the ability to predict multiple tokens simultaneously. vLLM treats this as MTP.

MTP is a speculative decoding method where the target model includes native multi-token
prediction capability. Unlike draft-model-based methods, you do not need to provide a separate
draft model.

MTP is short for Multi-Token Prediction. The EAGLE/MTP-style speculative method and draft-model configuration that Disaggregated Prefill and Decode for LLM Serving on AWS lists as a must-match item for the compatibility hash refers to what this section and the previous one describe. In a deployment that splits prefill and decode across separate pools, both sides have to agree on this setting. Why they have to agree, and how to make them agree, is held by that article.

7.6 Methods That Run No Model at All

Drafts don't necessarily need to come from a model. vLLM offers two methods: one that identifies potential candidates by looking for the same sequence in the input, and another that utilizes a suffix tree constructed from previous requests.

These approaches don't require additional models, making them easier to implement. vLLM's documentation notes that while using a model generally provides greater latency reduction, the non-model approaches offer more modest improvements.

What matters is that none of this changes the verification rule. Wherever the draft comes from, what decides acceptance is the target model's distribution. So the table in Chapter 4 is indexed by verification rule, not by how the draft gets made.

7.7 How to Choose

How the draft gets made	What it needs in addition	Where it fits
A separate model	The weights and the runtime resources for a small model	A suitable small model already exists in the same family
Adding heads	Training the heads	You are not going to replace the target, and you can pay the training cost once
Reusing features	Training the draft side, and coupling it to the target	You want the acceptance rate as high as it goes, and the target is fixed
Built into the model	None	The model you are already using supports it out of the box
A method that needs no model	None	You want to see first whether it helps at all. The input carries a lot of repetition

Whichever one you choose, the conclusion of Chapter 4 does not move. The verification rule decides the level of the guarantee. How the draft gets made moves the acceptance rate, which is to say the speed.

8. Speculation Buys Serial Steps With Arithmetic

8.1 The Arithmetic Goes Up

Speculative decoding is not a technique for saving arithmetic. It is a technique that spends more arithmetic to cut the number of serial steps.

The paper by Leviathan et al. formalizes this point as a theorem. Because the target model processes k+1 positions in parallel during each iteration, the number of operations executed simultaneously increases by a factor of k+1. Furthermore, since operations for rejected positions are discarded, the total number of operations per generated token is greater than in a non-speculative approach.

The reason speculative decoding can appear faster is that there are situations where this increased computational load can be executed without significantly increasing response time.

8.2 At Larger Batches Verification Becomes Compute-Bound

Running it without adding much to the response time only works while the arithmetic units are idle. As seen in Section 3.2, the forward pass during decoding is bottlenecked by memory bandwidth. When processing only one request at a time, the processing units are mostly idle.

As the number of concurrently processed requests increases, the situation changes. Chen et al.'s paper clearly outlines the scope of this approach.

complements many existing techniques for reducing latency in the small batch size setting

Hydra, version 2, gives this boundary a section of its own.

Speculative decoding techniques are typically evaluated in the batch-size-1 setting. When there
is only a single sequence in the batch, decoding is extremely memory bandwidth-bound, and large
numbers of FLOPs can be consumed in the verification step of speculative decoding "for free"
without significantly increasing the latency per decoding step. However, at larger batch sizes
it is easier for verification to become compute-bound, and the number of tokens verified per
sequence per step must be more tightly controlled to avoid saturating the GPU's compute
capacity and entering the regime where speculative decoding becomes unprofitable.

The same section also reports measurements. Hydra++, the tuned variant the paper proposes, improved throughput by 2.70x over standard decoding at batch size 1. In the same configuration at batch size 8, the improvement had shrunk to 1.63x.

These numbers come from the Hydra paper. They are not measurements taken for this site. The model was a 7 billion parameter base model, the rule was greedy verification, and the measurements were taken on a single GPU on MT-Bench. Rather than focusing on the absolute numbers, the key takeaway is that increasing the batch size to eight resulted in a reduction in the improvement.

8.3 Engines Respond by Shrinking the Draft Length

The inference engine is aware of this problem and has mechanisms to handle it. The vLLM documentation lays out the cause as follows.

SD methods need to verify K tokens for each sequence during decoding. As BS increases, the
effective BS becomes BS*K which increases the compute requirement during verification. When
this BS*K goes beyond a critical BS then SD negatively impacts the decode speed (TPOT).

The remedy is to switch the draft length per concurrency band. The example configuration in the same document takes a larger k while concurrency is low, steps it down as concurrency rises, and sets k to zero from a certain band onward. Zero means that no speculation is performed at all within that band.

The engine gives you a way to have it switch speculation off by itself once things get busy. That is the most direct evidence there is that speculative decoding is not always faster.

The page on vLLM's speculative decoding explicitly states its scope from the beginning.

This document shows how to use Speculative Decoding with vLLM to reduce inter-token latency
under medium-to-low QPS (queries per second), memory-bound workloads.

8.4 The Batch Size Is Not Yours to Fix

This is where the previous article connects.

In inference servers that perform continuous batching, the batch size is automatically determined by the load. Even if your individual request is the same, the size of the batch it's included in can change depending on the other requests arriving simultaneously.

So whether speculative decoding pays changes with the hour of the day. It may be effective during quiet periods, but ineffective, or even detrimental, during busy periods. The setting in Section 8.3 is the remedy for that.

And for the same reason, the output with speculation enabled is not stable. Changes in batch size can lead to changes in numerical values, which in turn can affect the positions where the output is accepted. Consequently, the output itself may change. As Section 4.7 showed, this property was there before speculation went in, and speculation does not fix it.

9. Three Paths on AWS

9.1 The Three Paths

There are three paths for using speculative decoding on AWS.

The first is Amazon SageMaker AI inference optimization. You run an optimization job to produce an artifact, then deploy that artifact to an endpoint. This approach allows you to delegate the preparation of the draft to the service.

The second is AWS Neuron. This is an SDK for running models on Inferentia and Trainium, and it includes speculative decoding as a library feature. The configuration is determined during the compilation process.

The third is a self-managed inference engine. You stand up vLLM on your own GPUs and turn speculation on in its configuration.

⚠ These three are the paths you choose. The Developer Guide now splits inference optimization into two approaches and places the optimization job on the manual one, describing the other as the approach it recommends for most customers. That other approach states that it applies goal-aligned optimizations such as speculative decoding for throughput. What you can and cannot establish about a path you did not choose is held by LLM Weight Quantization on AWS.

9.2 What Stays in Your Hands

The three differences aren't about speed; they're about what you can decide for yourself.

What you decide	Amazon SageMaker AI inference optimization	AWS Neuron	Self-managed vLLM
Where the draft comes from	A service-provided draft model, your own draft model, or EAGLE heads trained on your data	You supply it	You supply it, or use a method that needs no model
Which methods you may use	The ones the documents describe	The ones the documents describe for the stack you use	Everything the engine supports
Switching the verification rule	Not documented	Not documented	Follows the method you pick
Draft length that follows load	Not documented	Not documented	Set per concurrency band, all the way down to zero
Seeing the acceptance rate	The optimization job reports latency and throughput, not this	Not documented	Per request and aggregated
Deployment options you keep	Cut down if you pick the service-provided draft model. Chapter 11	Yours	Yours

The most important row in this table is the last one. Picking the service-provided draft model changes where you are allowed to deploy the result. Chapter 11 discusses this in detail.

10. Two Mechanisms Under One Feature Name on Amazon SageMaker AI

10.1 The Mechanism the Developer Guide Describes

The inference optimization page of the Amazon SageMaker AI Developer Guide lists speculative decoding as one of the optimization techniques and gives the explanation quoted in Section 3.1. What it describes there is the approach that runs a separate draft model.

The same page also states the following regarding the source of the draft model:

SageMaker AI offers a pre-built draft model that you can use, so you don't have to build your
own. If you prefer to use your own custom draft model, SageMaker AI also supports this option.

The page detailing how to create an optimization job lists three options available in the console. They are using the draft model the service provides, picking a draft model out of the SageMaker JumpStart catalog, and pointing at your own draft model by its Amazon S3 location.

If you pick the service-provided draft model, enabling network isolation is mandatory. The same page explicitly notes this as a requirement.

10.2 The Value the API Accepts

Here, the Developer Guide and the API Reference contain conflicting information.

The API for optimization jobs accepts an array called OptimizationConfigs. The elements of that array form a union, and you specify one of four things: compilation, quantization, sharding, or speculative decoding. When examining the settings for speculative decoding, only one value is accepted.

Technique
    The speculative decoding technique to apply during model optimization.
    Type: String
    Valid Values: EAGLE
    Required: Yes

Therefore, according to the API, the optimization job can only utilize the EAGLE technique for speculative decoding. The method described in the Developer Guide, which involves using pre-built draft models, is not represented within this enumerated list.

This discrepancy is not an error. It means two different mechanisms sit under the same feature name. The optimization job page in the Developer Guide provides instructions for specifying draft models using the SDK, as well as instructions for specifying EAGLE using the lower-level API.

When designing your implementation, you must first decide which mechanism you intend to use. The decision is made as follows:

What you want	What you specify	In scope for Chapter 11
The service-provided draft model	Set ModelProvider in the SDK's speculative_decoding_config	Yes
A draft model you supply	Same setting, with the Amazon S3 location in ModelSource	No
EAGLE heads trained on your own data	Set Technique to EAGLE in ModelSpeculativeDecodingConfig and give it a TrainingDataSource	No

10.3 What the EAGLE Optimization Job Does

An optimization job with EAGLE specified does not convert the model. It trains the prediction heads on the draft side. Give it training data and it produces heads fitted to that data.

The AWS What's New announcement introduced this capability on November 25, 2025.

SageMaker AI automatically selects between EAGLE 2 and EAGLE 3 based on your model
architecture, and provides built-in optimization jobs that use either curated datasets or your
own application data to train specialized prediction heads.

Which of EAGLE 2 and EAGLE 3 gets used follows from the model architecture. The Developer Guide's supported models reference carries the mapping.

The format of the training data has specific requirements. The API Reference lists three supported formats, and also states that requests captured from your own endpoint can be used as they are.

The data must be formatted as ShareGPT, OpenAI Completions or OpenAI Chat Completions. The
input can also be unencrypted captured data from a SageMaker endpoint as long as the endpoint
uses one of the above formats.

As Section 6.5 showed, the acceptance rate moves with the nature of the input. The design that allows you to train the prediction head with your production traffic is a direct response to this characteristic.

10.4 Do Not Keep Your Own List of Supported Models

The Developer Guide contains a reference list of supported models, presented in two separate tables.

The first is model by model and gives four attributes: the supported quantization formats, whether speculative decoding is available, whether fast loading is available, and the libraries used for compilation. A dedicated column indicates whether speculative decoding is supported.

The second table lists architectures and specifies whether each architecture is compatible with EAGLE 2 or EAGLE 3.

This article does not duplicate these tables. What is supported moves faster than this article can. Copying them into a design document is not a good idea either. The only factors to consider are the following three points:

First, the model-by-model table and the architecture-by-architecture table belong to different mechanisms. The first is about the draft-model approach and the second about the EAGLE approach.

Second, the model-by-model table carries rows that say outright that speculative decoding is not available. Whether it is available splits by model.

Third, always verify this information using the official documentation. The verification date for this article is August 23, 2026, and it does not reflect any changes made after that date.

11. What the Managed Draft Model Takes Away

This chapter forms another central point of this article.

Pick the draft model that Amazon SageMaker AI provides, and requirements and restrictions attach to the model you optimize with it. This article calls that option the managed draft model. The Developer Guide gathers all of it into one section and splits it three ways, into requirements, restrictions, and the environment variables you can still set.

11.1 Requirements

There are four requirements that must be met.

First, you have to use a model that SageMaker JumpStart provides. It is not available for an arbitrary model.

Second, you have to enable network isolation for the model deployment. Containers with network isolation enabled cannot send any outbound traffic.

Third, if you deploy to a large model inference container, there is a floor on the DJLServing container version. The Developer Guide specifies a version of 0.28.0 or higher.

Fourth, if you fine-tune the JumpStart model, the weights have to be in the safetensors format.

11.2 Restrictions

There are eight things that will not be possible. The Developer Guide lists them as follows:

First, you cannot use the model in a local test environment built with local mode. That takes away verifying it on your own machine.

Second, you cannot reach the model container through the AWS Systems Manager Agent. That removes the option of inspecting the container's processes from a shell.

Third, you cannot configure a core dump for when the process crashes.

Fourth, you cannot deploy it to multi-model endpoints, multi-container endpoints, or endpoints that host inference components. That rules out putting several models on one endpoint.

Fifth, you cannot create a model package. That takes away the artifact you would publish on AWS Marketplace.

Sixth, you cannot use your own inference code in the model container.

Seventh, you cannot use a requirements.txt in the model container. That removes the way to declare extra package dependencies.

Eighth, you cannot enable the Hugging Face parameter trust_remote_code.

11.3 The Environment Variables You Can Still Set

Environment variables are limited to the following three categories. The Developer Guide states:

You can configure the container only with the following environment variables:

The first category consists of environment variables common to containers used for large model inference.

The second category includes environment variables common to packages provided through Hugging Face Hub repositories.

The third category encompasses environment variables common to PyTorch and CUDA.

No other environment variables can be configured.

11.4 Requirements and Restrictions Are Not the Same Thing

Requirements and restrictions are different in kind, and they mean different things for a design.

A requirement is a condition that must be met before a feature can be used. Fail one and you cannot use the feature at all. The question of whether you can use it is settled right there.

A restriction is something that stops being available as a result of enabling the feature. What you could pick before you turned it on, you can no longer pick the moment you do. This is the kind you find out about late, and finding out late means redesigning.

The one with the largest blast radius is losing multi-model endpoints and inference components. Putting several models on one endpoint to raise utilization is a common design. Build the capacity plan on that and then run into this restriction, and the plan itself collapses.

11.5 What This Means for the Design

The managed draft model trades the work of preparing a draft model for freedom over where you deploy.

Whether this trade-off is worthwhile is determined by factors unrelated to the effectiveness of speculative decoding. It turns on whether you want to verify locally, whether you want to inspect the container, whether you want several models on one endpoint, and whether you need your own inference code. None of these factors have anything to do with speculative decoding.

So this choice is not one to weigh on speed. Weigh it on operational freedom.

The Developer Guide scopes this section to models optimized with the draft model that the service provides. The path where you supply your own draft model, and the path where you train EAGLE heads, are therefore not in scope for it. That does not mean no other constraints exist, so read the documents for whichever path you pick.

12. What the AWS Neuron Documents Actually Say

12.1 The Implementations NxD Inference Lists

For speculative decoding on AWS Neuron, the current path is NeuronX Distributed Inference. Its features configuration guide lists three implementations under the speculative decoding section.

NxD Inference supports the following speculative decoding implementations:

The first is the implementation that uses a draft model, where a separate draft model predicts the next n tokens. The two models are compiled independently.

The second is Medusa, where several small heads predict next tokens and the target model verifies all of the predictions at the same time.

The third is EAGLE, which uses additional context from the target model to improve generation efficiency. The documentation states that what it supports is EAGLE v1, with a flat draft structure.

12.2 Fused Speculation Is Not a Fourth Method

There is an AWS machine learning blog post that presents speculative decoding in NxD Inference as four kinds, with fused speculation standing alongside the others as a method of its own.

The heading structure of the documentation itself is different. Fused speculation is a sub-heading under the EAGLE section, described as a mechanism that EAGLE uses.

EAGLE speculation uses a feature called fused speculation, where the draft model and target
model are fused into a single compiled model to improve performance.

So fused speculation is an implementation device that folds draft and target into one compiled artifact, not a separate method with a different verification rule. It does not add a row to the table in Chapter 4.

This distinction matters when you write the number of methods into a design document. If you are counting methods, recount them under the headings of the documentation itself rather than reading them off the blog post.

12.3 The Constraints Attached to Each

The three implementations each have separate constraints.

For the implementation that uses a draft model, the documentation states:

Speculative decoding with a draft model currently supports only batch sizes of 1.

The same constraint applies to Medusa. On top of that, you have to use a model fine-tuned for Medusa, and you have to supply a Medusa tree.

For EAGLE, the nature of the constraints is different. It requires a draft model fine-tuned for EAGLE, and that draft model must include the weights of the target model's language model head. These weights are shared between the draft and target models.

The documentation is also blunt about what the flat draft structure implies.

Because NxD Inference uses a flat draft structure, it predicts only one token per draft
iteration. Although NxD Inference doesn't support EAGLE with a tree structure, you can train an
EAGLE checkpoint in the same way. Note that depending on your use case and dataset, you might
see lower acceptance rate with the flat draft structure compared with using a tree structure.

The official documentation says in so many words that the acceptance rate may come out lower. As Chapter 6 showed, the acceptance rate feeds straight into speed.

The publicly available EAGLE checkpoints also cannot be used as they are. The documentation explains that those checkpoints do not carry the target model's language model head weights, so you have to copy those weights across yourself before using them. The documentation provides code for this purpose.

12.4 The vLLM Neuron Framework Is a Separate Document

AWS Neuron's documentation includes another document that addresses speculative decoding. This document details the design of the vLLM Neuron framework, specifically for Trainium2 and Trainium3.

This document and the features configuration guide from Section 12.1 say different things about which methods are available.

method
    Yes
    Must be "eagle3". Currently the only supported speculation method on Neuron.

Inside the same SDK, a document that lists three implementations and a document that says only one method is supported sit side by side. Neither is wrong. Their scopes are different. The first describes using the NxD Inference library directly. The second describes using the vLLM Neuron framework.

Therefore, when researching what is available with AWS Neuron, it's necessary to first determine which stack you will be using before consulting the documentation. Build a compatibility table off one of them alone and it will not match what actually happens.

This document also goes into the verification rule. What it says about sampling on the draft side reinforces the argument of Chapter 4.

Note on draft model sampling: regardless of what sampling configuration is used for the target
model (greedy, temperature, top-k, etc.), the draft model always uses greedy sampling
internally. This is a design choice - the draft model's role is to predict the most likely
continuation, and greedy decoding is the most efficient strategy for that purpose. The
rejection sampler ensures the final output distribution matches the target model's sampling
configuration.

How the draft model samples does not affect the output distribution. What affects it is the verification rule. That agrees with the conclusion of Chapter 4.

12.5 The Older Library Is No Longer a Path

AWS Neuron gained support for speculative decoding in 2024, and according to AWS What's New, the initial implementation was announced as a feature of the Transformers NeuronX library. That library is no longer one of the paths.

As of 9/16/2025, support has ended for the Transformers NeuronX library and the corresponding
documentation has been archived. If you are still using Transformers NeuronX, Neuron recommends
that you migrate to the NxD Inference library, which provides a PyTorch-based inference
library.

If an older article or an internal design document still carries Transformers NeuronX instructions, that is a migration item. AWS Neuron provides a migration guide.

13. How to Check It After You Turn It On

13.1 Exact Match Is Not a Production Test

Do not build a production regression test on the assumption that, once speculative decoding is on, the output comes back bitwise identical to what came back before.

Strictly speaking, writing such a test is possible. vLLM carries an item in its own test suite that confirms greedy sampling gives the same result with and without speculation. But that test runs with everything pinned that a production server does not pin. In production, the batch size changes as the set of requests in flight changes.

The reason is the one from Section 4.7. Turning speculation on changes the number of positions that go through the target model's forward pass. A change in the shape changes the numbers. Once the numbers change, a different token can be selected even under greedy decoding.

And the primary source states this outright. Chen et al. write that identical outputs cannot be expected with speculation on and off, and that what can be expected is only that the samples come from the same distribution.

So the work of adding speculative decoding comes bundled with the work of redesigning the regression tests. How to change them is held by the previous article Reproducible LLM Inference. The key principles are to evaluate the final state and invariant conditions rather than strict output matching, to run the tests multiple times and compare the resulting distributions, and to set thresholds with sufficient margin above the expected variation between runs.

13.2 Where to Read the Acceptance Rate

When the speedup is not what you expected, the first thing to look at is the acceptance rate.

vLLM has a mechanism for returning acceptance statistics on a request basis. However, it doesn't collect this data by default. You need to specify the level of data collection when the server starts.

The values that come back are the mean number of tokens committed in one verification step, the fraction of proposed drafts that were accepted, and the distribution of how many were accepted. Getting the distribution back matters, because the average alone cannot tell you whether the draft occasionally lands everything or consistently lands a moderate amount.

This same mechanism also has an aggregation option, allowing you to extract statistics for the entire server.

The vLLM documentation marks this feature as experimental in so many words. It says the shape may change in a future release, so pin the version if you build monitoring that depends on these values.

Amazon SageMaker AI's EAGLE optimization jobs output measurement results upon completion. However, as mentioned in an AWS blog post, these results focus on latency and throughput improvements, not the acceptance rate. The collation behind this article found no documented way to read the acceptance rate itself.

13.3 What to Compare

There are three key aspects to compare.

First, the speed itself. However, it's crucial to measure this with a load that closely resembles a real-world scenario. As Chapter 8 showed, a result taken at a concurrency of one does not predict production.

Second, the acceptance rate. You need it to separate the case where the draft is not landing from the case where it is landing but the arithmetic has saturated.

Third, the quality. How you compare it depends on which verification rule you are using.

If the verification rule claims to preserve the distribution, measuring quality is, in principle, unnecessary. The EAGLE paper quoted in Section 4.5 sets out that logic. It is still worth measuring, but to confirm that the implementation behaves as claimed, not to doubt the method itself.

If the verification rule does not claim to preserve the distribution, measuring quality is mandatory. The Hydra measurement in Section 4.4 shows that the result can go either way in this case. Because adjusting the threshold parameter can affect both the acceptance length and the quality, it's necessary to monitor both simultaneously.

Which job to measure it with, and how, is held by Amazon Bedrock Model Evaluation Practical Guide. This article writes only what should be compared and delegates the rest.

13.4 What Is Worth Recording

Measurements taken with speculation enabled cannot be compared to those taken without it. This is because the kernel and computation graph are different. Therefore, always record the following information alongside your measurements:

Whether speculation was enabled. Where the draft came from, and its version. The draft length. The verification rule, and the threshold parameters if it has any. The concurrency at the time of measurement. The version of the target model and the version of the engine. The tensor parallel size.

Of these, the verification rule and its threshold are the ones that drop out of the record. They drop out most easily when you are running on the defaults, and then a later change to those defaults leaves you unable to work out what happened.

14. A Checklist for Choosing

Order matters. Decide on these points in sequence, from top to bottom.

First, check that decoding is what is rate-limiting. If the problem is the time to the first token, speculative decoding is not the answer. That is a question about the prefill side.

Second, check the concurrency you actually run at. If it is busy all the time and the arithmetic is saturated, speculation will not help and may hurt. That is the judgment in Chapter 8.

Third, decide which level of sameness you need. If you need bitwise identity, speculation is not an option. The top level does not hold in production serving to begin with, so what gets revisited first is the requirement. If distribution equivalence is enough, pick a verification rule that claims to preserve the distribution. If quality parity is enough, the field of options widens and an obligation to measure comes with it.

Fourth, check the verification rule. Confirm which rule the implementation you are considering uses, and whether a relaxation knob is on by default. Read it off the table in Chapter 4.

Fifth, decide where the draft comes from. Is there already a small model you can use? Can you train heads? Does the model carry it natively? Read it off the table in Chapter 7.

Sixth, choose the path. This is the first point at which Amazon SageMaker AI, AWS Neuron, and self-managed serving get compared. And if the managed draft model is on the table, read the requirements and restrictions in Chapter 11 first.

Seventh, decide how to measure. Exact match is not available. Take the three items in Chapter 13 and measure them at a concurrency close to production.

Eighth, decide what to record. Settle the list before you measure, not after. The list is in Section 13.4.

15. Failure Modes and Anti-Patterns

This chapter restates the content of the article as a list of things to avoid.

First, reading the sentence that quality is not compromised as meaning that the same output comes back. Which of the three levels is being claimed differs from document to document. Use the procedure in Section 5.4.

Second, writing that speculative decoding preserves the distribution, as one undifferentiated thing. Some verification rules claim preservation and others state that they do not require it. Use the table in Chapter 4.

Third, the reverse, writing that a method which does not require the distributions to match produces lower quality. What the primary source says is that it does not require the distributions to match, not that the quality drops. The measurement split by configuration.

Fourth, checking the name of the method and not the configuration. As Section 4.3 showed, the original algorithm ships with a knob for opting out of distribution preservation.

Fifth, leaving behind full match regression tests in production after incorporating speculative decoding. This will fail. If the reason for failure is attributed to model degradation, the troubleshooting efforts will be entirely misguided.

Sixth, using acceleration rates measured with a concurrency of one to estimate performance in production. As Chapter 8 says, the gain shrinks under load.

Seventh, entering into service level agreements based solely on average acceleration rates. As Section 6.3 quoted, lengthening the draft increases the variance consistently.

Eighth, creating your own list of supported models and methods and including it in design documentation. As Section 10.4 says, delegate to the official pages and write down the date you checked.

Ninth, using bullet points from the AWS Machine Learning blog as justification for the number of methods. As Section 12.2 says, recount them under the headings of the documentation itself.

Tenth, combining two AWS Neuron documents to create a single compatibility table. As Section 12.4 says, the two have different scopes.

Eleventh, picking the managed draft model on speed alone. As Chapter 11 says, what you lose is freedom over where you deploy.

Twelfth, applying procedures from Transformers NeuronX to new designs. As Section 12.5 says, it is no longer one of the paths.

Thirteenth, presenting vendor or research paper-published acceleration rates as your own measurements. Treat these only as properly cited references, including the source, configuration, model, and dataset.

16. Frequently Asked Questions

Q. Does enabling speculative decoding change the output?

Yes, it can. Even when using verification rules that claim distribution preservation, the shape of the forward pass for the target model can change depending on whether speculation is enabled, potentially leading to different numerical values and selected tokens. The primary source states outright that the same output cannot be expected with speculation on and off.

Q. So, does that mean the quality decreases?

No. A different output and a worse output are different things. When using verification rules that claim distribution preservation, the output is still drawn from the same probability distribution. While individual responses may vary, aggregate metrics won't reveal a difference.

Q. Does setting temperature to zero give the same output?

No. temperature reaches only the sampling layer. If the underlying numbers change, a deterministic sampler will deterministically pick a different token. The previous article covers this in detail.

Q. Does Medusa decrease quality?

Not according to the primary source. What it says is that the rule does not require the distributions to match. A third-party measurement found that, depending on the threshold and on how good the heads are, the quality either reaches the original model or does not. There is no way to settle it without measuring.

Q. How much faster is it?

It depends on the acceptance rate, the relative cost of the draft model, and the degree of concurrency. Published figures vary widely, and all represent measurements taken with specific models and configurations. This article does not recommend relying on published speed multipliers. Measure it with inputs and concurrency levels that are similar to your own traffic.

Q. Is a larger draft length better?

No. The expected accepted length grows as a power of the acceptance rate, so the returns fall off quickly, while the number of draft model calls grows in proportion. Past a certain point the whole thing gets slower, and the variance of the response time rises as well.

Q. Why does the effect diminish as the batch size increases?

Speculation is a technique that increases the total computational load, and it leverages idle computational resources to absorb that increase. As the number of requests processed concurrently increases, the available resources to absorb that increase become limited. Eventually, verification becomes the bottleneck. The inference engine has a setting that allows it to reduce the draft length to zero when the load increases.

Q. Does the draft model need to have the same vocabulary as the target model?

By default, yes. vLLM has a mechanism that takes the intersection of the two vocabularies so that a model from a different family can be used, but in that case sampling on the draft side is restricted to greedy.

Q. What do you give up by using the draft model that Amazon SageMaker AI provides?

Local-mode verification, container access through the SSM Agent, core dump configuration, deployment to multi-model endpoints, multi-container endpoints, and inference components, model package creation, custom inference code, requirements.txt, and enabling trust_remote_code will no longer be possible. The environment variables you can set are also limited to three categories.

Q. Do those restrictions apply to EAGLE optimization jobs too?

No. The Developer Guide scopes the requirements and restrictions covered in Chapter 11 to models optimized with the draft model that the service provides. The path where you supply your own draft model, and the path where you train EAGLE heads, are not in that scope. That does not mean no other constraints exist, so read the documents for whichever path you pick.

Q. How many speculation methods does AWS Neuron support?

It depends on the stack you use. The NxD Inference features configuration guide lists three implementations. The design document for the vLLM Neuron framework currently states that only one method is supported. Do not build a compatibility table off just one of them.

Q. Can speculative decoding be combined with disaggregated prefill and decode?

Yes. The design for combining them is held by an existing article. The must-match items for the compatibility hash include the speculative method and the draft model configuration, so both sides have to agree. Details can be found in Disaggregated Prefill and Decode for LLM Serving on AWS.

Q. Is speculation used with managed inference endpoints as well?

It's not possible to tell from the outside. The internal acceleration methods are not publicly disclosed. So the framework this article uses, which is to read the level of the guarantee off the verification rule, cannot be applied here. The design has to proceed on the assumption that it cannot be applied, which means comparing distributions.

17. Summary

The level of the guarantee is decided by the verification rule, not by how the draft gets made. Rules that use rejection sampling preserve the output distribution, while rules that use a threshold do not require this preservation. In some cases, you may be able to choose between different rules, even for the same draft creation method.

And the algorithm in the original paper ships with a knob for opting out of distribution preservation. Simply checking the method name is not enough. The configuration has to be checked as well before you know which level of the guarantee is still live.

quality, distribution, and bitwise are promises at different levels. The Amazon SageMaker AI Developer Guide promises quality, the AWS Neuron design document promises distribution, and for greedy decoding it goes as far as bit-exact. All of these are correct for the specific subjects they describe. A reader has to check, every time, which level is the one being promised.

The top level is claimed for greedy decoding at the level of the algorithm, and it cannot be assumed in production serving. Enabling speculation alters the shape of the forward pass for the target model, resulting in different numerical values. The primary sources state outright that the same output cannot be expected with speculation on and off. Therefore, incorporating speculation requires a corresponding change in the design of your regression tests.

Whether it becomes faster depends on the acceptance rate and batch size. The acceptance rate is a measure of how close the draft and target distributions are, and it can vary based on the input. As the batch size increases, verification becomes the bottleneck, and beyond a certain point, speculation can actually slow down the process. The inference engine provides a setting to disable speculation during periods of high load.

The three paths on AWS differ not in speed but in what you get to decide. Amazon SageMaker AI inference optimization takes the preparation of the draft off your hands, and in return, picking the draft model that the service provides carries away your freedom over where to deploy. AWS Neuron allows you to choose different methods depending on the stack you are using. With self-hosting, you retain control over method selection, dynamic adjustment of the draft length, and observation of the acceptance rate.

And whether to pick the managed draft model is not a question about speed. What you give up is local verification, inspecting the container, putting several models on one endpoint, and your own inference code. None of them has anything to do with speculative decoding, and it is exactly because they do not that finding out late collapses the design.

18. References

• Fast Inference from Transformers via Speculative Decoding - arXiv:2211.17192
• Accelerating Large Language Model Decoding with Speculative Sampling - arXiv:2302.01318
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads - arXiv:2401.10774
• Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding - arXiv:2402.05109
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty - arXiv:2401.15077
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees - arXiv:2406.16858
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test - arXiv:2503.01840
• Speculative Decoding - vLLM Documentation
• Draft Models - vLLM Documentation
• MTP (Multi-Token Prediction) - vLLM Documentation
• Dynamic Speculative Decoding - vLLM Documentation
• Per-Request Acceptance Metrics - vLLM Documentation
• Inference optimization for Amazon SageMaker AI models - Amazon SageMaker AI Developer Guide
• Create an inference optimization job - Amazon SageMaker AI Developer Guide
• Supported models reference - Amazon SageMaker AI Developer Guide
• ModelSpeculativeDecodingConfig - Amazon SageMaker AI API Reference
• CreateOptimizationJob - Amazon SageMaker AI API Reference
• Amazon SageMaker AI now supports EAGLE speculative decoding - AWS What's New
• Amazon SageMaker AI introduces EAGLE based adaptive speculative decoding to accelerate generative AI inference - AWS Machine Learning Blog
• NxD Inference Features Configuration Guide - AWS Neuron Documentation
• Speculative Decoding (EAGLE3) in the vLLM Neuron Framework - AWS Neuron Documentation
• Transformers NeuronX - AWS Neuron Documentation
• Optimize model inference for latency - Amazon Bedrock User Guide
• Reproducible LLM Inference - Batch Invariance, the Cost of Determinism, and Where Bitwise Identity Is Actually Required
• Disaggregated Prefill and Decode for LLM Serving on AWS - The KV Transfer, the Routing Threshold, and What Disaggregation Does Not Fix
• Self-Managed LLM Inference on Amazon EKS - Serving Open-Weight Models with vLLM, Neuron/GPU, and Karpenter
• Amazon Bedrock Inference Throughput and Latency Optimization - Quotas, Provisioned Throughput, Latency-Optimized Inference, Prompt Caching, and Intelligent Prompt Routing
• Amazon Bedrock Model Evaluation Practical Guide - Automatic Metrics, LLM-as-a-Judge, RAG Evaluation, and CI/CD Quality Gates
• LLM Inference Resilience Patterns on AWS - Retries with Backoff and Jitter, Circuit Breakers, and Stream Recovery
• AWS Custom Silicon History and Timeline - Nitro System, Graviton, Inferentia, Trainium, and Introduction
• AWS History and Timeline regarding Amazon SageMaker - Overview, Functions, Features, Summary of Updates, and Introduction
• Open-Weights LLM Release History and Timeline - Llama, Mistral, Qwen, DeepSeek, and gpt-oss Model Families, Capability Evolution, and Licenses

References:
Tech Blog with curated related content