hidekazu-konishi.com

Reproducible LLM Inference - Batch Invariance, the Cost of Determinism, and Where Bitwise Identity Is Actually Required

First Published: 2026-08-24
Last Updated: 2026-08-24

You are building an evaluation pipeline. You send the same prompt twice and two different answers come back. You set temperature to 0, try again, and the results mostly line up. Then, after running for a while, they differ again. Put it in CI and the nightly build passes while the daytime build fails. And the failure will not reproduce on your own machine.

Many people encounter this problem and settle on the explanation that it's unavoidable due to the use of floating-point numbers. They reason that GPUs perform calculations in parallel, and because floating-point addition is not associative, the order in which the operations are performed can affect the result. Therefore, they conclude that achieving perfect consistency is unrealistic, and a tolerance must be accepted. This explanation is widely circulated, and it's partially correct.

However, this explanation does not actually pinpoint the root cause of what is happening. That becomes clear once you follow the explanation through to a prediction. If the issue were due to non-deterministic parallel execution, then running the same matrix multiplication with the same data a thousand times should produce a thousand slightly different results. In reality, the results are bitwise identical. The GPU is undoubtedly performing parallel calculations, and the numbers are undoubtedly floating-point, yet there's no non-determinism.

This article aims to identify the true cause. The reason why the model sometimes produces different outputs for the same input is not due to parallel execution or floating-point numbers themselves. It is because the inference engine's kernel changes how it splits a reduction as the batch size changes. A change in the split alters the order of additions, which in turn affects the rounding, and changes in rounding affect the logits. If the logits change, even with a temperature of 0 and a greedy decoding method, the model might select a different token.

Three conclusions are worth stating up front.

First, the root cause is not in the sampling process. Attempts to make sampling deterministic, such as setting temperature to 0, fixing the seed, or restricting top_p, reach only the random number layer. If the underlying numerical values have changed, a deterministic sampler will inevitably select a different token. As long as this fundamental misunderstanding persists, adjusting parameters will not resolve the issue.

Second, in production the batch size fluctuates with the load. Inference servers that perform continuous batching process incoming requests together in a single forward pass. This means that even if your request is identical, your output can move when the other requests in flight alongside it change. This is the most counterintuitive conclusion of this article. However, the content of other people's requests is not mixed into your output. It simply results from differences in the path taken due to rounding errors; no information is leaked. Chapter 6 takes this up directly.

Third, determinism can be bought, and it is paid for in time. And it is not necessary to purchase it in every situation. Open-source inference engines do carry a way to switch determinism on, and when it is on, the output lines up bitwise. However, this will slow down the process. The degree to which it slows down depends on the implementation and configuration, and published measurements vary considerably. This article cites the numbers with their source and their configuration, and then draws the line between where determinism is worth pursuing and where it is not.

All specifications and numerical results presented in this article have been verified using primary sources. The verification date is August 23, 2026. The sources split by claim type. The causes and experiments related to non-determinism are documented in the technical blog of Thinking Machines Lab. The behavior of inference engines is described in the official documentation for vLLM and SGLang. What a proposed method guarantees comes from the arXiv originals. The parameters for managed inference are documented in the official AWS documentation.

The division of labor with the existing articles is worth setting out first. Several articles on this site already use the word determinism, but the subject matter of those articles and this article are distinct. Discussions about agent control flows changing with each execution can be found in Agent Reliability Engineering Design Guide. Information about deterministically replaying workflows is available in AWS Lambda Durable Functions Practical Guide. Layers for verifying the correctness of outputs are discussed in LLM Output Verification Patterns. This article focuses solely on whether the same input produces the same numerical output.

Table of Contents

1. 1. The Same Input Does Not Give You the Same Output
2. 2. Three Things That One Word Is Being Used For
3. 3. What You Actually Observe
4. 4. Why the Usual Explanation Is Not Enough
5. 5. The Real Cause Is That Kernels Are Not Batch-Invariant
6. 6. Your Request Is the Same, but Who Was Next to It Changes the Output
7. 7. How to Get Numerical Determinism Back
8. 8. What It Costs
9. 9. What a Seed Decides and What It Does Not
10. 10. Parallelism Breaks Determinism Too
11. 11. Making Everything Deterministic May Be Over-Constrained
12. 12. What You Cannot Do on Managed Inference
13. 13. Where Determinism Is Needed and Where It Is Not
14. 14. Failure Modes and Anti-Patterns
15. 15. Frequently Asked Questions
16. 16. Summary
17. 17. References

1. The Same Input Does Not Give You the Same Output

1.1 Intended Audience

This article is aimed at readers who are encountering challenges while attempting to build systems.

Specifically, there are three kinds of reader.

First, those building pipelines to evaluate model outputs and finding inconsistent results. For example, they may be measuring agreement with a reference solution, but find that running the same dataset twice produces different numbers.

Second, those trying to prevent model or prompt regressions in their CI/CD pipelines. They may have designed a system to halt changes if the output on a golden dataset deviates, but the system is stopping unexpectedly even when no changes have been made.

Third, those managing their own inference infrastructure. They may be struggling to determine whether changes in output are due to an updated engine version or simply inherent variability.

In all these cases, the core issue is an attempt to determine sameness based on a single criterion, which ultimately proves unreliable.

This article assumes a modern inference engine that utilizes continuous batching and paged KV caches. Readers who have deployed systems like vLLM or SGLang, or even those who have used managed inference endpoints such as Amazon Bedrock, will find this relevant. You do not need to be able to write a kernel yourself. This article carries no kernel implementation code.

1.2 What This Article Does Not Cover

This article does not address discussions about how adding elements to accelerate inference might alter numerical values. A separate article, Speculative Decoding and Draft Models on AWS, covers what happens when you turn on speculative decoding, and LLM Weight Quantization on AWS covers the effects of reducing weight precision. This article focuses solely on the initial phenomenon where values change despite no modifications being made.

Similarly, this article will not cover topics such as reproducibility on the training side, including discussions about data ordering or initialization. It will also not cover how to write a kernel yourself. Details on which services to use and how to run evaluation jobs are documented in the Amazon Bedrock Model Evaluation Practical Guide, so this article writes only what should be compared and delegates the rest. A compatibility table for inference parameters across different providers is maintained in the LLM API Parameter Compatibility Reference, and therefore, this article will not create such a table.

1.3 What the Existing Articles Already Hold

Before reading this article, it's helpful to understand the following correspondence to avoid confusion. There are multiple instances where the same term refers to different concepts.

Existing Document	Content Covered by the Document	Relationship to This Article
Agent Reliability Engineering Design Guide	Non-deterministic control flow. Running the same task twice may result in the agent calling different tools in a different order.	While the terminology may be the same, the subject matter differs. This article focuses on the numerical aspects. This article provides the rationale behind the solutions recommended in that document.
AWS Lambda Durable Functions Practical Guide	Deterministic replay of workflows. Ensures that interrupted processes follow the same path when resumed.	Not covered in this article.
LLM Output Verification Patterns	Verification layers for output. Includes schema validation, justification, judge models, and manual review.	This article distinguishes between correctness and identity. This article focuses solely on identity.
Amazon Bedrock Model Evaluation Practical Guide	Methods for running evaluation jobs. Includes program evaluation, judge models, manual evaluation, and CI gate integration.	That document addresses the fact that results are not reproducible and provides solutions. This article provides the reasons behind those solutions.
LLM API Parameter Compatibility Reference	Compatibility table for four different providers. Includes a row for the seed parameter.	This article will not create a table; it will only describe what the seed parameter controls and what it does not.
Self-Managed LLM Inference on Amazon EKS	Considerations for self-hosting and serving configurations.	This article does not argue whether or not to self-host. It only addresses the scenario where someone who has chosen to self-host seeks determinism.
Disaggregated Prefill and Decode for LLM Serving on AWS	Separation of prefill and decode, along with KV forwarding. Includes chunked prefill and prefix caching as components.	This article only describes what these components do in relation to determinism.

This division of labor is a prerequisite for this article. This article will not rewrite solutions already described in existing documents, such as comparing distributions, adding buffer in gates, or measuring the distribution of judge models. Instead, this article will explain why those solutions are necessary.

1.4 Verification Date and Primary Sources

This article checked every technical claim against the materials below, grouped by claim type. Every verification date is August 23, 2026.

Claim Type	Primary Source
Causal relationships, experiments, and performance metrics related to non-determinism	Thinking Machines Lab technical blog, Defeating Nondeterminism in LLM Inference (September 10, 2025)
What the inference engine can do, flag names, supported backends, and availability status	vLLM official documentation and SGLang official documentation
Measurements on the SGLang side	LMSYS technical blog, Towards Deterministic Inference in SGLang and Reproducible RL Training (September 22, 2025, updated September 24)
What a proposed method guarantees	The arXiv original, with the v1 date given
Inference parameters for managed inference	Amazon Bedrock user guide and API reference

This article is not a setup guide. The specific flags to enable are documented in the official documentation, and that information is subject to change. This article aims to explain the reasoning behind the need for each flag and to provide a framework for users to independently determine whether or not to enable them.

2. Three Things That One Word Is Being Used For

This chapter is the most important in this article. If you skip it, the subsequent discussion will become unclear.

The problem lies in the fact that the terminology used in this field has two independent axes, and the single word deterministic ends up covering both axes. The first axis concerns what is being asserted, and the second concerns what is being fixed or constrained. Each is defined in turn.

2.1 A Three-Level Vocabulary - What Is Being Claimed

When describing how a modified inference system compares with the version before it, there are in fact three different levels of claim. The vocabulary below names all three. They are listed here from strongest to weakest.

The first level is bitwise identity. For the same request, the same sequence of bytes and the same tokens come back. A string comparison matches. This is the strongest claim, and an exact-match regression test is only usable while this level holds.

The second level is distribution equivalence, written as distribution-preserving when the property itself is being named. Individual outputs may differ. What is asserted is that the probability distribution the outputs are drawn from is the same. It does not say that the same random draw produces the same output. It says that over infinitely many draws, the same distribution appears.

The third level is quality parity. The distribution is allowed to differ. All that is claimed is that the two are indistinguishable once the metrics are aggregated. Promises at this level look like equal scores on a benchmark, or human raters finding no difference.

These three levels are hierarchical. If a system exhibits bitwise identity, it also demonstrates distribution equivalence, and if it demonstrates distribution equivalence, it also exhibits quality parity. The reverse does not hold. This is where practical errors most often occur. Assuming that a system exhibiting only quality parity is equivalent to bitwise identity can break regression tests.

Level	English	Claim	Typical Failure Example
First Level	bitwise identity	The same sequence of bytes is returned for the same request.	Batch size changes. Parallelism changes. Hardware changes. Engine version is upgraded.
Second Level	distribution equivalence	Outputs are drawn from the same probability distribution.	A technique is introduced that relaxes the acceptance criteria used in verification.
Third Level	quality parity	The systems are indistinguishable when evaluated using aggregated metrics.	A change is made that reduces accuracy, but the degradation only becomes apparent on metrics that are not currently being measured.

This ladder is not unique to this article. bitwise identical is the wording the Thinking Machines Lab blog uses. The second level is the property the original speculative decoding paper states as without changing the distribution. quality is the word the AWS documentation uses. What this chapter does is line up terms that each field uses separately and put them on one ladder.

This article deals only with the first level. The second and third levels break in ways that separate articles take up. However, the second and third cannot be discussed until the first is defined, so the definition is settled here.

2.2 A Second Axis - Invariance With Respect to What

The second axis concerns what remains unchanged when certain elements are allowed to vary. The same word deterministic is used for at least four different forms of invariance.

The blog of Thinking Machines Lab effectively organizes this complexity, with the key point being that all four statements can be true simultaneously.

Some kernels on GPUs are nondeterministic.
However, all the kernels used in a language model's forward pass are deterministic.
Moreover, the forward pass of an LLM inference server (like vLLM) can also be claimed to be deterministic.
Nevertheless, from the perspective of anybody using the inference server, the results are nondeterministic.

The apparent contradiction arises because the four statements each address a different form of invariance. They can be summarized as follows:

Invariance	What is Fixed	Holds True
Run-to-run invariance	The same process, the same input, and the same batch, run twice	Holds. The forward pass of an LLM is run-to-run deterministic.
Batch invariance	The input holds, and only the batch size and the position in the batch change	Does not hold. This is the subject of this article.
Invariance across parallel sizes	The input and the batch are fixed, and the tensor parallel size changes	Does not hold by default. Chapter 10 takes it up.
Invariance across versions and hardware	The input holds, and the GPU generation or the library version changes	Does not hold. Nothing guarantees it.

The heart of the problem is the situation where run-to-run invariance holds and the system still looks nondeterministic from the user's side. The server consistently returns the same answers for a given set of requests. However, the user can only specify their own requests. The other requests in flight alongside it lie outside their control. To the user, these other requests are not inputs; they represent the system's non-deterministic nature.

Throughout this article, when determinism appears without qualification, it means numerical determinism, specifically the second row in the table above. To distinguish it from deterministic control flow or deterministic replay of workflows, it is explicitly qualified where that matters.

2.3 The Word batch Also Carries Two Meanings

One more word has to be cut at first use. In this field, batch points at two entirely different things.

The first meaning is a collection of requests processed simultaneously through a single forward pass by the inference engine. This is a continuously adjusted group, and its size changes dynamically based on the load. Whenever this article says batch, it means this one.

The second meaning is a system that aggregates a large volume of input and processes it as asynchronous jobs. Amazon Bedrock has a feature called Batch inference, which you drive by submitting a job with CreateModelInvocationJob. It operates under a separate quota from interactive pathways. This article does not cover that feature. Further details can be found at Amazon Bedrock Inference Throughput and Latency Optimization.

These two meanings are not entirely unrelated. Even within asynchronous jobs, a batch (in the first meaning) is formed when the model is actually executed. Therefore, submitting a job asynchronously does not pin the batch size.

2.4 Sameness and Correctness Are Different Problems

One final point to make. The fact that a system produces the same output every time, and the fact that its output is correct, are entirely different properties.

A system that consistently returns the same errors is completely deterministic, yet entirely wrong. Conversely, a system that consistently provides correct answers, even if expressed in different ways, is non-deterministic, yet correct. Determinism is not a proxy for correctness.

This distinction is important because the motivations for seeking determinism are often misunderstood. The desire for determinism is not about improving the output itself. It is about being able to isolate the effects of changes. When the output changes, you can determine whether that change is due to a modification you made, or whether the output was inherently unstable. This is not quality in itself, but rather a foundation for measuring quality.

How to verify the correctness of the output is detailed in LLM Output Verification Patterns. This article does not delve into the topic of correctness.

3. What You Actually Observe

3.1 A Thousand Requests Produce Eighty Different Completions

Thinking Machines Lab has published experimental data quantifying this phenomenon. The numerical values cited in this article are all derived from this experiment and are not the result of the author's own measurements.

The setup is as follows: the model, Qwen/Qwen3-235B-A22B-Instruct-2507, is used in non-thinking mode. The sole prompt used is Tell me about Richard Feynman. The temperature setting is set to 0. The same prompt is submitted 1000 times, with each submission generating 1000 tokens.

The results were as follows:

Observation	Value
Total number of generated completions	1000
Number of distinct completions	80
Frequency of the most common completion	78
Number of tokens matching at the beginning	102
Token position where divergence first occurred	103rd token

The content of the divergences is also publicly available. All 1000 completions generated the same string of characters up to Feynman was born on May 11, 1918, in. After that, 992 completions proceeded to Queens, New York and 8 completions proceeded to New York City.

The original source states the following regarding the divergence point:

Looking at where the completions differ, we see that the completions are actually identical for
the first 102 tokens! The first instance of diverging completions occurs at the 103rd token.

Results from the same experiment also appear with the batch-invariant kernels switched on.

On the other hand, when we enable our batch-invariant kernels, all of our 1000 completions are identical.

The number 80 needs reading carefully. This does not mean that the output was unusual only 80 out of 1000 times. Instead, the 1000 completions were divided into 80 distinct groups, with the largest group containing only 78 completions. In other words, even the most frequent output accounts for less than 8 percent of the total.

3.2 One Flipped Token Changes Everything After It

One token flips at position 103, and as many as 80 distinct completions come out of it. The reason is the structure of autoregressive generation.

Language models incorporate previously generated tokens as input to determine the next token. Therefore, if a token changes in one location, the subsequent input is entirely modified, and all subsequent tokens are determined under different conditions. The divergence originates at a single point, but its impact extends to the very end.

The challenge in practical applications arises because the differences can appear minimal. Often, these differences stem from rounding errors, beginning at the level of the least significant bits. However, the observed differences manifest as complete paragraph variations. Consequently, it's easy to overlook the underlying numerical issue and instead attribute the behavior to the model being unpredictable.

There is a pattern to where flips occur. A token flips where the top candidate and the runner-up have logits that sit close together. A slight difference in the least significant bits can then change the ranking. If the difference is significant, even minor rounding variations will not affect the ranking. This property serves as the starting point for a proposal that Chapter 11 takes up.

3.3 A Single Matrix Multiplication Shows It

This phenomenon can be reproduced with just a few lines of code, without even waiting for the generation of a thousand tokens. The example below is the one the original source gives.

import torch
torch.set_default_device('cuda')

B = 2048
D = 4096
a = torch.linspace(-1000, 1000, B * D).reshape(B, D)
b = torch.linspace(-1000, 1000, D * D).reshape(D, D)
# Doing a matrix vector multiplication by taking
# the first element of the batch
out1 = torch.mm(a[:1], b)
# Doing a matrix matrix multiplication and then taking
# the first element of the batch
out2 = torch.mm(a, b)[:1]
print((out1 - out2).abs().max()) # tensor(1669.2500, device='cuda:0')

The process is simple. It compares the result of multiplying only the first row of a with the result obtained after multiplying a entirely, and then taking the first row of that result. Mathematically, these are identical calculations. Matrix multiplication requires that each element of the batch dimension be independent. Regardless of what the other rows contain, or how many rows there are, the result for a given row should remain unchanged.

However, in practice, a difference of up to 1669.25 is observed. Simply changing the batch size from 2048 to 1 results in a different calculation result for the same row.

3.4 It Does Not Vary From Run to Run

The code above returns the same difference no matter how many times you run it.

The original source explicitly addresses this. The behavior is deterministic; running the same script twice will always produce the same result. However, this may change if the hardware or the version of PyTorch is different. The original source states it as follows:

It is not "hardware/software version invariant" — your GPU/PyTorch version may return a
different value, but it should deterministically return the same value.

This means it's not a bug that fails randomly. If the input and batch remain the same, the results are completely reproducible. The issue only arises when the batch changes. This is where the difficulty in debugging lies. Run it by hand and the same result comes back every time. Put it in production and it stops coming back. By hand, the batch size is always 1.

4. Why the Usual Explanation Is Not Enough

4.1 The Explanation in Circulation

When explaining this phenomenon, the most common explanation is as follows: GPUs perform calculations in parallel using a large number of cores. The order in which each core completes its work varies with each execution. Floating-point addition is not associative, meaning that changing the order of operations can alter the final result.

The original source refers to this as the concurrency + floating point hypothesis, and assesses it as not incorrect, but not a complete picture.

While this hypothesis is not entirely wrong, it doesn't reveal the full picture.

As long as this hypothesis is accepted, the problem remains unsolved. If the cause is attributed to parallel processing, the only solutions are to either eliminate parallel processing or accept the issue. In reality, neither of those options is necessary.

4.2 The Prediction It Makes Does Not Match Reality

That the hypothesis is incomplete becomes clear once you test a prediction it makes. If the nondeterminism came from the order in which concurrent work finishes, then repeating the same matrix multiplication on the same data should make the result wobble. The original source tested exactly that.

A = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16)
B = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16)
ref = torch.mm(A, B)
for _ in range(1000):
    assert (torch.mm(A, B) - ref).abs().max().item() == 0

This assertion passes all 1000 times. The GPU is running in parallel, the numbers are floating point, and the matrices are large enough. The results are still bitwise identical.

Therefore, parallel execution and floating-point arithmetic alone cannot account for the observed non-deterministic behavior. They only explain the possibility of numerical differences. When they occur, and why they occur then, needs a separate explanation.

4.3 Ordering Becomes Nondeterministic When Atomic Adds Are Used

So, when does the kernel become non-deterministic on each execution? The answer is when using atomic add.

When multiple cores need to add values to the same location, atomic add provides a means to reflect the contributions of all cores without guaranteeing a specific order. While the hardware ensures that all additions are processed, it does not guarantee the order in which they are processed. Whichever core finishes first sets the order. Therefore, kernels that include this operation become non-deterministic on each execution.

This is where the argument turns. The original source states that atomic adds barely appear in the forward pass of an LLM.

Although concurrent atomic adds do make a kernel nondeterministic, atomic adds are not
necessary for the vast majority of kernels. In fact, in the typical forward pass of an LLM,
there is usually not a single atomic add present.

There are two reasons for this. First, there is sufficient parallelism in the batch dimension, so there is no need to parallelize the reduction dimension. There is no need to run the reduction for a single 100-element vector across 100 cores; you can hand 500 vectors to 500 cores, one each. Second, techniques for getting determinism without giving up performance are already widespread. A split reduction or a tree reduction is used, and the final merge is either finished in a non-parallel clean-up step or ordered with a semaphore.

The original source concludes:

However, the forward pass of an LLM involves no operations that require atomic adds. Thus, the
forward pass in an LLM is in fact "run-to-run deterministic."

⚠ Watch the scope of this claim. The document is referring specifically to the forward pass of LLMs. While atomic add is rarely needed, there are operations where it is actually required, such as the backpropagation of FlashAttention, which is commonly used in LLMs. Backpropagation deals with training, not inference. Furthermore, the original source uses the words typical and usually, so it does not state that the count is always zero in every implementation.

Therefore, it is correct to state that parallel execution is irrelevant only with the qualification that this applies specifically to the forward pass of LLM inference.

4.4 What Is Left, Then

Three things are settled at this point. The kernel is deterministic in its execution. Similarly, the inference server is deterministic in that it provides the same answer for a given set of requests. However, from the user's perspective, it still appears non-deterministic.

There is only one explanation left. The output of a user's request is, in fact dependent on the other requests in flight alongside it. The original source states this dependency in a single line.

As it turns out, our request's output does depend on the parallel user requests.

Chapter 5 takes up the nature of that dependency. And Chapter 6 will address why this dependency should not be interpreted as information leakage.

5. The Real Cause Is That Kernels Are Not Batch-Invariant

5.1 What Batch Invariance Means

Batch invariance is the property that the computed result for each element of a batch does not change when the batch size changes.

Mathematically, this should naturally hold true. Matrix multiplication is independent for each element along the batch dimension. Regardless of the values of other elements, or the number of elements in the batch, the result for a given element should not change. The original source emphasizes this point.

This is a fairly unusual property from a mathematical perspective. Matrix multiplication should
be "independent" along every element in the batch — neither the other elements in the batch nor
how large the batch is should affect the computation results of a specific element in the
batch.

However, the implementation does not possess this property. The difference of 1669.25 observed in Chapter 3 serves as a counterexample.

The original source makes a more nuanced distinction. While many matrix multiplication implementations are not batch-invariant, they do possess a weaker property: the result does not change when an element's position in the batch changes. However, a parallelization strategy called stream-k fails to even satisfy this weaker property. It splits along the reduction dimension differently for different output tiles, to get cleaner load balancing.

5.2 The Causal Chain

The path by which changes in batch size lead to changes in output tokens is as follows:

First, when a request arrives, the inference engine gathers it together with whatever else is in flight into a single forward pass. It is at this point that the batch size is determined. Second, the kernel examines the batch size and selects a splitting strategy that can effectively utilize the cores. Third, if the splitting strategy changes, the point at which partial sums are divided and added changes. Fourth, if the division point changes, the order of addition changes, and the rounding of floating-point numbers changes. Fifth, if the rounding changes, the values of the logits change. Sixth, if the logits change, the ranking of tokens can shift, particularly for tokens where the top candidate and runner-up are closely matched.

The last step is where it matters. Greedy decoding with a temperature of 0 is a deterministic operation that simply selects the logit with the highest value. Even though it is a deterministic operation, if the input logits change, the output will change. No matter how much you fix the sampling side, this path cannot be blocked.

5.3 Why Kernels Change the Splitting Strategy

The reason is for performance. GPUs have numerous processing units, and leaving them idle results in slower operation.

For example, when calculating RMSNorm, the most straightforward parallelization method is to assign each element of the batch to a separate core. With this approach, the reduction operation can be completed entirely within each core, eliminating the need for communication between cores. Whether the batch size is 200 or 2000, only the number of rows assigned to each core increases; the reduction order for a given row does not change.

The problem arises when the batch size decreases. If there are more cores than elements, some cores go idle. At this point, a good kernel engineer splits the reduction dimension instead, to buy parallelism back. At that moment, the reduction order for a given row changes. This breaks the batch-invariance property.

In other words, the very optimizations that enhance performance are what break the batch-invariance property. It is not a bug. Therefore, enabling batch invariance requires deliberate intervention, and enabling it results in a performance decrease.

5.4 The Three Operations That Involve a Reduction

Not all kernels are problematic. Element-wise operations, by their nature, do not involve reduction and are therefore batch-invariant. The only operations you have to worry about are the ones that contain a reduction. The original source lists three such operations, arranged in order of increasing difficulty.

we only need to worry about the 3 operations that involve reductions — RMSNorm, matrix
multiplication, and attention

First, there's RMSNorm. This involves calculating the root mean square across the feature dimension. The fix is comparatively simple. Either leave the small-batch case unoptimized, or always use a splitting strategy that carries enough parallelism even at very small batch sizes. The original source notes that, when dealing with small batch sizes, the kernel will finish quickly anyway, so a slight slowdown is not critical.

Second, there's matrix multiplication. This operation has two additional constraints. One is split reduction, also known as split-K, which is used when the two dimensions corresponding to the batch dimension are small. The other is the tensor core instruction itself, which has different internal reduction orderings for each instruction. With small batch sizes, the system may switch to smaller tile instructions or fall back to a path that does not use tensor cores. Both of these scenarios break batch invariance.

The solution is to use the same kernel configuration for all shapes. Naturally that runs slower than optimizing per shape. Regarding the performance impact, the original source states:

Despite obtaining batch invariance, we only lose about 20% performance compared to cuBLAS.

⚠ This 20 percent refers specifically to matrix multiplication alone, and not the overall slowdown in inference. Chapter 8 covers the end-to-end numbers.

Third, attention. This is the hardest of the three, and the next section separates out why.

5.5 Why Attention Is the Hardest Case

Attention presents two unique challenges that the other methods do not.

The first is that it performs reduction across two dimensions. While RMSNorm and matrix multiplication only perform reduction across the feature dimension, attention performs reduction across both the feature dimension and the sequence dimension.

The second is that the way the sequence is divided can change based on the inference engine's needs. With chunked prefill on, a single sequence can travel through the engine as several chunks. With prefix caching on, the front of the sequence comes from the cache and only the tail is computed fresh. Even for the same sequence, the portion that comes from the cache and the portion that is newly calculated can vary depending on the situation.

Therefore, the invariance required for attention cannot be achieved with batch size alone. The original source describes it as follows:

In order to achieve "batch invariance", it's necessary that the reduction order for a given
token does not depend on how many other tokens from its sequence are being simultaneously
processed.

Specifically, how does it break down? Suppose the block size is 32, the KV cache contains 80 elements, and 48 new elements are being computed. If reduction is performed separately on the cached portion and the newly computed portion, the cached portion results in 3 blocks, and the newly computed portion results in 2 blocks, for a total of 5 blocks. However, the total number of elements is 128, and if they are treated as a whole, 4 blocks are enough. Cut the blocks differently and the reduction order differs.

The solution presented in the original source is to update the KV cache and the page table before calling the attention kernel. This ensures that the arrangement of keys and values is always consistent, allowing reduction to be performed using the same division regardless of which tokens are cached.

There is another layer to this as well. In the decode stage, the length of the query side can become extremely short, making batch dimension and head dimension, along with query length, insufficient for parallelism. Therefore, strategies like Split-KV and FlashDecoding, which involve dividing the KV dimension, are used. This case is not as easy to set aside as RMSNorm and matrix multiplication were. With a long KV cache, the attention kernel can take a long time even when it is processing only one request.

Simply fixing the number of divisions is not enough. The original source requires fixing the size of the divisions. For example, if the KV length is 1000, instead of dividing it into 4 parts of 250 each, you would have 3 divisions of size 256 and one division of size 232. The number of divisions can vary depending on the situation, but the positions of the boundaries are determined solely by the length of the input. This is the fixed split-size strategy.

The original source also provides examples where existing implementations fail to meet this requirement. FlashInfer's balanced scheduling algorithm picks the largest split size that still saturates every core on the GPU, and that makes its reduction strategy not batch-invariant.

5.6 This Is Not Specific to GPUs

The original source states:

This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs
will also have this source of nondeterminism.

The root cause is that a parallel machine changes its splitting strategy according to the batch size. This behavior is independent of the type of accelerator used. Therefore, the idea that switching away from GPUs will resolve the issue is not valid.

6. Your Request Is the Same, but Who Was Next to It Changes the Output

This chapter is at the heart of this article. Chapters 1 through 5 established that the output can change when the batch size changes. The only remaining question is: Does the batch size change in a real-world setting?

6.1 Continuous Batching Changes the Batch Size With Load

The batch size can change, and it changes for reasons that users cannot control.

Modern inference servers run continuous batching. Incoming requests go into a queue, and whatever can be served at that moment is gathered into one forward pass. Completed requests are removed, and new ones are added to the queue. The traffic arriving at that moment decides what goes into each batch.

Therefore, the same request might be processed with a different batch size depending on whether you are using the server alone late at night, or concurrently with other users during the day. The original source summarizes this in a single sentence.

the primary reason nearly all LLM inference endpoints are nondeterministic is that the load
(and thus batch-size) nondeterministically varies!

In other words, this means: Even if your request is exactly the same, the output you get back can move with whatever else happens to be in flight alongside it.

The original source explains the discrepancy between the server's perspective and the user's perspective as follows:

Although the inference server itself can be claimed to be "deterministic", the story is
different for an individual user. From the perspective of an individual user, the other
concurrent users are not an "input" to the system but rather a nondeterministic property of the
system.

The server is not lying. Given a specific set of requests, it will always return the same answer. However, users can only specify the requests within that set that they are directly concerned with. It is the portion that the user cannot specify that affects the result.

6.2 Nothing Is Leaking Between Requests

This conclusion can be misleading at first glance. The fact that another user's request influences your output might lead one to believe that information is being leaked. However, this is not the case. The original source explicitly states this.

Not because we're somehow leaking information across batches — instead, it's because our
forward pass lacks "batch invariance", causing our request's output to depend on the batch size
of our forward pass.

The only thing that is changing is the path through which rounding errors propagate. Another user's tokens, embeddings, or KV cache are not being incorporated into your calculations. The matrix multiplication is still computed independently along the batch dimension. What mixes in is the position where the reduction is cut, and nothing else.

That position depends on the other requests through exactly one integer, the batch size. What reaches you is the head count, not the content.

This article does not treat the phenomenon as a confidentiality problem. It stands where the original source explicitly stands. What it treats is an operational problem: evaluation and regression testing stop working.

6.3 Two Matching Runs Prove Nothing

This structure reveals that a common practice in real-world testing is actually invalid. Sending the same prompt twice and getting a match does not let you conclude that the system is deterministic.

If the prompt is processed with the same batch size on both runs, it will likely produce the same result. However, this does not prove determinism; it simply means that you happened to use the same conditions in both instances. In a development environment, when no one else is using the system, the batch size will almost always be 1. Therefore, you will consistently see matching results locally.

And in production it does not hold. This is the root cause of the phenomenon where nightly builds pass, but daytime builds fail. The shared endpoints experience varying levels of load depending on the time of day when the CI system interacts with the model.

The correct approach is to vary the batch size. The design of SGLang's verification command is not accidental; it works by submitting the same prompt with different batch sizes and counting the number of differing outputs. If the system is truly deterministic, this number should be 1.

6.4 A Single Request Can Break It Too

There is another counterintuitive path where outputs can change, even when no other users are active.

The root cause lies in how sequences are processed, as discussed in Chapter 5. With prefix caching on, the same prompt computes differently depending on whether its front sits in the cache. With chunked prefill on, a long prompt travels in chunks, and where those divisions occur depends on the chunk budget at that moment.

Therefore, outputs can change even in scenarios like these: submitting a request, allowing it to warm the cache, and then submitting the same request again. Or, submitting another request that shares only the initial part of the prompt before submitting the main request. Mixing prompts of different lengths can also lead to variations.

This behavior occurs even with a batch size of 1. SGLang's verification commands include tests that vary the length of the prefix, and a comparison between cached and uncached prefill, on top of the plain change of batch size, specifically to account for this.

The design of chunked prefill and prefix caching itself is described in Disaggregated Prefill and Decode for LLM Serving on AWS. This article only discusses how these mechanisms affect determinism.

7. How to Get Numerical Determinism Back

7.1 There Are Only So Many Places You Can Reach It

The only side that can control how a batch is formed is the side running the inference engine. Users accessing managed inference endpoints do not have this option, a consequence that Chapter 12 takes up.

Whether to self-host at all belongs to Self-Managed LLM Inference on Amazon EKS. This article only covers what someone who has already chosen to self-host decides next.

This is not an all-or-nothing choice. A small endpoint dedicated to evaluation and regression testing can run with determinism on, while production traffic is served in the normal mode. Chapter 13 covers that arrangement.

7.2 Enabling It in vLLM

vLLM has batch invariance as a feature. ⚠ Watch the availability status. The official documentation notes the following on both the development and stable versions.

Batch invariance is currently in beta. Some features are still under active development.

This is not GA. On August 23, 2026, the verification date for this article, the note was present in both versions of the documentation. Before you consider adopting it, check first that this note has not changed.

A single environment variable turns it on.

export VLLM_BATCH_INVARIANT=1

When starting the server, use the same variable as a prefix. Even for offline inference, setting the environment variable before loading vllm will work.

There are hardware requirements.

Batch invariance requires NVIDIA GPUs with compute capability 8.0 or higher.

⚠ Do not interpret this sentence as the sole requirement. vLLM has a derivative project for Ascend NPUs, which has its own separate page and requirements.

Batch invariance currently requires Ascend Atlas A2 and A3 inference products NPUs.

This section further states that you need to set an environment variable before building to incorporate custom operator libraries. In other words, the requirements vary depending on the hardware lineage, and even the same feature may be managed on a different page.

Regarding models, the official documentation provides a list of verified models. These include DeepSeek models, dense and MoE models from Qwen3, Qwen2.5 models, Llama 3 models, GPT-OSS, Mistral, and Phi models. What matters is not the list but the sentence that closes it.

Other models may also work, but these have been explicitly validated.

The list is not exhaustive. It is a statement of verification, not a claim that models not listed will not function. Drop that caveat, write that the supported models are the ones listed, and you have made a stronger claim than the official documentation makes.

Regarding implementation, the official documentation also says what changes once you switch it on. It uses deterministic implementations for operations such as attention, ensures consistent numerical behavior even when the batch size changes, and disables optimizations that could introduce non-determinism. An example of an optimization that is disabled is custom all-reduce in tensor parallel mode.

Regarding performance, the official documentation only provides general directions and does not provide specific numerical values.

Enabling batch invariance may impact performance compared to the default non-deterministic
mode. This trade-off is intentional to guarantee reproducibility.

Therefore, you should not cite the numbers discussed in Chapter 8 as official values for vLLM. The source of those numbers is elsewhere.

7.3 Enabling It in SGLang

SGLang carries the same feature, and a startup flag switches it on.

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --attention-backend fa3 \
    --enable-deterministic-inference

⚠ You have to choose a backend. The official documentation limits the supported backends.

Deterministic inference is only supported with the following three attention backends:
FlashInfer, FlashAttention 3 (FA3), and Triton.

Furthermore, compatibility with other features varies depending on the chosen backend. The official compatibility table is as follows:

Attention Backend	CUDA Graph	Chunked Prefill	Radix Cache	Non-Greedy Sampling
FlashInfer	Supported	Supported	Not Supported	Supported
FlashAttention 3 (FA3)	Supported	Supported	Supported	Supported
Triton	Supported	Supported	Supported	Supported

The key takeaway from this table is the row for FlashInfer. If you choose FlashInfer to ensure determinism, the radix cache will become unavailable. The radix cache is a mechanism that avoids redundant recomputation of common prefixes. Disabling it can result in slower performance for workloads with a high volume of requests sharing the same prefix. The trade-off for determinism extends not only to performance but also to the available features.

The official documentation also provides commands to verify whether it has been successfully enabled.

python3 -m sglang.test.test_deterministic --test-mode single --n-trials 50
python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50
python3 -m sglang.test.test_deterministic --test-mode radix_cache

The three tests are designed to target different failure modes. The first test involves sending the same prompt with various batch sizes. The second test uses prompts with different prefix lengths. The third test compares cached and un-cached prefill operations. The expected result for all three tests is that the number of distinct samples should be 1.

7.4 Do Not Build Your Own Support Matrix

Four lists have appeared so far: supported models, supported hardware, supported backends, and compatible features. Every one of them moves. One concrete example turned up during verification.

The SGLang blog, as of September 2025, stated that only dense models were currently supported, and expanding to MoE models was a challenge for the future. However, the official documentation released in August 2026 included an example of launching the MoE model Qwen3-30B-A3B. The underlying assumptions have changed in less than a year.

Therefore, this article will not provide a comprehensive compatibility table. Instead, it gives you the criteria for checking your own environment.

Criteria	Verification Method
Does your attention backend support the system?	Refer to the official documentation's section on supported backends. The default backend may not be supported.
Does your hardware meet the requirements?	NVIDIA and Ascend have separate pages detailing hardware requirements, including generational specifications.
Has your model been verified?	The list of verified models is not exhaustive. If your model is not listed, you should test it yourself.
Do the features you want to use work together?	Some features, such as radix cache, may not function properly with certain backends.
Has the availability status changed?	Check if the beta designation has been removed, or if any new restrictions have been added.

Copy a list into an article and it starts rotting the moment you copy it. The criteria do not rot.

7.5 What Stays Unaligned Even With Determinism On

Switching determinism on still leaves some invariances unaligned. Of the four invariances in Chapter 2, what these features address is the second row, batch invariance.

Invariance across versions and hardware is not guaranteed by anything at all. The numbers can change with a different GPU generation, and they can change when the engine or the framework moves up a version. This is what the note quoted from the original source in Chapter 3 says.

Therefore, even when running regression tests in a deterministic environment, the two executions being compared must operate on the same version and the same hardware. When a new version is released, baseline values must be re-established. This rests on the same reason the self-managed inference article gives for pinning inference container tags exactly.

8. What It Costs

8.1 Settle the Unit First

Before this chapter starts, settle the unit. The same phenomenon can be reported using two different methods of calculation.

For example, suppose the time required for a particular task increases from 26 seconds to 42 seconds. This can be described as a 61.5 percent increase in time, or it can be described as a 38.1 percent decrease in throughput. Both represent the same underlying fact, and the conversion formula is as follows: when time increases by a factor of r, throughput decreases to 1/r.

This article will use a metric based on the increase in time. Where a source uses the other unit, this article says so. ⚠ Be cautious when comparing data using mixed units, as it can lead to an inaccurate perception of the impact, either underestimating or overestimating the cost.

8.2 The Thinking Machines Lab Measurement

The original source describes an experiment where an API server was set up using a single GPU and ran 1000 sequences for Qwen-3-8B. The output length for each sequence ranged from 90 to 110 tokens.

Configuration	Time Required
vLLM default	26 seconds
Unoptimized Deterministic vLLM	55 seconds
+ Improved Attention Kernel	42 seconds

⚠ The model used in this experiment is different from the one used in the experiments described in Chapter 3. Chapter 3 used Qwen/Qwen3-235B-A22B-Instruct-2507, while this experiment uses Qwen-3-8B. This is a separate experiment in the same blog, and the two must not be mixed.

The naive implementation takes about 2.1 times as long. With the improved attention kernel, about 1.6 times. The original source states plainly that it did not put much effort into optimizing performance, and attributes most of the slowdown to the fact that the FlexAttention integration inside vLLM has not been heavily optimized yet.

⚠ The original source does not report this result as a percentage. What it publishes is the seconds above, and nothing else.

8.3 The Measurement on the SGLang Side

The SGLang development team has published measurements taken with a different configuration. These values are reported as percentages.

The configuration involved three different workloads equivalent to reinforcement learning rollout scenarios, each processing 256 requests. It compared end-to-end latency in normal mode and deterministic mode. The model used was Qwen3-8B, with tensor parallelism set to 1, and utilized an H200 GPU with 140GB of memory.

Deterministic inference is generally usable, with most slowdowns ranging from 25% to 45%, and
average slowdown of FlashInfer and FlashAttention 3 backends being 34.35%.

The breakdown is as follows. Every figure is an increase in time.

Attention Backend	Input 1024, Output 1024	Input 4096, Output 4096	Input 8192, Output 8192
FlashInfer	+42.6%	+46.0%	+24.4%
FA3	+27.2%	+30.2%	+35.7%
Triton	+55.1%	+44.64%	+44.80%

⚠ In these measurements, the radix cache was disabled for all performance tests. The reason stated at the time was that radix cache support for FlashInfer and Triton was still in progress. As the table in Section 7.3 shows, Triton has since gained it, so the condition behind this measurement no longer covers the same set of backends.

8.4 Where the 61.5 Percent Figure Comes From

The SGLang blog positions its own 34.35 percent as follows:

Compared to the 61.5% slowdown reported in TML's blog, SGLang achieves an average slowdown of
only 34.35% with the FlashInfer and FlashAttention 3 backends

⚠ A note of caution is necessary here. The blog post from Thinking Machines Lab does not explicitly state the 61.5% figure. It only mentions the number of seconds (as described in Section 8.2). The 61.5% figure is a value calculated by SGLang, representing the increase from 26 seconds to 42 seconds.

Therefore, it is not accurate to directly attribute this number to Thinking Machines Lab. To be precise, Thinking Machines Lab reported the data in terms of seconds, and the SGLang blog calculated that into a percentage increase, then compared it to their own values.

Furthermore, the comparison itself is based on certain assumptions. The two measurements use different models, GPUs, request counts, and workloads. They are not comparable figures. Rather than interpreting it as proof that one side is superior, it is more appropriate to view it as material for understanding the order of magnitude of the trade-offs involved.

8.5 The 2.79x From CUDA Graphs Is a Different Measurement

The same blog post also presents another striking figure. It is easy to confuse with the previous one, so it is set out separately.

The results show an at least 2.79x speedup across all attention kernels when CUDA graphs is utilized.

This figure is the ratio of total throughput with CUDA graphs enabled against disabled, under deterministic mode. The measurement configuration is as follows: 16 requests, input length of 1024, output length of 1024, using the Qwen3-8B model, with tensor parallelism set to 1, and utilizing an H100 GPU with 80GB of memory.

Attention Backend	CUDA Graph Disabled	CUDA Graph Enabled
FlashInfer	441.73 tokens per second	1245.51 tokens per second
FA3	447.64 tokens per second	1247.64 tokens per second
Triton	419.64 tokens per second	1228.36 tokens per second

⚠ This is a different measurement from the 34.35 percent figure. The number of requests, the GPU used, and even the metric being measured are all different. The 34.35 percent is how much slower deterministic mode is than normal mode, while the 2.79x figure reflects the performance gains achieved through optimization within deterministic mode. These two figures should not be combined or used to explain one another.

8.6 How the Projects Themselves Recommend It

The developers themselves openly acknowledge the trade-offs involved.

We acknowledge that deterministic inference is significantly slower than normal mode. We
recommend using it primarily for debugging and reproducibility.

Furthermore, their stated goals include reducing the performance difference to less than 20 percent, ideally bringing it to a level comparable to normal mode. As things stand, it is not presented as a feature to leave switched on across all production traffic.

The vLLM side points the same way, calling the effect on performance a deliberate trade made to guarantee reproducibility.

This understanding directly informs the article's conclusion. The feature should only be purchased and used in specific situations, for a defined period, not left on by default. Chapter 13 covers which situations those are.

9. What a Seed Decides and What It Does Not

9.1 The Random Layer and the Numeric Layer Are Separate

When encountering this phenomenon, the first approach people often try is fixing the sampling. This involves setting temperature to 0, providing a seed, or restricting top_p. None of these solutions work, or they appear to work temporarily before the issue reoccurs. The root cause lies in the underlying structure.

At least two layers are involved in determining each token.

Layer	Function	Controls
Numerical Layer	Calculates the forward pass and generates logits.	Kernel splitting strategy, batch size, parallelism, hardware, engine version.
Sampling Layer	Converts logits into a probability distribution and selects a token from that distribution.	temperature, top_p, top_k, seed.

The seed and the sampling parameters reach only the sampling layer. If the logits change up in the numeric layer, a deterministic sampler will deterministically pick a different token. The sampler is working correctly. Its input is simply not the same.

9.2 What Setting temperature to Zero Guarantees

When temperature is set to 0, the sampling process simply selects the token with the highest logit. This is indeed a deterministic operation. Because no random numbers are involved, the non-deterministic element is eliminated.

What does not go away is the judgment of which token holds that maximum. If the difference between the top candidate and the next best is small, even a change in the least significant bit can alter the ranking. The divergence at the 103rd token in the Chapter 3 experiment happened at exactly such a token.

This point is also explicitly stated in the official documentation provided by the model providers. The Anthropic Messages API reference, in its description of temperature, states:

Note that even with `temperature` of `0.0`, the results will not be fully deterministic.

Even the API providers themselves state that a temperature of zero does not guarantee complete determinism. If you overlook this and treat a temperature of zero as a guarantee of reproducibility, everything you design after that sits off center.

9.3 Non-Greedy Sampling Can Be Deterministic Too

There is one fact that defies intuition. Deterministic output does not require greedy decoding. Even when the temperature is greater than 0, it's possible to achieve deterministic results.

The official SGLang documentation explicitly states this functionality.

SGLang supports deterministic inference even with non-greedy sampling by using sampling seeds.
This is particularly useful for reinforcement learning scenarios like GRPO (Group Relative
Policy Optimization) where you need multiple diverse but reproducible responses.

The mechanism involves replacing the sampling operation itself. According to the development team's explanation, torch.multinomial is inherently non-deterministic when used in batch processing. Therefore, they are introducing an operation that perturbs the logits using Gumbel noise generated from a seeded hash function.

Instead of relying on torch.multinomial, which is inherently nondeterministic under batching,
this operator perturbs logits with Gumbel noise generated from a seeded hash function. As a
result, the same (inputs, seed) pair always yields the same sample, even when temperature >
0.

This distinction is crucial. With the same seed, the same output is produced; with a different seed, a different output is generated. This allows for the creation of a collection of samples that are diverse yet reproducible. In scenarios like reinforcement learning rollouts, where one prompt has to yield several distinct responses, greedy decoding often falls short. The point of this feature is that determinism does not have to be given up there either.

From the user's perspective, a default sampling seed is provided. By passing a different seed with each request, reproducible variations of the output can be obtained.

⚠ Do not drop the precondition. This only works on top of batch-invariant kernels. A seed on its own leaves the numerical layer moving.

9.4 Providers Treat the Seed Differently

Whether or not you can provide a seed value depends on the provider. This article does not create a compatibility table. The relationship between inference parameters for four different providers is documented in LLM API Parameter Compatibility Reference, which includes a row for seed.

This article aims to clarify the assumptions you need to have when reviewing that row. Here are two specific points.

First, Amazon Bedrock's Converse API does not include a seed parameter in its basic parameters. The official documentation lists the following four basic parameters within inferenceConfig:

Field	Meaning
maxTokens	The maximum number of tokens to allow in the generated response.
stopSequences	A list of stop sequences that cause the model to stop generating the response.
temperature	The likelihood of the model selecting higher-probability options while generating a response.
topP	The percentage of most-likely candidates that the model considers for the next token.

A parameter outside that base set has to travel through a model-specific request field.

If you need to pass additional parameters that the model supports, use the
additionalModelRequestFields request field in the call to Converse or ConverseStream.

Therefore, whether or not a seed value can be used depends on the model. What decides it is whether the model accepts the parameter, not what the service API offers. This is a place where the platform side and the model side are mixed together, which is why the existing article's table is laid out the way it is.

Second, Anthropic's Messages API does not have a seed parameter. The reference documentation, as of the date of verification, does not list seed. The note referenced in Section 9.2 can be interpreted as a clarification to that effect.

9.5 How to Read a Document That Says Deterministic

Once you have this, you can decide which of the three levels in Chapter 2 a document means when it uses the word deterministic.

For example, the Amazon Nova user guide states that lower temperature values make the output more deterministic. This statement is correct. What it claims is that the spread of the sampling narrows. It does not claim bitwise identity. It is a relative statement, saying that the output becomes more predictable, and it is not a promise that the same byte sequence comes back.

⚠ The provider is not writing something incorrect. The accident happens when the reader takes a relative description as an absolute guarantee. Read it against the following mapping.

What the document says	Which level in Chapter 2	What you may expect
More deterministic, more predictable	None of them. This is about the spread of the sampling.	The diversity of the output decreases.
Preserves the distribution	Second level	The statistical properties are preserved. Individual outputs may still vary.
Does not degrade quality	Third level	The aggregate metrics remain consistent.
Bitwise identical, deterministic output	First level	The output will match when compared as strings. Always check what is being held constant.

10. Parallelism Breaks Determinism Too

10.1 These Are Two Separate Problems

The relationship between tensor parallelism and determinism can be divided into two issues that are easily confused.

The first is whether determinism can be achieved while maintaining a fixed size for tensor parallelism. This one is addressed, though how far it reaches depends on the parallel size, as Section 10.2 shows. As an example of an optimization that gets disabled when batch invariance is switched on, vLLM names custom all-reduce in tensor parallel mode. This is a measure to ensure that there are no non-deterministic paths involved in the aggregation between GPUs.

The second issue is whether the results are consistent between two separate runs with different sizes of tensor parallelism. This remains an unresolved issue. Do not conflate the two.

10.2 With the Size Held Fixed

The SGLang development team, when outlining future challenges, states the following regarding tensor parallelism:

Tensor Parallelism: TP1 and TP2 are deterministic due to consistent floating-point addition
order; larger TP setups require modifications to reduce kernels for determinism.

⚠ This statement sits in the future-work section. This describes the current implementation status and does not represent a permanent specification. When working on this, it is necessary to verify the latest status.

10.3 Across Different Sizes

This area has dedicated research. The paper arXiv:2511.17826, initially posted on November 21, 2025, is titled Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch.

The paper positions the problem as follows:

While prior work has addressed batch-size-related nondeterminism through batch-invariant
kernels, determinism across different TP sizes remains an open problem, particularly in RL
settings, where the training engine typically uses Fully Sharded Data Parallel (i.e., TP = 1)
while the rollout engine relies on multi-GPU TP to maximize the inference throughput, creating
a natural mismatch between the two.

The key point is that the issue arises structurally within reinforcement learning configurations. The training side utilizes fully sharded data parallelism, effectively operating at a parallelism degree of 1, while the rollout side uses tensor parallelism across multiple GPUs to maximize throughput. The two are simply running at different parallel sizes to begin with.

The proposed method aims to create primitive matrix multiplication and reduction operations that consistently produce the same bitwise results, regardless of the parallelism degree, by aligning the order of reduction operations both within and outside the GPUs using a unified, hierarchical binary tree structure. The authors state that it is implemented in Triton and integrated into the vLLM framework alongside fully sharded data parallelism.

⚠ This is a proposed method, not a standard feature. As of the verification date for this article, no statement guaranteeing determinism across different tensor parallel sizes appears in the official documentation of the inference engines named above.

10.4 What This Means in Practice

Regardless of the direction of this research, there are concrete steps you can take now.

When comparing the two implementations, it is essential to ensure that the tensor parallelism size is consistent. This aligns with the requirement, discussed in Section 7.5, to match the software and hardware configurations.

Furthermore, it is crucial to record the degree of parallelism alongside the evaluation results. Without that record, when you compare the numbers later you cannot tell whether the difference came from the model or from the configuration. The items to be recorded are detailed in Section 12.5.

11. Making Everything Deterministic May Be Over-Constrained

Up to this point, the only means of reaching determinism this article has looked at is replacing the kernel. Since the start of 2026, that approach has itself come under review. ⚠ These are proposals, not product features you can adopt today. The means a reader can choose today are the ones in Chapter 7.

11.1 The Objection That the Cost Is Fixed

The paper arXiv:2601.17768, first submitted on January 25, 2026, is titled LLM-42: Enabling Determinism in LLM Inference with Verified Speculation.

The paper identifies two existing methods and outlines the drawbacks of each. While disabling dynamic batching can achieve determinism, it significantly reduces throughput. Regarding the approach of making kernels batch-invariant, the paper states:

Another approach is to make kernels batch-invariant; however, this tightly couples determinism
to kernel design, requiring new implementations. This coupling also imposes fixed runtime
overheads, regardless of how much of the workload actually requires determinism.

The bone of the objection is that determinism gets tightly coupled to kernel design, and that the price of that coupling is a fixed cost. Even if only 1 percent of server traffic requires determinism, changing the kernel will slow down the remaining 99 percent.

The proposed method borrows from the concept of speculative decoding. It generates tokens using a fast, non-deterministic path, and then ensures determinism through lightweight verification and rollback processes. The verification component recreates candidate tokens under a fixed-shape reduction plan, confirming that only those that produce consistent results are finalized, while others are rolled back. The claim is that, because it can largely leverage existing kernels, the additional cost is proportional to the volume of traffic that requires determinism.

11.2 Flips Are Sparse

The paper arXiv:2605.30218, first submitted on May 28, 2026, is titled MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference.

The paper, like this article, starts from the same observed phenomenon.

Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can
emit different tokens when decoded alone or inside a larger batch.

What this paper measured is how often a flip actually happens. Across five models, it counts what share of synchronous decode steps carry a batch-induced token flip. The results show 0.48 percent for Llama-3.1-8B on the MATH500 dataset, and every model tested stayed between 0.3 and 1.3 percent across the MATH500, GSM8K, and HumanEval datasets.

In other words, flips do not happen at most steps. And as Section 3.2 noted, they concentrate at steps where the gap between the top logit and the second is small. The paper reports that key and value perturbations stay flat right up to a flip, while a small gap between the top two logits captures much of the flip risk.

The proposed approach aims to incorporate this observation into the conditions for triggering the verifier. It suggests continuing normal decoding in steps with larger differences, but only running the verifier in steps with smaller differences. The reported results indicate that using this approach, Llama-3.1-8B and Qwen2.5-14B achieved complete sequence-level determinism while triggering the verifier at rates of 18.56 percent and 15.05 percent, respectively. A comparison with a constantly-running verification system is described as follows:

reducing LLM-42's latency increment by 2.23x/1.99x relative to always-on verification

The report states that this resulted in a reduction of the latency increase caused by constant verification, to approximately 1/2.23 and 1/1.99 for the two models.

⚠ The same abstract also states a harder condition. For DSR1-Distill-Qwen-7B, the same approach achieves determinism, but the verifier trigger rate increases to 49.50 percent. Whether the trigger rate remains low depends on the specific model and dataset.

11.3 These Are Proposals, Not Established Facts

⚠ The three papers this article cites as proposals are all recent. The arXiv:2601.17768, arXiv:2605.30218, and arXiv:2511.17826 papers have initial publication dates of January 2026, May 2026, and November 2025, respectively. They should be considered as published proposals, not as established methods.

And they are not something a reader can adopt today. As of the verification date for this article, nothing indicates that these methods ship as standard features in the major inference engines. If you are considering adopting them, you will need to independently verify their implementation and maturity.

11.4 They Still Carry Design Implications

The premise these proposals share has practical value. Determinism is not all-or-nothing.

First, the share of traffic that needs it changes the design. Whether every request needs determinism or only the evaluation slice does changes what shape the cost should take. If everything needs it, paying a fixed cost is the honest move. If only a slice needs it, separating that slice is cheaper.

Second, the risky steps are not spread evenly. Flips concentrate where the logit gap is small. This characteristic can be applied not only to verification triggers, but also to evaluation design. It is possible that questions with outputs prone to fluctuation and those with more stable outputs can be identified in advance.

Third, determinism can be designed as a service level. You decide which paths, for how long, and at which of the three levels you guarantee sameness. This is the same decision Chapter 13 makes.

12. What You Cannot Do on Managed Inference

12.1 What You Cannot Control

Since users cannot specify how batches are formed, there is no way to influence numerical determinism within managed inference endpoints. This is because the user has no point of contact anywhere along the causal chain in Chapter 5.

Stage in the Chain of Causality	Controllable by the User?
Content of the user's request	Yes
Other requests processed concurrently	No
Batch formation and size	No
Kernel implementation and splitting strategy	No
Tensor parallelism size	No
Engine version and hardware generation	No
Sampling parameters	Yes, but they reach only the sampling layer.

You touch only the two ends, and everything in between is out of sight. This is not a defect. It is what a managed service is. In exchange for having the operations taken off your hands, you give up the right to specify the internals.

12.2 Exact-Match Regression Testing Does Not Work

One design consequence follows from this fact. For managed inference, where the batch configuration cannot be controlled, you cannot build a mechanism that judges regressions by exact output match.

Saying you cannot build it does not mean it never runs. The issue is that it might work sometimes. It might pass during periods of low traffic, but fail when the load increases. A successful pass does not guarantee correctness, and a failure does not necessarily indicate a regression.

And this is not a matter of choice. It is a matter of premise. The methods described in Chapter 7 are only viable for self-hosted environments. By choosing a managed service, bitwise identity has already dropped off the list of options.

12.3 The Remedies the Existing Articles Already Give

The existing articles on this site already describe the right way to handle this situation. This article does not rewrite them.

The Amazon Bedrock Model Evaluation Practical Guide explains that evaluation metrics are not always deterministic with each run, that the scores from judge models can also vary from run to run, and that gate thresholds should be set with sufficient margin to avoid failing builds based on variations between observed runs.

The Agent Reliability Engineering Design Guide recommends comparing distributions rather than individual runs, using the final state rather than the trajectory for judgment, and confining deterministic testing to only the essential components.

What this article adds is the reason those remedies become necessary. The variability is not due to the models being unpredictable; it's because the way batches are constructed changes. Understanding this reason allows for a different approach to designing solutions. For example, the degree of variability often correlates with the load. Therefore, baseline measurements are only meaningful if taken under conditions that closely resemble actual load scenarios.

12.4 The Line Between Reproducible and Bitwise Identical

The line here is thin, so it is worth drawing explicitly.

LLMOps Observability and Evaluation Architecture on AWS correctly identifies that offline evaluation is reproducible as a characteristic suitable for a CI gate. This statement is accurate. It means that the input holds still. By using the same golden dataset and candidate configurations, any score changes can be attributed to actual variations.

It is not claiming bitwise identity. In fact, the same article treats a nondeterministic judge model as a failure mode, and prescribes sampling several times and taking a majority, plus measuring the judge model's variance explicitly. That prescription is the right one.

This article supplements that discussion with a single point. The document mentions using a lower temperature or zero temperature as one way to reduce variability. Lowering the temperature is an effective method for reducing the randomness of sampling. However, as Section 9.2 showed, it is not a guarantee of sameness. Even with a temperature of zero, variability may persist, and that is not a configuration error. The cause sits in the numeric layer. Therefore, the multiple samples and variability measurements that the document simultaneously recommends are not a replacement for temperature settings; they should be used in conjunction.

12.5 Metadata Worth Recording

Since exact matching is not possible, the validity of comparisons must be ensured through detailed records. When documenting evaluation results, it is helpful to include the following items, as they can assist in identifying the root causes of discrepancies later on.

Item to Record	Reason Required
Model Identifier and Version	The identifier changes as the version number increases.
Inference Engine and Version	Same as above. If managed, record that it is not possible to record this information.
Tensor Parallelism Size	See Chapter 10. Discrepancies can occur when this parameter differs.
Whether Deterministic Functionality is Enabled	Ensure that results with deterministic functionality enabled are not compared with those without.
Attention Backend	Switching determinism on narrows which backends you can use.
Load Conditions at Measurement Time	The degree of variation often correlates with the load.
Sampling Parameters and Seed	Dependent on conditions in lower layers.
Measurement Date	Useful for referencing original data.

In managed environments, it may not be possible to record the first few items in this table. Recording the fact that they cannot be recorded is itself worth doing. It is the clue you will want when you later chase down where a difference came from.

13. Where Determinism Is Needed and Where It Is Not

This article does not advocate for determinism. As Chapter 8 showed, there are costs. As Chapter 12 showed, there are positions from which the choice is not available. Therefore, what is ultimately needed is a process for determining whether to pursue a particular course of action, based on one's individual circumstances.

13.1 The Order of the Decision

There are three branching paths, and the order in which they are considered is deliberate.

The first branching path concerns the purpose. Which of the three levels in Chapter 2 you need is settled here at the same time. If you need to isolate the effects of a change, you need bitwise identity. If you only want to measure how good the model is, distribution equivalence is enough. Discussing methods before defining your objective risks making the method itself the objective. Furthermore, the first level is needed far less often than one might think. If distribution equivalence is enough, it's often cheaper and provides more information to use multiple samples for statistical analysis.

The second branching path asks whether the solution is even within reach. If you are using managed inference, the first level is not an option. If this path is closed, you will need to steer your design towards the direction outlined in Chapter 12.

The third branching path asks whether you can afford the cost. The time goes to roughly 1.2 to 2.1 times what it was, constraints appear on which features you can use, and the range of supported hardware and models narrows. The origins and composition of this cost are detailed in Chapter 8. If you cannot afford it, do not pursue it. That is not a defeat.

13.2 Where It Is Needed

The official vLLM documentation lists four scenarios where this feature is particularly important.

Framework debugging: Deterministic outputs make it easier to debug issues in the inference
framework, as the same input will always produce the same output regardless of batching.
Model debugging: Helps identify issues in model implementations by ensuring consistent behavior
across different batch configurations.
Reinforcement Learning (RL): RL training often requires deterministic rollouts for
reproducibility and stable training.
Large-scale inference systems: Systems that use vLLM as a component benefit from deterministic
behavior for testing, validation, and consistency guarantees.

The descriptions on the SGLang side also point to similar areas: reinforcement learning, testing and debugging, and production reliability.

These scenarios share one structure. Every one of them is a situation where you need to isolate the effect of a change. The table below adds one row that the official lists do not carry.

Scenario	What needs to be isolated?
Inference Engine Debugging	Determining whether a change in output is due to your modifications or inherent variability.
Model Implementation Debugging	Verifying that the ported model implementation produces the same numerical results as the reference implementation.
Reinforcement Learning	Ensuring that the sampling side and the training side are evaluating the same policy.
Large-Scale System Validation	Confirming that higher-level tests remain stable when the system is integrated as a component.
A change to the kernel, the quantization, or the speculative decoding (added by this article)	Whether the output moved because of the change, or because of the batch.

The final row directly connects to the other two articles in this series. When you make changes to accelerate inference or reduce the precision of weights, you are likely to want to measure their impact. If the foundation upon which you are measuring is unstable, you will not be able to distinguish between the signal and the noise. It is in this context that determinism first gains meaning as a measurement tool.

This category also includes situations where you need to demonstrate that the same input consistently produces the same output, for regulatory compliance or audits. However, caution is advised. As described in Section 7.5, identity across different versions and hardware is not guaranteed. Therefore, you can only demonstrate reproducibility within a specific environment, not absolute, permanent reproducibility. You should first determine whether the audit requirements call for one or the other.

13.3 Reinforcement Learning Changes the Nature of the Problem

Of the four, reinforcement learning is the one with a different character. For the other three, the absence of determinism makes things hard to measure. In reinforcement learning, the absence of determinism can break the training itself.

The original source introduces this point as an observation by researchers.

As researchers have noted, the different numerics between training and inference implicitly
turns our on-policy RL into off-policy RL.

The structure is as follows: the training side, which updates the policy, and the sampling side, which collects experience, typically operate with separate implementations. The training side runs with fully sharded data parallelism, while the sampling side runs with an inference engine. If the numerics of the two differ, the policy the training side assumes and the policy that actually produced the samples are different objects. If the learning rules are based on the assumption that the policies are identical, this discrepancy will accumulate as an uncorrected error.

The original source also addresses the order of operations.

Of course, it is impossible to get bitwise identical results between training and inference if
we can't even get bitwise identical results from two identical inference requests.

Determinism on the inference side is required first. It is impossible to align inference and learning if even two inferences do not produce consistent results.

The experimental setup described in the original source utilizes a verifiable reward-based reinforcement learning framework. The dataset is Bigmath, the initial policy is Qwen 2.5-VL instruct 8B, and the maximum rollout length is 4096. Three behaviors were reported.

Configuration	Reported Behavior
No out-of-policy correction using importance weighting	The learning process collapses, with a sharp increase in loss around Step 318, accompanied by a corresponding spike in KL divergence.
Out-of-policy correction using importance weighting	The learning process proceeds stably. KL divergence remains around 0.001, occasionally experiencing brief spikes.
The sampler and the trainer agree bitwise	The run is fully on policy, and KL divergence remains flat at 0.

⚠ This experiment is from the original source, not from the author's own measurements. Read the numbers together with the whole configuration.

SGLang also reports achieving reproducible learning by combining it with a reinforcement learning framework. They state that determinism on the inference side alone is not sufficient; settings are also required on the training side. These include downgrading a version of FlashAttention to ensure deterministic backpropagation, enabling the training framework's deterministic mode, fixing environment variables related to collective communication algorithms and numerical libraries' workspaces, and enabling PyTorch's deterministic algorithm. Determinism on the inference side is a necessary but not sufficient condition.

⚠ Reproducibility on the training side is outside the scope of this article. What is described here only covers how determinism connects to the inference side.

13.4 Where It Is Not Needed

In the following scenarios, there is no need to pursue determinism.

First, interactive use. A response a person reads does not have to be identical every time. A slightly different wording for the same question can even read as more natural. There is no reason to pay the cost of time here.

Second, evaluation that is trying to measure quality itself. If you want to know how good a model is, the distribution over several outputs carries more information than one fixed output. How stable it is disappears the moment you fix the output. The remedies from the existing articles cited in Chapter 12 point exactly this way.

Third, paths where production throughput matters. You would be paying the Chapter 8 cost across all production traffic. The projects themselves recommend the feature mainly for debugging and reproducibility.

Fourth, the stage where the cause of the variation has not been pinned down yet. There may be reasons for the output to change beyond a lack of batch invariance. The prompts might contain timestamps or identifiers. The context being retrieved may be different. The model version may be updated. Even if you enable a deterministic kernel, these issues will not be resolved. It is faster to first identify and isolate the underlying cause.

13.5 Taking It Only Where You Need It

You do not need to think in terms of all or nothing. From a practical standpoint, the most cost-effective approach is to isolate only those paths that require deterministic behavior.

For evaluation and regression testing, it's best to create a separate endpoint with determinism enabled. In production, traffic should be handled in the standard mode. With that arrangement, the range over which you pay the cost closes down to the run time of the evaluation.

However, there's an important caveat. Metrics measured on the evaluation endpoint may not directly reflect the behavior in production. Switching determinism on changes both the kernel underneath and the features you can use. Therefore, this configuration allows you to measure the relative impact of changes, but not the absolute performance in a production environment.

The proposal introduced in Chapter 11 can be interpreted as an attempt to implement this separation at a more granular level. The concept of incurring costs proportional to the volume of requests aligns with the principle of isolating specific paths.

14. Failure Modes and Anti-Patterns

What follows restates the article as a list of things not to do.

First, treating a temperature of zero as a guarantee of determinism. This only applies to the sampling layer. As noted in Section 9.2, the model provider explicitly states that it does not guarantee complete determinism.

Second, concluding that the system is deterministic because two runs matched. It is possible that the same batch size was selected in both runs. See Section 6.3. Testing should involve varying the batch size.

Third, reading an unstable test as a quality problem in the model. The model has not changed. What changed is the set of requests in flight alongside it. Get the cause wrong and swapping the model will not fix it.

Fourth, mixing a time-increase rate and a throughput-decrease rate in the same comparison. See Section 8.1. The same fact can be written both ways. Check which one the source used, every time.

Fifth, putting the CUDA graphs speedup and the deterministic-mode slowdown in the same table. See Section 8.5. These are separate measurements, and one does not negate the other.

Sixth, writing your own list of supported models or backends into an article or a design document. See Section 7.4. There is a real case where the premise moved in under a year. Hold criteria instead, and go to the official pages for the list.

Seventh, treating a beta feature as GA. vLLM's batch invariance is currently marked as a beta feature. Requirements may change as the feature's status evolves.

Eighth, treating a proposal from a paper as a product feature. See Section 11.3. Independently verify the implementation and its level of maturity.

Ninth, comparing measurements taken with determinism on against measurements taken with it off. The kernels used are different, so performance and output cannot be compared. Maintain separate records indicating whether determinism was enabled or disabled (see Section 12.5).

Tenth, comparing evaluation results without recording the tensor parallel size. See Chapter 10. Cross it and it breaks.

Eleventh, leaving the feature enabled across all production traffic. That runs against what the projects themselves recommend. Limit its use to only the necessary paths (see Section 13.5).

Twelfth, reporting the phenomenon as an information leak. See Section 6.2. Nothing is leaking. A wrong report sends the whole remediation in the wrong direction.

15. Frequently Asked Questions

Q. Will setting temperature to 0 guarantee deterministic results?

No. While the sampling layers become deterministic, the underlying logits (input values) still change. The ranking of tokens with small differences between the top candidates can shift, and this change propagates, altering the subsequent generation. Anthropic's Messages API documentation also explicitly states that even with a temperature of zero, complete determinism is not achieved.

Q. Will providing a seed resolve the issue?

No. A seed only controls how random numbers are generated; it does not touch the kernel's reduction order. Conversely, using a seed with batch-invariant kernels enabled can produce reproducible sampling, even when the temperature is greater than zero.

Q. Is the cause related to GPU parallel execution?

No. Even when performing the same matrix multiplication with the same data repeatedly (e.g., a thousand times), the results are bitwise identical. The root cause is that the kernel changes its reduction splitting strategy based on the batch size. Importantly, this non-determinism is not specific to GPUs; it originates from the same source in inference endpoints provided by CPUs and TPUs as well.

Q. If other requests affect your output, is something leaking?

No. Nothing is leaking. What changes are the paths through which rounding errors propagate. The dependency on other requests is limited to a single integer: the batch size. The original source explicitly states that this is not a case of information leakage.

Q. Can the output change even when only a single request is in flight?

Yes. With prefix caching on, the computation varies with whether part of the input sits in the cache. With chunked prefill on, where a long prompt gets cut depends on the chunk budget at that moment.

Q. How much slower is it?

The published figures span a range. In the Thinking Machines Lab measurement, the time to process 1000 sequences on Qwen-3-8B goes from 26 seconds to 42 seconds. In the SGLang measurement, the time increases by an average of 34.35 percent across three workloads on the FlashInfer and FA3 backends. The two were taken under different configurations, so they cannot be compared directly. Read them as an indication of the order of magnitude.

Q. Can determinism be achieved with managed inference?

No. This is because users have no control over how batches are formed, the kernels used, the degree of parallelism, or the version of the engine. The design moves toward comparing distributions, allowing tolerances, and using judge models.

Q. How should regression tests be designed?

The fundamental approach is to change the subject of evaluation. Instead of seeking exact output matches, focus on evaluating the final state or invariant conditions. Run multiple executions and compare the resulting distributions. When setting thresholds, ensure they provide a margin greater than the observed variance between executions. Specific implementation details can be found in existing articles on model evaluation and agent reliability.

Q. With determinism on, does the output stay the same if you swap the model?

No. Determinism only guarantees invariance across different batch sizes. If the model version, the engine version, the GPU generation, or the tensor parallel size changes, the numbers can change with it.

Q. Is it possible to configure the system to make only the evaluation endpoints deterministic?

Yes. In practice, this is often the most cost-effective configuration. However, performance metrics measured in deterministic mode do not represent the actual performance in a production environment. They only measure the relative impact of changes.

Q. If the size of the tensor parallelism is changed, will the output remain the same?

No. Holding the tensor parallel size fixed is addressed, and Section 10.2 shows how far that reaches. Crossing sizes remains an open problem. Ensure that both executions being compared have the same degree of parallelism.

Q. Does the same reasoning apply to output changes caused by speculative decoding or quantization?

No. That is a different matter. This article addresses the phenomenon where the output changes even without any intentional modifications. The impact of deliberate changes made for speed or size is a subject for separate articles. However, this article provides a foundational framework for measuring those impacts.

16. Summary

The cause of not getting the same output from the same input is not on the sampling side. The inference engine's kernel changes its splitting strategy for reduction based on the batch size. When the split changes, the addition order changes, the rounding changes, and the logits change. Even under greedy decoding, a different token can then be selected.

Parallel execution by itself is not the cause. Repeating the same matrix multiplication yields bitwise identical results every time. The forward pass of LLMs contains very few operations where the order is non-deterministic.

In production environments, the batch size fluctuates due to load. This means that even if your request is the same, a change in the other requests in flight alongside it can move the output. This does not constitute an information leak. Only the path through which rounding errors propagate is changing.

Sameness comes in three levels. They are bitwise identity, distribution equivalence, and quality parity. The three sit inside one another, and you cannot climb from a weaker level back to a stronger one. Every time a document uses the word deterministic, you have to check which of the three it is claiming.

Determinism can be bought, and it is paid for in time. In vLLM it is switched on with an environment variable, and in SGLang with a startup flag. As of the verification date, vLLM is in beta, and SGLang is limited to three attention backends. The published cost is that the same work takes roughly 1.2 to 2.1 times as long, depending on the implementation and the configuration.

And it does not have to be bought in every situation. Determinism is truly necessary when you need to isolate the effects of changes. This applies to debugging inference infrastructure and model implementations, reinforcement learning, and evaluating a change to the kernel, the quantization, or the speculative decoding. It is not required for interactive use or for evaluations that measure quality itself.

If you are using managed inference, this option does not exist from the outset. You cannot control how batches are assembled. The design decision there is to give up on judging by exact match and move to distributions, tolerances, and judge models. What this article supplied is the reason that decision becomes necessary. The remedies themselves are held by the existing articles.

17. References

• Defeating Nondeterminism in LLM Inference - Thinking Machines Lab
• batch_invariant_ops - thinking-machines-lab on GitHub
• Batch Invariance - vLLM Documentation
• Batch Invariance - vLLM Ascend Documentation
• Deterministic Inference - SGLang Documentation
• Towards Deterministic Inference in SGLang and Reproducible RL Training - LMSYS Org
• LLM-42: Enabling Determinism in LLM Inference with Verified Speculation - arXiv:2601.17768
• Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch - arXiv:2511.17826
• MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference - arXiv:2605.30218
• Inference using Converse API - Amazon Bedrock User Guide
• InferenceConfiguration - Amazon Bedrock API Reference
• Using the Converse API - Amazon Nova User Guide
• Messages - Anthropic API Reference
• Amazon Bedrock Model Evaluation Practical Guide - Automatic Metrics, LLM-as-a-Judge, RAG Evaluation, and CI/CD Quality Gates
• Agent Reliability Engineering Design Guide - Retries, Loop Detection, Timeout Budgets, and Human Escalation for AI Agents
• LLM API Parameter Compatibility Reference - Anthropic, OpenAI, Google Gemini, and Amazon Bedrock
• Self-Managed LLM Inference on Amazon EKS - Serving Open-Weight Models with vLLM, Neuron/GPU, and Karpenter
• Disaggregated Prefill and Decode for LLM Serving on AWS - The KV Transfer, the Routing Threshold, and What Disaggregation Does Not Fix
• LLM Output Verification Patterns - Grounding Checks, Self-Verification, Cross-Model Review, and Citation Enforcement
• LLMOps Observability and Evaluation Architecture on AWS - Tracing, Metrics, and Automated Evaluation Gates with CloudWatch and OpenTelemetry
• Amazon Bedrock Inference Throughput and Latency Optimization - Quotas, Provisioned Throughput, Latency-Optimized Inference, Prompt Caching, and Intelligent Prompt Routing
• AWS Lambda Durable Functions Practical Guide - Checkpoint and Replay Determinism and When to Use Step Functions Instead

References:
Tech Blog with curated related content