hidekazu-konishi.com

LLM Weight Quantization on AWS - Weight-Only and Activation Formats, What Each Costs in Accuracy, and When the Platform Quantizes for You

First Published: 2026-08-24
Last Updated: 2026-08-24

You have decided to run an open-weight model on AWS. The checkpoint is in hand. It does not fit on the GPU you wanted to put it on. Or it does fit, but you have the feeling that a smaller instance would be enough. You look into it and arrive at the word quantization, with names like INT4-AWQ, FP8, and INT8-SmoothQuant lined up next to each other. Which one to pick, and what picking it costs you, is written down nowhere.

This article aims to fill that gap.

Quantization itself is not complex. It simply involves reducing the number of bits used to represent the model's weights, thereby reducing the model's memory footprint. The challenge lies in discerning what is being reduced, what is being lost in return, and whether a particular choice is truly viable for your specific configuration.

The skeleton of the conclusion comes first, in four parts.

First, one word covers at least three separate targets: the weights, the activations, and the KV cache. And a method that reduces only the weights is a different kind of technique from one that reduces the weights and the activations both. Which of the two works depends on whether the situation is stalled on memory bandwidth or on arithmetic. The AWS documents present them in the same column, so you have to be able to read that sitting next to each other does not make them the same kind of thing.

Second, AWS itself states the accuracy cost at four different strengths. The Developer Guide says only that accuracy may drop and gives no magnitude, the AWS Neuron documentation says it drops slightly, the Amazon Nova inference container User Guide says the impact on the weights is minimal, and the same guide says the KV cache brings minor numerical differences. Notably, only the last of the four explicitly tells the reader to check it on their own workload. None of these statements are incorrect; each is accurate within the context it describes. The key for the reader is to identify which statements apply to which specific scenarios.

Third, the available formats are the product of four constraints: the format itself, the accelerator generation, the serving stack, and the model. Only where those four overlap is there a format you can actually pick. The Developer Guide states this outright, in the form that the instance type you deploy to changes which optimization techniques you can choose.

Fourth, quantization is a choice, except when it stops being one. For certain combinations of model and instance type, AWS states outright that FP8 quantization of the weights is enabled automatically and can neither be disabled nor overridden. This is where the reader's premise that quantization is something you choose breaks down, and it is one of the two centers of this article.

All specifications and figures presented in this article have been verified against primary sources. The verification date is August 23, 2026. The primary source is assigned separately for each type of claim. What is available on AWS goes to the AWS official documentation, what a method actually does goes to the arXiv original, and how an inference engine handles it goes to the official vLLM documentation. This categorization itself is relevant to the core theme of this article, and will be addressed directly in Chapter 5.

Table of Contents

1. 1. Quantization Is a Choice, Except When It Is Not
2. 2. Three Different Things the Same Word Points At
3. 3. Why Making It Smaller Makes It Faster, and When It Does Not
4. 4. What the Representative Methods Actually Do
5. 5. What Each Document Says the Accuracy Costs
6. 6. The Paths on AWS
7. 7. Quantized Without Choosing It
8. 8. The Format Is Bound to the Hardware
9. 9. How to Measure What You Paid
10. 10. When Not to Quantize
11. 11. A Checklist for Choosing
12. 12. Failure Modes and Anti-Patterns
13. 13. Frequently Asked Questions
14. 14. Summary
15. 15. References

1. Quantization Is a Choice, Except When It Is Not

1.1 Intended Audience

This article is aimed at readers who have already decided to run open-weight models on AWS and are considering quantization because the model does not fit on the accelerator they want to put it on.

Readers may have encountered the names of various quantization formats, but may be unfamiliar with four key aspects. First, what each format gives up. Second, whether a particular format is actually a viable option for their specific infrastructure. Third, how to detect any degradation in accuracy resulting from quantization. And fourth, whether there is a path where quantization is already in effect even though they never chose it.

This article is specifically written for those involved in designing inference platforms and those responsible for deciding which models to deploy on which instance types. No prior knowledge of kernel development is required, and this article will not include any implementation code for kernels. It will also not derive the mathematical formulas behind quantization algorithms, nor will it serve as a tutorial on how to use quantization libraries.

1.2 What This Article Does Not Cover

Vector quantization is outside the scope of this article. While the same terminology may be used, it is an entirely different technology. Discussions about compressing embedding vectors to reclaim index memory are available in Vector Database Selection on AWS. The quantization discussed in this article specifically concerns model weights and activations. This distinction will be clarified again at the beginning of Chapter 2.

KV cache quantization will be addressed briefly as a separate topic, with further details deferred. Discussions about compatibility in architectures that separate prefill and decoding are found in Disaggregated Prefill and Decode for LLM Serving on AWS.

This article also does not address the phenomenon where output changes despite no modifications. That topic is covered in this series' Reproducible LLM Inference. Instead, this article assumes that phenomenon and focuses on what happens when you intentionally degrade accuracy.

The methodology of speculative decoding is not covered either. Those are detailed in Speculative Decoding and Draft Models on AWS. This article does not include discussions about quantizing draft models.

How to run the job that measures whether accuracy dropped is not covered either. Amazon Bedrock Model Evaluation Practical Guide holds that. This article writes only what should be compared and delegates the rest.

The decision of whether or not to self-host is also not discussed. That decision is addressed in Self-Managed LLM Inference on Amazon EKS. This article begins with the assumption that the reader has already chosen to self-host.

Price comparisons are not covered either. The motivation for quantization is the reduction in required accelerator memory, which in turn allows for the selection of smaller instance types. This article will cover the structure of that process, but will not delve into specific pricing details.

Finally, this article does not cover quantization during training. Mixed-precision training is a separate topic. This article focuses on converting already-trained weights.

1.3 What the Existing Articles Already Hold

Holding the following correspondence in mind before you start reading will save some confusion. There are multiple instances where the same word refers to different things.

Existing Article	What That Article Holds	Relationship to This Article
Vector Database Selection on AWS	Vector indexing quantization. A discussion of compressing embedding vectors to reclaim resident memory.	The word collides completely. This is an entirely different technology. This article only addresses model weights.
Reproducible LLM Inference	Why the same input does not return the same numbers. Defines the three-level vocabulary.	This article does not redefine that vocabulary. It refers to it and uses it. The reason a drop in accuracy cannot be measured by exact match also comes from here.
Speculative Decoding and Draft Models on AWS	The methodology of speculative decoding. Holds the speculative decoding side of the inference optimization job.	A different technique under the same inference optimization framework. This article only addresses the quantization aspect.
Self-Managed LLM Inference on Amazon EKS	The pros and cons of self-hosting and serving configurations. Highlights quantization as a primary reason to choose self-hosting.	This article expands on that single point. It does not discuss whether or not to self-host.
Disaggregated Prefill and Decode for LLM Serving on AWS	Separating prefill and decoding, and KV transfer. Lists the data type of the KV cache as a mandatory requirement for hash compatibility.	This article only explains what that item means. It does not delve into handshake procedures or transfer modes.
Amazon Bedrock Model Evaluation Practical Guide	How to run evaluation jobs.	This article describes what should be compared, delegating the selection of which jobs to use.
LLM Output Verification Patterns	A layer for verifying the correctness of output.	This article does not delve into the concept of correctness itself.
AWS Custom Silicon History and Timeline and Amazon EC2 Instance Types History and Timeline	A timeline of when each generation was released.	This article does not create a timeline. It also does not create a table mapping generations to specific features.
Open-Weights LLM Release History and Timeline	A lineage of open-weight models.	This article does not compare the capabilities of different models.
Amazon Bedrock Model Catalog 2026	A catalog of models on the managed side.	This article does not address model selection on the managed side.

This division of labor is a premise of this article. Content the existing articles already carry is not rewritten here.

1.4 Verification Date and Primary Sources

This article assigns canonical sources to each type of technical claim and cross-references them. All verification dates are August 23, 2026.

Claim Type	Primary Source
What is available on AWS: supported formats, environment variables, default values, and constraints	Amazon SageMaker AI Developer Guide and API Reference, Amazon Nova User Guide, AWS Neuron documentation, AWS What's New
What a method does, and the reported impact on accuracy	The arXiv original, with the version stated
How an inference engine handles it: supported formats and the KV cache data type	Official vLLM documentation

This article is not a step-by-step guide. Specific flags to enable functionality are documented in the official documentation, and that content may change. This article aims to explain what those choices mean and to provide a framework for you to determine whether or not to make those choices in your own situation.

⚠ This article does not provide a list of supported models, nor does it provide a table of supported formats by instance generation. The official documentation holds both, and both move. What this article holds is the decision criteria. The lists themselves are delegated to the official pages.

2. Three Different Things the Same Word Points At

This chapter is short. One word is doing three separate jobs here, and the rest of the article assumes they have been told apart.

2.1 The Quantization in This Article Is the Model's Weights

When this article says quantization, it means representing the model's weights and activations with a data type that uses fewer bits.

The same word is used for an entirely different technology as well. Compressing embedding vectors to shrink the resident memory of a search index is also called quantization. Product quantization, scalar quantization, and binary quantization all fall under that heading. This site already covers that technology in Vector Database Selection on AWS.

While both use the same name, they differ significantly in their scope, purpose, and evaluation methods. Vector index quantization aims to preserve recall, while quantization of model weights aims to maintain the quality of the output. This article will not delve into vector index quantization at all.

From here on, quantization without further qualification means weight quantization.

2.2 Weights, Activations, and the KV Cache Are Separate Axes

Under the single word quantization there are at least three separate targets. The three are chosen independently, they take effect independently, and they break independently.

The first target is the weights. The model's parameters themselves are held at a lower bit width. What falls is the memory the model occupies while resident, and, as a second-order effect, the number of bytes read out of memory for every token produced. Since this involves converting a pre-trained model, the conversion process is performed only once.

The second target is the activations. The values that flow between layers are represented at a lower bit width. What falls is not memory but the path along which the matrix multiplication runs. Only when the weights and the activations are both in the same low-precision type can the arithmetic units built for that type be used at all. Quantizing the activations is therefore a technique that buys arithmetic, not one that buys memory.

The third target is the KV cache. The key and value tensors held during generation are kept at a lower bit width. What falls is the memory that piles up per request, and the result is more concurrent requests and longer contexts. Where it takes effect is different from weight quantization.

That the three are separate things shows up in the structure of the primary sources as well. The AWS Neuron NxD Inference Features Configuration Guide, for example, places two subheadings, Model Weight Quantization and KV Cache Quantization, under the section titled Quantization. The Amazon Nova SageMaker inference container uses separate environment variables, QUANTIZATION_DTYPE for the weights and KV_CACHE_DTYPE for the KV cache. vLLM also treats KV cache quantization as a separate feature, configuring it with a distinct flag called kv_cache_dtype.

The first two targets are the ones this article goes into. For KV cache quantization, this chapter cuts it away as a separate axis, and Chapter 9 takes up only what it means for a design. The point that a configuration separating prefill and decode requires the KV cache data type to match on both sides is held by Disaggregated Prefill and Decode for LLM Serving on AWS.

2.3 The Three-Level Vocabulary Carries Over Unchanged

Reproducible LLM Inference, in this series, splits the levels at which two systems can be called the same into three and gives each one a name. This article uses that vocabulary without defining it again.

Level	Name	What it claims
Level 1	bitwise identity	The same byte sequence comes back for the same request. A string comparison matches.
Level 2	distribution equivalence	The output is drawn from the same probability distribution. Individual outputs may differ.
Level 3	quality parity	Aggregated metrics cannot tell them apart. The distribution is allowed to differ.

The three are nested. If the first level holds, the second holds as well, and if the second holds, the third holds as well. The reverse does not.

Which level quantization claims is not in doubt. It claims Level 3.

Reducing the bit width of the weights will change the result of matrix multiplications. If the result changes, the probability distribution changes. Quantization therefore claims neither Level 1 nor Level 2. That is the decisive difference from those speculative decoding methods that claim Level 2 as a theorem. What is claimed on the speculation side is covered in Speculative Decoding and Draft Models on AWS.

This difference goes straight to practice. Line up the output from before and after quantization and the mismatch is known in advance. An exact match therefore does not work as a check. What to compare instead is Chapter 9.

⚠ This article does not apply the word lossless, in its compression sense, to quantization. Reversibility in the sense of restoring the original byte sequence exactly does not exist here. Information is gone the moment the precision drops.

2.4 W4A16 and W8A8 Are the Real Unit

The name of a format on its own does not say what was quantized. The papers handle this with a notation that puts the bit width of the weights and of the activations side by side.

• W4A16 refers to a configuration that uses 4 bits for weights and 16 bits for activations. Since the activations are not quantized, this represents quantization of only the weights.
• W8A8 refers to a configuration that uses 8 bits for both weights and activations. This indicates quantization of both weights and activations.

The original paper on AWQ refers to its target as W4A16, while the original paper on SmoothQuant refers to its target as W8A8. Even though both put the weights at 8 bits, W8A16 and W8A8 represent entirely different configurations. The former uses 8 bits for the weights only, while the latter uses 8 bits for both activations and weights.

This article will use this notation, supplementing the format names as needed. This is because simply listing the names can lead to confusion, otherwise weight-only methods and methods that cover both weights and activations end up in the same column.

The notation does not replace the explanation in the text. Even when a format is described as W4A16, the specific implementation of the 4-bit representation can vary depending on the method used. The notation simply indicates which components have been quantized.

3. Why Making It Smaller Makes It Faster, and When It Does Not

3.1 Decoding Is Stalled on Memory Bandwidth

Quantization does not help speed because the model is small in itself. It helps because the whole set of weights is read out of memory every time a single token is generated. Make it smaller and that read gets lighter.

The original paper on AWQ states this structure in its very first sentence.

the astronomical model size raises the hardware barrier for serving (memory size) and slows
down token generation (memory bandwidth)

The same paper explains the reason for choosing weight-only quantization as follows.

it not only reduces the hardware barrier (requiring a smaller memory size) but also speeds
up the token generation (remedies memory-bound workload)

AWS Neuron documentation also explains the same principle.

NxD Inference supports quantization, where model weights and data are converted to a smaller
data type to reduce memory bandwidth usage, which improves model performance.

In essence, halving the number of bits used for the weights reduces the number of bytes that need to be moved for each token. The computational workload remains the same. Only the amount of data being moved is reduced. And during the process of generating tokens one by one, the processor is idle, waiting for the data to arrive. Because the wait time is shorter, the process is faster.

The official vLLM documentation summarizes this trade-off in a single sentence.

Quantization trades off model precision for smaller memory footprint, allowing large models
to be run on a wider range of devices.

3.2 Quantizing Only the Weights Does Not Reach the Integer Path

This is the area most prone to misunderstanding. Even when using weight-only quantization, where weights are stored using 4 bits, the matrix multiplication itself is not performed using 4-bit operations.

The original paper on SmoothQuant, in its current version, states the following regarding weight-only quantization:

It converts the quantized weights to FP16 on the fly for matmul during inference and can
also lead to speed up due to the reduced data loading, especially for the generation stage
with batch size 1.

Weight-only quantization holds the values at low precision and converts them back to the original precision right before the multiply. What it gains is the reduction in data movement. The arithmetic itself still runs at the higher precision.

So, what is needed to accelerate the calculations? The same paper answers:

we need to quantize both weights and activations into INT8 (i.e., W8A8) to utilize the
integer kernels (e.g., INT8 GEMM), which are supported by a wide range of hardware

Only when activations are also represented using the same low-precision format does the computation enter the path that uses the integer arithmetic units. This is the definitive distinction between weight-only quantization and quantization that includes both weights and activations. The original paper on SmoothQuant describes its own approach as follows:

SmoothQuant has better hardware efficiency than existing techniques using mixed-precision
activation quantization or weight-only quantization.

In summary, weight-only quantization saves on memory. Quantizing both weights and activations leverages computational resources. Under the same word, what is being bought is different.

3.3 Dequantization Is Not Free

Converting what was held at low precision back again, right before it is used, takes time of its own. The AWS machine learning blog lists four benefits of quantization, and regarding improved decoding latency, it specifies the following condition:

Faster decoding latency - Because the decoding process is memory bandwidth bound, less data
movement from reduced weight sizes directly improves decoding latency, unless offset by
dequantization overhead.

AWS itself writes that the reduction in movement can be offset by the cost of dequantization. Which of the two wins depends on the implementation and the configuration. A result where quantization brought no speedup is therefore not an anomaly. It is one of the outcomes the structure allows.

The KV cache also has a similar structure. AWS Neuron documentation explicitly states that when the KV cache is quantized, the data must be converted back to its original data type before use. There is a back-and-forth process there as well: storing data in lower precision, then converting it back to higher precision for use.

3.4 There Are Regions Where It Does Not Get Faster

As the number of requests being processed simultaneously increases, the situation changes.

Once the weights have been loaded, the amount of computation performed per weight read increases as the number of requests processed using those weights grows. Beyond a certain point, the bottleneck shifts from memory bandwidth to computational capability. In areas where memory bandwidth is no longer the bottleneck, reducing the amount of data movement yields diminishing returns.

The original paper on SmoothQuant, again in its current version, states this structure explicitly, as a comparison against weight-only methods.

GPTQ may perform better at handling a small number of input tokens (1 in its experiments)
since the process is highly memory-bounded. In contrast, SmoothQuant may serve better with a
batching setting or for the context stage (i.e., when the number of processed tokens is more
significant).

The same paper states that the two configurations are orthogonal to each other.

Finally, we think the two settings are somewhat orthogonal.

Any question of which is faster comes with a condition attached, namely which region the measurement was taken in. The answer may vary depending on whether tokens are being output one by one, whether long inputs are being processed all at once, or whether numerous requests are being processed simultaneously.

It's important to note that the number of requests processed concurrently is not a value that readers can arbitrarily fix. In inference engines that utilize continuous batching, this number fluctuates based on the load. The underlying reasons for this are discussed in Reproducible LLM Inference.

This article does not present any actual performance measurements. The values the papers and the vendors report were each measured on a different implementation, a different configuration, and a different model. This article only describes the structure. If you require specific numbers, you will need to measure them using your own configuration.

4. What the Representative Methods Actually Do

The Amazon SageMaker AI Developer Guide lists three supported quantization formats: INT4-AWQ, FP8, and INT8-SmoothQuant. Although these three are listed together, they have distinct characteristics. This chapter will revisit the original sources to understand what each one does.

This chapter will also discuss GPTQ. GPTQ is not included in the above three methods. It is not an option in this particular workflow. It is covered anyway, for two reasons. First, the inference engine lists it among the formats it supports, so you can pick it on the path where you serve the model yourself. Second, the original paper on AWQ names GPTQ explicitly as its comparison. Without the two side by side, what separates one weight-only method from another does not come into view.

⚠ Do not read the coverage of GPTQ below as meaning that GPTQ is selectable in the inference optimization job.

4.1 GPTQ Compensates the Error With Second-Order Information

The original paper on GPTQ is arXiv:2210.17323, and the first version was submitted on October 31, 2022. The abstract of that version describes the method as follows.

a new one-shot weight quantization method based on approximate second-order information

GPTQ is a post-training quantization method that targets the weights only. The conversion happens once, and no retraining is involved. Regarding bit width, the same abstract states:

reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation
relative to the uncompressed baseline

⚠ That negligible is a word the paper applied to its own experiments. It specifies which models were used, with which datasets, and using which metrics. This article does not reproduce this term as a guarantee for the reader's specific configurations.

GPTQ requires calibration data. The first version of the original paper explicitly details the calibration data used in its experiments.

Our entire GPTQ calibration data consists of 128 random 2048 token segments from the C4 dataset

Furthermore, the same paper itself notes that this calibration data can influence the interpretation of the results. The table presenting the experimental results includes a disclaimer stating that the calibration data was taken from the training side of the evaluation dataset, meaning the results are not entirely zero-shot. The source of the calibration data is a choice that has implications later on.

4.2 AWQ Uses the Activations to Decide Which Weights to Protect

The original paper on AWQ is arXiv:2306.00978, and the first version was submitted on June 1, 2023. As the name suggests, it is a weight-only quantization method that takes the activations as its cue.

a hardware-friendly approach for LLM low-bit weight-only quantization

The initial observation that led to this approach is:

protecting only 1% of salient weights can greatly reduce quantization error

The observation it starts from is that weights are not all equally important. So how do you tell which ones matter? That is where the name of the method comes from.

search for the optimal per-channel scaling that protects the salient weights by observing the
activation, not weights

Instead of looking at the distribution of weights, AWQ identifies the channels of weights to preserve by examining the distribution of activations. While it's a weight-only quantization method, it utilizes activations in its decision-making process. The Amazon SageMaker AI Developer Guide explicitly uses the term weight-only for this characteristic.

4.3 What Separates Two Weight-Only Methods Is Not the Bit Width

Both GPTQ and AWQ are quantization methods that focus solely on the weights, primarily targeting a quantization level around 4 bits. So, what is the difference between the two?

The original paper on AWQ, in its current version, explicitly names GPTQ as the closest prior work, summarizing it as a method that compensates for the error using second-order information. It then continues:

However, it may overfit the calibration set during reconstruction, distorting the learned
features on out-of-distribution domains

And it describes its own characteristics as follows:

AWQ does not rely on any backpropagation or reconstruction, so it can well preserve LLMs'
generalization ability on different domains and modalities, without overfitting to the
calibration set

The key distinction between these two methods that focus solely on weights is not the number of bits, but rather the magnitude of the impact that calibration data has on the results.

This represents a design choice for the user. The choice of calibration data can determine the areas where the quantized model performs best. If you calibrate using your actual production traffic, the model will perform strongly within that distribution. However, there are no guarantees about its behavior when it encounters data outside of that distribution.

⚠ This is an area that AWS documentation does not address. The Developer Guide briefly mentions INT4-AWQ with a single line of explanation, simply listing it among supported formats. The question of how to select calibration data is not discussed.

4.4 SmoothQuant Moves the Difficulty From the Activations to the Weights

The original paper on SmoothQuant is arXiv:2211.10438, and the first version was submitted on November 18, 2022. Its target is different from the two before it.

a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ)
solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs

Both the weights and the activations go to 8 bits. That is the configuration Chapter 3 described as the one that gets onto the low-precision arithmetic path.

The trouble is that activations do not quantize as cleanly as weights. The original paper explains this reasoning as follows:

We observe that systematic outliers appear at fixed activation channels.

And it proposes the idea of transferring that difficulty.

smooths the activation outliers by migrating the quantization difficulty from activations to
weights with a mathematically equivalent transformation

Instead of eliminating the difficulty, it transfers it. The method subtracts the difficulty from the activations and adds that amount to the weights. As a whole, this is a mathematically equivalent transformation, so the transformation itself is not an approximation. The approximation arises when both, after the transfer, are rounded to 8 bits.

To determine how much to transfer, it's necessary to understand the range of values the activations take. The original paper, in its current version, describes this process as follows:

The smoothing factor s is obtained on calibration samples and the entire transformation is
performed offline.

This method also requires calibration data. Combining this with the previous two methods, all three techniques discussed in this chapter involve collecting statistics by passing some input through the model before transforming the trained weights. The question of how to select the calibration data is a question that invariably arises after choosing a method.

4.5 Activations Are Hard Because of the Outliers

Why is activation quantization particularly challenging? Because extremely large values repeatedly appear within specific channels.

Data types with a limited number of bits can represent a narrow range. When a single, exceptionally large value is included within that range, the only recourse is to reduce the scale, effectively coarsening the resolution. Reducing the scale inevitably diminishes the precision for the vast majority of the remaining values. The outliers themselves are not the problem. The problem is that they take over the scale.

The original paper on SmoothQuant states that this phenomenon is linked to the model's size. It describes how, when the number of parameters exceeds 6.7 billion, systematic outliers with large values begin to appear in the activations. Therefore, the difficulty in quantizing activations is not apparent in smaller models, but emerges in larger models.

4.6 FP8 Is the Name of a Format, Not the Name of a Method

The Amazon SageMaker AI Developer Guide describes three supported formats as follows, using the original terminology.

INT4-AWQ - A 4-bit data format. Activation-aware Weight Quantization (AWQ) is a quantization
technique for LLMs that is efficient, accurate, low-bit, and weight-only.
FP8 - 8-bit Floating Point (FP8) is a low-precision format for floating point numbers. It
balances memory efficiency and model accuracy by representing values with fewer bits than
standard FP16 floating point format.
INT8-SmoothQuant - AN 8-bit data format. SmoothQuant is a mixed-precision quantization method
that scales activations and weights jointly by balancing their dynamic ranges.

When comparing the three, you will notice that only the middle one has a different characteristic.

INT4-AWQ and INT8-SmoothQuant both consist of a data type name and a method name. The Developer Guide itself describes the former as weight-only and the latter as activations and weights. In other words, the distinction between using only weights or using both weights and activations is already documented within AWS's documentation. While they may appear similar when simply looking at the columns, reading the descriptions clarifies the difference.

On the other hand, FP8 does not have a method name. It is simply the name of an 8-bit floating-point data type, and it does not specify the procedure used to convert data to that format.

Furthermore, FP8 itself is not a single entity. The AWS Neuron data types documentation treats FP8 as several encodings that divide range and precision differently. There are encodings that prioritize a wider representable range by allocating more bits to the exponent, and encodings that prioritize precision by allocating more bits to the mantissa. Under the same name, FP8, the priority differs.

Therefore, avoid making comparisons based solely on the format name. Asking which is better, INT4-AWQ or FP8, will not yield a clear answer. One is a name that specifies both the data type and the method, while the other is simply a container name.

4.7 The Four Rearranged by What Each Reduces

What this chapter covered, rearranged by what each one reduces and what it buys in return. What goes invisible once they are lined up in the same column is made visible here.

Method or format	What it reduces	What it mainly buys	Does it need calibration data?	Source
GPTQ	The weights only	Memory, and the volume read per token	Yes. The original paper states exactly what calibration data its experiments used.	arXiv:2210.17323
AWQ	The weights only, but the weights to protect are decided from the activations	Memory, and the volume read per token	Yes. However, the original paper claims that it is less prone to overfitting the calibration data, as it does not perform reconstruction.	arXiv:2306.00978
SmoothQuant	Both the weights and the activations	The low-precision arithmetic path	Yes. The original paper states that the smoothing factor is obtained on calibration samples.	arXiv:2211.10438
FP8	Specifies the data type only. What gets reduced depends on the configuration	Depends on the configuration	Depends on the configuration	The data type specification

Only the fourth row is different in nature. The first three are names of methods, while the fourth is the name of a data type. They are listed in the same table because AWS documentation presents them in the same column.

⚠ This table does not indicate the superiority of any method. Which method is suitable depends on whether the bottleneck is memory bandwidth or computational capability. Beyond that, it depends on whether your accelerator generation carries the format at all, whether the serving stack implements it, and whether the official documentation lists it for that model. Those last three, plus the format itself, make the four that Chapter 8 takes up.

5. What Each Document Says the Accuracy Costs

This chapter is one of the two centers of this article.

When considering quantization, readers are most interested in understanding the extent to which accuracy is reduced. And when you compile what various primary sources say about this, you find that the AWS documents alone split the strength of the assertion four ways.

5.1 Four AWS Sentences Written at Four Different Strengths

The first three are listed here from weakest to strongest, and the fourth sits on a different axis, since its target is the KV cache and not the weights. What follows is what turned up in the documents this article collated. It is not an exhaustive survey of everything AWS publishes.

The first statement is from the Amazon SageMaker AI Developer Guide, in the section on quantization.

However, the quantized model might be less accurate than the source model that you optimized.

It says only that accuracy may drop, and says nothing about how far. Among the four, this is the most cautious wording.

The second statement is from the AWS Neuron NxD Inference Features Configuration Guide.

Note: Quantization slightly reduces accuracy due to using data types with lower precision
and/or lower range.

It asserts that accuracy does drop, and goes further by putting a magnitude on it: slightly. A reason is attached as well, which is that the data types used have lower precision or narrower range.

The third statement is from the Amazon Nova SageMaker inference container User Guide, in the section on weight quantization.

with minimal impact on output quality

It states that the impact is minimal. What that sentence covers, though, is limited to running Amazon Nova models in that container. It has the narrowest scope of the four, and in exchange it makes the strongest assertion.

The fourth statement is from the same page, in the section on data types for the KV cache.

at the cost of minor numerical differences in output

It states that the numbers shift slightly. And this section is the only one of the four that then adds the following.

Test your use case to verify that output quality meets your requirements, as lower precision
may produce slightly different results.

5.2 What the Papers Report Comes With Conditions

The papers do not claim that accuracy holds either. What they report is that the drop was small under a stated set of conditions.

• The first version of the original paper on GPTQ writes negligible accuracy degradation relative to the uncompressed baseline for the case of reducing to 3 or 4 bits per weight. What that covers is the set of models that paper ran experiments on.
• The first version of the original paper on SmoothQuant calls itself accuracy-preserving and writes with negligible loss in accuracy. What that covers is the set of models that paper ran experiments on.
• The first version of the original paper on AWQ states that it does better than the methods it compares against. It does not put the claim in a form that guarantees an absolute amount of degradation.

⚠ All three are reports made under stated conditions, with the model, the dataset, the metric, and the bit width all specified. They are not guarantees about the reader's configuration. This article does not carry these words over as results the reader is entitled to expect.

And the metric a paper measures is not necessarily the quality the reader wants to protect. A small drop on the metric the paper chose does not mean that the output format used in the reader's own work survives intact. This point will be addressed in Chapter 9.

5.3 Each One Is Correct About What It Describes

Do not read any of this as one of the AWS documents being wrong. The four are each describing something different.

Sentence	What the sentence describes
might be less accurate	Any format applied to any model. The scope is the widest of the four, so no magnitude can be given.
slightly reduces accuracy	The AWS Neuron implementation, reduced to a data type that implementation supports. The data types are bounded, so a magnitude can be given.
minimal impact on output quality	An Amazon Nova model taken to FP8 in the Amazon Nova inference container. Both the model and the format are fixed, so it can be stated most strongly.
minor numerical differences in output	The KV cache taken to FP8 in that same container. The target is not the weights, so it is a separate sentence.

In other words, the narrower the scope, the stronger the assertion. That is a document behaving correctly.

⚠ Errors arise when the reader misinterprets the scope. If you take the minimal impact statement about Amazon Nova containers and apply it to an estimate of what would happen when you run your own model on a different stack with INT4, you are extrapolating beyond the intended scope. Even if the quote itself is accurate, the claim is invalid if it's being applied to an incorrect configuration.

5.4 Only One of Them Tells You to Test

Set the four sentences side by side and this is where the most practical difference shows up.

Of those four, only the section on the KV cache data type instructs the reader to verify the results in their own environment. The remaining three statements conclude by mentioning a potential loss of accuracy, or a small reduction in accuracy.

This is not a flaw in the documentation. The purpose of a document describing specifications is to explain what the feature does. It is not to set the reader's acceptance criteria.

However, depending on which of the four statements a reader chooses to focus on, their next steps will change. Someone who reads only about a potential loss might choose to measure the results. Someone who reads about a minimal impact might not bother to measure.

So this article states its own position. Whichever path you take, do not skip comparing the model before and after quantization on your own workload. How to compare is Chapter 9.

6. The Paths on AWS

6.1 The Two Approaches the Developer Guide Now Names

Amazon SageMaker AI inference optimization currently consists of two approaches. This is stated at the beginning of the Developer Guide.

Amazon SageMaker AI provides two approaches to optimize your generative AI model inference:

The first is inference recommendations.

Inference recommendations. SageMaker AI automatically analyzes your model and workload,
evaluates instance types, applies optimizations, and returns validated, deployment-ready
configurations with real performance metrics. This is the recommended approach for most
customers.

The second is manual optimization.

Manual optimization. For customers who want a do-it-yourself approach, you can apply
individual optimization techniques such as quantization, speculative decoding, and
compilation. You choose which techniques to apply, run optimization jobs, and evaluate the
results yourself.

In other words, the path where you manually specify the quantization format sits on the manual optimization side of the Developer Guide's own split. The guide states that, for most users, the recommended approach is to not specify this manually.

Inference recommendations was announced in AWS What's New on April 21, 2026, and use from the Studio screen was added on August 20, 2026. That is three days before the verification date of this article.

The same page also touches on a third path, separate from those two. It states that pre-optimized versions are available for some models, and that you can deploy one without optimizing the model yourself. This is not choosing, and it is not letting the service choose. It is using something that has already been chosen.

This arrangement is the subject of this article itself. A path where you choose and a path where the service chooses sit side by side, and the recommendation is placed on the second one. How far the reader can find out what was applied is Chapter 7. The rest of this chapter works through the side where you choose.

6.2 The Inference Optimization Job

With manual optimization, you run an optimization job to create artifacts, and then deploy those artifacts to an endpoint.

When creating a job, the first decision you make is the instance type you want to deploy to, as this determines the available optimization methods. The Developer Guide states this outright in the Studio procedure.

The instance type affects what optimization techniques you can choose. For most types that
use GPU hardware, the supported techniques are Quantization and Speculative decoding. If you
choose an instance that uses custom silicon, like the AWS Inferentia instance ml.inf2.8xlarge,
the supported technique is Compilation, which you can use to compile the model for that
specific hardware type.

Specifically, if you choose a custom silicon instance, the available methods for this job will be limited to compilation. This does not mean that quantization itself is not possible; rather, it means that this particular job path does not offer it. Quantization on AWS Neuron is covered in Chapter 8.

Using the Python SDK, you pass a quantization_config to the optimize() function. The example provided in the Developer Guide uses environment variables to specify the format.

optimized_model = model_builder.optimize(
    instance_type="instance-type",
    accept_eula=True,
    quantization_config={
        "OverrideEnvironment": {
            "OPTION_QUANTIZE": "awq",
        },
    },
    output_path="s3://output-path",
)

6.3 The API Carries No Enumeration of Formats

There is a reason that the format is passed as an environment variable in the example above. This is because the API itself does not impose any restrictions on the format.

The API for optimization jobs accepts an array called OptimizationConfigs. Each element within that array is a union, allowing the user to specify one of four options.

This data type is a UNION, so only one of the following members can be specified when used
or returned.
ModelCompilationConfig / ModelQuantizationConfig / ModelShardingConfig /
ModelSpeculativeDecodingConfig

When quantization is selected, only two options are available.

Image
    The URI of an LMI DLC in Amazon ECR. SageMaker uses this image to run the optimization.
OverrideEnvironment
    Environment variables that override the default ones in the model container.

The API documentation contains no listing of supported formats. Instead, the user specifies the container image to use for the optimization process, as well as the environment variables to pass to that container.

This structure contrasts with the speculative decoding side. The speculative decoding configuration includes a Technique field, which lists the Valid Values. Both are members of the same union in the same job, and yet one side has its values constrained by the API and the other does not. The details of what is happening on the speculative decoding side are covered in Speculative Decoding and Draft Models on AWS.

You therefore cannot read which formats are available off the API. The container and the model decide that. The following section will discuss the implications of this.

6.4 Do Not Keep Your Own List of Supported Formats

The Developer Guide states this in the quantization section:

The data formats that SageMaker AI supports for quantization vary from model to model.

That sentence follows directly from the API structure in the previous section. The API does not define the supported formats. The container and the model do. Therefore, if the model changes, the supported formats will also change.

What holds the correspondence is the supported models reference in the Developer Guide. It contains tables listing each model, with a column indicating the supported formats for quantization. The same table also carries a column for whether speculative decoding is supported, and that column is covered in Speculative Decoding and Draft Models on AWS.

This article will not duplicate that table. The correspondence moves faster than this article can. Copying it into a design document is not a good idea either. What follows instead is only the kinds of value that appear in that column, as measured on the verification date.

The first kind is a row with several formats listed in it. A weight-only format and a format covering weights and activations both sit in the same cell. There is room for the reader to choose.

The second kind is a row where a later generation of the same model series carries fewer supported formats. As of the verification date, the table carried rows where a newer generation of the same model series listed fewer supported formats than an older one. A newer model does not necessarily come with more choices.

The third kind is a row that states there are no supported formats. And this one appears for two entirely different reasons. First, the model may be distributed initially with only low-precision formats. The precision has already been reduced, so this job has nothing left to reduce. The second is the case of the variants built for custom silicon. As the previous section showed, compilation is what that path applies.

⚠ Do not read a row that says there are no supported formats as meaning the model cannot be quantized. All it means is that this feature on this path does not produce them.

⚠ Do not read the model names listed in the table as recommendations. As of the verification date, the models listed in the model-by-model table were mostly older generations. The same page also contains another table that covers current-generation architectures. Because the tables have different levels of currency, you should always verify the information for the models you are using with official documentation.

6.5 The Container Environment Variable

The next path specifies the format through an environment variable carried by the container that serves the model. Rather than running a job to produce an artifact, the setting is passed when the endpoint is created.

The Amazon Nova SageMaker inference container takes this shape. Its User Guide describes weight quantization as follows.

Sets the quantization data type for model weights. Quantization compresses the model's
weights into a lower-precision format (FP8 instead of the default higher precision), which
reduces the amount of GPU memory the model requires.

The setting goes into the Environment block of CreateModel.

"Environment": {
    "QUANTIZATION_DTYPE": "fp8"
}

As of the verification date, the only valid value is fp8. The same page states explicitly that the feature was introduced in container version v1.3. On an image older than that, the setting simply does not exist. The same page recommends always using the latest image, and states which version the latest tag currently points at.

⚠ The default value has to be quoted with its conditional clause attached. What the default value field in the User Guide contains is this single run of text.

Disabled. However, FP8 quantization is automatically enabled for certain model and
instance type combinations. See the note below.

A statement that the setting is disabled and a statement that it is enabled automatically under certain conditions sit inside the same field. Quote only the first half and the second half is gone. Chapter 7 takes up that second half.

The same page also lists another environment variable specifically for the KV cache.

"Environment": {
    "KV_CACHE_DTYPE": "fp8"
}

These are two distinct settings. One is related to the weights, while the other is related to the KV cache. They also have different default values. The default for the weights setting is disabled, while the default for the KV cache setting matches the model's data type. The distinction between these environment variables directly reflects the axis difference described in Chapter 2.

The same page also states a version for combining this with speculative decoding. FP8 quantization of the weights became usable together with Eagle3 speculative decoding from v1.4. What happens on the speculation side is covered in Speculative Decoding and Draft Models on AWS. This article writes only that the combination is tied to a version.

6.6 Bringing a Checkpoint That Is Already Quantized

The next path finishes the quantization outside the platform entirely.

The inference engine has the capability to load quantized checkpoints. The official vLLM documentation provides a list of supported quantization formats, with each page detailing how to load them. The user's task is to provide a quantized artifact, rather than performing the quantization themselves, and then specify it.

A straightforward example of this process involves models that are distributed in a low-precision format from the outset. In this case, the user simply utilizes the quantized artifact provided by the distributor.

What characterizes this path is that responsibility sits somewhere definite. The distributor determines the quantization method, the bit width used, and the calibration data employed. If you perform the quantization yourself, you are responsible for these decisions. When using a pre-quantized artifact, it becomes crucial to verify whether this information is publicly available.

When importing a quantized artifact that you did not create yourself, the details of the quantization process reside outside of the artifact itself. These details will be addressed in Chapter 9.

6.7 Serving It Yourself

The last path is to stand up the inference engine yourself and specify quantization in its configuration.

What this path gives you is the breadth of supported formats and the granularity of the settings. The official vLLM documentation lists several implementations, including options for quantizing only the weights, quantizing both weights and activations, and quantizing the KV cache.

At the same time, this path puts every decision in your hands. You must choose which format to use, verify that the chosen format is compatible with your hardware accelerator, and confirm that the selected format is implemented in the specific version of the inference engine you are using. On this path, the constraints in the next chapter and in Chapter 8 all sit on your side.

The decision of whether or not to self-host is addressed in Self-Managed LLM Inference on Amazon EKS. That article highlights the ability to select your own checkpoints, serving engine versions, quantization methods, and inference parameters as a primary reason for choosing self-hosting. This article expands on that particular point.

6.8 What Each Path Takes On and What It Leaves You

Reordering the paths in this chapter by what each one lets you decide gives five of them. Bringing your own and serving it yourself land on the same side, because in both cases the format is yours to choose, so they are grouped as one.

The five distinctions are not about speed. They relate to what users can determine themselves, and whether the results of those determinations can be verified later.

Path	Who picks the format	Can you tell what was applied	Can you turn it off	Can you choose the calibration data	Range of formats available
Inference recommendations	The service picks	The Developer Guide does not list the specific optimizations applied.	You change the goal you specify	Not stated	The range the service covers
Pre-optimized configuration	Already decided	The Developer Guide does not describe the methods applied.	Select a different configuration	Not stated	Range of provided configurations
Inference optimization job	You pass it in as an environment variable	What you specified	Run the job again	Delegated to the container	Depends on the model and the container
Container environment variable	You set it, inside the valid values	What you specified	There is an exception. Chapter 7	Not stated	The valid values the container accepts
Bring your own, or serve it yourself	You decide	What you decided	Yes	Yes	Depends on the engine and the hardware

The most important column in this table is the third. On the paths where you specified it yourself, you know what was applied. On the paths where you did not, the documentation gives no way to find out.

⚠ The top two rows of the third column are not a claim that quantization is happening. They are a claim that what was applied is not visible to the reader. Those are two different things, and Chapter 7 draws the line between them.

7. Quantized Without Choosing It

This chapter is the other center of this article.

Up to Chapter 6, quantization has been written about as something you choose. However, there are instances where this premise breaks down. This breakdown is not uniform, however. There are three layers, each with a different degree of certainty. You should not treat all three layers with the same level of conviction.

7.1 The One Case That Is Stated Outright

The User Guide for the Amazon Nova SageMaker inference container states outright that for certain combinations quantization is enabled automatically and can neither be disabled nor overridden.

The following model and instance type combinations require FP8 quantization. For these
configurations, quantization is enabled automatically and cannot be disabled or overridden:

As of the verification date, the listed combinations are as follows:

Amazon Nova Lite on ml.g6.12xlarge or ml.g6.24xlarge
Nova 2 Lite on ml.g6.48xlarge

⚠ This list may change. This article was verified on August 23, 2026, and does not reflect any subsequent changes. Do not copy it into a design document. The combinations in scope are held by that same User Guide.

The same User Guide includes another page with a compatibility table, which has a column indicating whether quantization is required. As of the verification date, the two pages agreed. Confirming that they agree is where the collation ends.

7.2 These Are Three Separate Facts

Do not fold what is written above into one statement. What is written there is three separate facts.

Fact	User Guide Description	Location
The default is disabled	Disabled.	The first half of the default value field
But for certain combinations it is enabled automatically	However, FP8 quantization is automatically enabled for certain model and instance type combinations.	The second half of the same default value field
In that case it can neither be disabled nor overridden	quantization is enabled automatically and cannot be disabled or overridden	A separate note placed apart from it

Fold the three together into a statement that this feature is disabled by default, and the reader's design breaks. Taking the default being disabled as grounds for concluding that leaving the setting alone means nothing was quantized is a reading that has dropped the second and third facts.

And writing only that this feature is enabled automatically breaks it in the other direction. In the great majority of configurations, the default stays disabled.

Therefore, it is important to present these three points separately. Omitting any one of them will lead to misinterpretations.

7.3 The Pair Decides, Not the Model and Not the Instance

This is where it is easiest to go wrong. What decides whether it is enabled automatically is neither the model on its own nor the instance type on its own. It is the pair.

As of the verification date, the table on that second page carried counterexamples that show it.

• There are rows where the model is the same and a different instance type makes it not required. Amazon Nova Lite is listed as required on ml.g6.12xlarge and ml.g6.24xlarge, and as not required on ml.g6.48xlarge.
• There are rows where the instance type is the same and a different model makes it not required. ml.g6.12xlarge and ml.g6.24xlarge are listed as not required on the Amazon Nova Micro rows.
• There are rows where the instance type is the same and a different model generation changes the answer. ml.g6.48xlarge is listed as required for Nova 2 Lite, and as not required for Amazon Nova Lite.

A rule of the form this combination is automatic cannot be derived from the model name alone, and it cannot be derived from the instance type alone.

⚠ Therefore, any attempt to summarize this information and create a design rule is almost certain to be incorrect. The only reliable approach is to consult the official tables for the specific combination you are using.

This article does not build an exhaustive table of combinations. What is listed above are counterexamples, meant to show that a single attribute does not decide the answer. They are not a list.

7.4 Two Cases the Documentation Does Not Speak To

The remaining two layers are of an entirely different character. Nothing states that quantization is applied automatically. What is missing is a statement of what gets applied at all.

The first is the path that deploys a pre-optimized configuration as it stands. The Developer Guide states that for some models an optimized version can be deployed without creating an optimization job.

Some models in JumpStart are pre-optimized by SageMaker AI, which means that you can deploy
optimized versions of these models without first creating an inference optimization job.

The table that presents the choices carried five columns as of the verification date. They are the instance type, the configuration name, the number of concurrent users, the latency to the first token, and the throughput per user. There is no column that says which techniques were applied. The configuration name is lmi-optimized.

⚠ There is a limit to what can be written here. This article does not write that the weights are quantized in these configurations. What can be written is only that the Developer Guide does not state which techniques were applied.

The same page also states that on this path, changing the instance type away from the default makes it impossible to deploy the pre-optimized configuration. The configuration is tied to the instance type.

The second is inference recommendations. The Developer Guide describes this feature as follows:

Inference recommendations analyzes your model's architecture, narrows the configuration
space, and applies goal-aligned optimizations such as speculative decoding for throughput and
kernel tuning for latency.

The phrase such as gives examples. It does not limit the set. The two named here are speculative decoding and kernel tuning, and that is not an enumeration of the optimizations that may be applied.

⚠ It is therefore not possible to write that this path applies quantization, and not possible to write that it does not. What can be written is only that the Developer Guide does not enumerate the set of optimizations that get applied.

7.5 Do Not Read All Three at the Same Strength

The three layers, ordered by how strongly you can state a conclusion.

Layer	What the documentation says	What the reader can state
The Amazon Nova inference container	States outright that it is enabled automatically and can neither be disabled nor overridden.	You can state that it is quantized. The combinations in scope are enumerated officially as well.
The pre-optimized configuration	Gives latency and throughput figures, and does not say which techniques were applied.	You can state neither that it is quantized nor that it is not. What is known is only that the published information does not let you tell.
Inference recommendations	Gives examples of the optimizations applied, and does not enumerate them.	Same as above.

Only the top row presents a case where quantization can be definitively concluded, even without explicit selection.

The remaining two layers are not claims about quantization at all. They are claims that what was applied is not visible to the reader. These two are separate issues that require different approaches.

⚠ Blending all three together into a summary that AWS quantizes things behind your back would be an unsupported claim about the lower two. This article does not write that.

7.6 What to Do About It in a Design

First, verify using the official documentation whether quantization is required for the specific combination of your model and instance type. Record the date of this verification. Re-evaluate this whenever you change the combination.

Second, if you choose a path that is not explicitly specified, design with the understanding that you are operating in a state where it's unclear what, if anything, is being applied. It's not inherently a problem to have this uncertainty. The issue arises when this unknown state is mistakenly assumed to mean that nothing is being applied.

Third, if there are requirements that depend on the assumption that quantization is not being used, explicitly verify that assumption. If there is no way to confirm whether that assumption holds true, then that requirement cannot be met.

Fourth, do not treat changes in instance type solely as performance-related modifications. Even with the same model, a different instance type can result in different weight precision. Changes in instance type may include alterations to the model's behavior.

8. The Format Is Bound to the Hardware

8.1 The Developer Guide Says the Instance Type Decides

The sentence from the Developer Guide quoted in Chapter 6 is worth using once more. The same sentence reads as a statement about paths and as a statement about hardware.

The instance type affects what optimization techniques you can choose.

AWS explicitly states that the choice of deployment instance type determines the available methods. While this is not limited to quantization, it applies to quantization as well.

Therefore, the question of which format to use is inextricably linked to the question of which instance type it will run on. The correct order is to first determine the desired instance type and then select the appropriate format. However, a frequent motivation for considering quantization is the desire to run it on a smaller instance type. The two are circular, so the only way through is to fix one provisionally and go back and forth.

8.2 On AWS Neuron, Whether FP8 Exists Depends on the Generation

On the AWS Neuron side, this coupling is even more explicit.

The AWS Neuron data types documentation describes the supported data types separately for each version of the NeuronCore. As far as the verification date shows, the following can be said.

• For the first generation of NeuronCore, the only supported data types the documentation lists are 32-bit and 16-bit floating point. 8-bit floating point is not in that list.
• From the next generation on, 8-bit floating point enters that list. And not as a single type: it appears as several encodings that divide range and precision differently.

Therefore, before selecting FP8, you must first verify whether that generation of NeuronCore actually supports the FP8 data type. If it does not, the limitation is not due to an implementation issue, but rather because the option simply is not available.

The NxD Inference Features Configuration Guide includes settings for weight quantization, allowing you to specify the supported data type and quantization method. The method selection involves choosing whether to define the scale based on the tensor or channel level.

Furthermore, selecting a specific data type may impose certain prerequisites. The same guide states that when using 8-bit floating-point with a focus on precision, you need to set the environment variable XLA_HANDLE_SPECIAL_SCALAR to 1. The same condition applies to the quantization of the KV cache. Choosing a particular format also means accepting the prerequisites associated with that format.

This article does not provide a table listing the supported data types for each generation. The AWS Neuron documentation already provides this information, and the number of generations continues to increase.

8.3 On GPUs the Serving Engine Publishes Its Own Compatibility Chart

On the GPU side, the inference engine's own documentation answers the same question.

The official vLLM documentation, specifically the quantization section, provides tables that indicate which hardware is supported for each quantization implementation. As far as the verification date shows, the following structure can be read off it.

• Among implementations that only quantize weights, some support older generations of hardware.
• Implementations that quantize both weights and activations to 8-bit floating point are limited to newer generations of hardware. Older generations are indicated as not supported.

As Chapter 3 set out, the benefit of getting onto the low-precision arithmetic path is available only on the generations that carry that arithmetic unit. Weight-only quantization reaches further precisely because it does not need that unit.

This article will not reproduce that table. The official vLLM documentation itself explicitly states that this compatibility table is subject to change.

This compatibility chart is subject to change as vLLM continues to evolve and expand its
support for different hardware platforms and quantization methods.

Maintaining its own compatibility table means that it will inevitably become outdated. The supported hardware configurations evolve more rapidly than this article. Readers should always refer to the official table and record the date on which they consulted it.

8.4 A Format Is Available Only Where Four Things Hold at Once

Everything up to here comes together as one structure. Whether a given format is actually available to you is the product of four conditions.

The first condition is the format itself. Is it weight-only, or does it cover weights and activations as well? How many bits are involved? What method is used for quantization?

The second condition is the accelerator generation. Does that generation support that data type? Even if it does, does it have a processing path for that type?

The third condition is the serving stack. Does the inference engine, the container, or the compiler you use implement that format? And if it does, is it implemented in the version you run?

The fourth condition is the model. Does the official reference list that format for that model? Or does the model already ship quantized?

Only the range where all four conditions hold at the same time is the set of formats you can actually pick.

The difficulty of quantization lies in understanding this product of conditions. It's not a matter of algorithmic complexity. Each condition is described in separate documents, and each condition can change independently. If you change even one of these four conditions, you will need to re-evaluate the other three.

Therefore, changing an instance type, updating an inference engine, or replacing a model all require checking the supported formats again. In configurations where pre-processing and decoding are separated, this alignment becomes even more critical. If the settings on either side are mismatched, the connection itself may fail. Further details can be found in Disaggregated Prefill and Decode for LLM Serving on AWS.

9. How to Measure What You Paid

9.1 Exact Match Cannot Measure This

A check that lines up the output from before and after quantization and looks for an exact match does not work.

The reason comes in two stages. The first is that quantization itself changes the output. As Chapter 2 set out, quantization claims neither bitwise identity nor distribution equivalence. The change is what was expected. It is not an anomaly to detect.

The second stage is that the output does not match even without quantization. For the same request, the numbers move with the number of other requests being processed alongside it. This phenomenon is addressed in Reproducible LLM Inference.

Because the two reasons overlap, observing a mismatch does not let you separate quantization from the movement of the batch. Exact match does not work as an instrument for measuring the effect of quantization.

⚠ Get this wrong and you either spend time chasing a harmless difference or miss a harmful one.

9.2 What to Compare

What you compare is an aggregated metric. In the vocabulary of Chapter 2, quantization claims only the third level, so the check is run at that level too.

First, determine meaningful metrics for your specific work beforehand. The metric a paper reports is the one that paper chose. It does not follow that the same metric means anything in your own work. If you do not decide on these metrics in advance, you risk selecting metrics that are convenient after the quantization process has already been completed.

Second, independently measure whether the output format is intact. Even if the quality of the output is maintained, the validity of the structured output may degrade. It's possible to experience a form of degradation where the average of the metrics remains stable, but the rate of failures increases. The verification layer for outputs is covered in LLM Output Verification Patterns.

Third, understand the relationship between the inputs used for evaluation and the inputs used for calibration. As described in Chapter 4, all the methods discussed in this article utilize calibration data. If there is a discrepancy between the distribution of the calibration data and the distribution of the production data, that discrepancy may not be visible in the evaluation data. If the inputs used for evaluation are too similar to the inputs used for calibration, you may overlook potential degradation.

Fourth, take the baseline as the model before quantization. Comparing quantized models against each other does not eliminate the possibility that both have degraded. The baseline is the model before quantization.

Which service you run the evaluation job on, and how, is outside the scope of this article. Amazon Bedrock Model Evaluation Practical Guide addresses that topic. For a configuration that watches this continuously, see LLMOps Observability and Evaluation Architecture on AWS.

9.3 What Is Worth Recording

The provenance of a quantization is often not readable from the model artifact itself. Recording it is the reader's job.

What to record	Why it is needed
The identifier of the original checkpoint	So that the baseline for comparison can be identified later.
What was reduced, whether weights only, weights and activations, or the KV cache as well	Otherwise which of the three axes in Chapter 2 was touched stops being recoverable.
The name of the format and of the method	The format name alone does not settle the method.
Where the calibration data came from	It may be deciding which domains the model is good at, and it is not readable from the artifact.
The accelerator generation and the instance type	One of the conditions under which the format holds at all. Change it and everything has to be looked up again.
The serving stack and its version	Same as above.
Whether this was a configuration where quantization was enabled automatically	If you did not specify it yourself, there is nothing to recover it from later without a record.
The verification date	Both the correspondence and the conditions for automatic enablement move.

The second-to-last item is the one that corresponds to the situation in Chapter 7. Settings that were not explicitly configured are more likely to be forgotten. If forgotten, they will be excluded from the list of potential causes when investigating unexpected behavior later on.

One more point: in a configuration that separates prefill and decode, the KV cache data type matching on both sides is a condition of the connection. Disaggregated Prefill and Decode for LLM Serving on AWS lists it among the items that must agree. The record is needed in operation as well.

10. When Not to Quantize

Quantization is not always the optimal choice. In the following scenarios, it may be better to avoid it, or at least explore alternative solutions first.

First, the case where it fits as it is. The primary benefit of quantization is reducing the amount of accelerator memory required. If the model already fits, you will not gain that benefit. What is left is the smaller read volume per token, and against it sit the accuracy cost, the verification effort, and the burden of managing provenance.

Second, if the bottleneck is not memory bandwidth. As mentioned in Chapter 3, weight-only quantization is most effective when memory bandwidth is the limiting factor. When processing long inputs or handling a large number of requests concurrently, this may not be the case. Applying quantization without first measuring where the bottleneck lies can make it difficult to understand why you are not seeing the desired results.

Third, when complete output reproducibility is a strict requirement. Quantization alters the output. If a configuration is already reaching for bitwise identity, quantization takes that premise away. It's impossible to satisfy both requirements simultaneously. Determine which requirement is truly paramount.

Fourth, when you lack the means to detect any degradation in accuracy. As discussed in Chapter 9, the impact of quantization can only be measured through aggregated metrics. Applying quantization without a mechanism to measure its effects means you will not have a way to detect any performance decline. In this case, it's better to prioritize developing a measurement system before considering quantization.

Fifth, when there is no ongoing system in place to verify the chosen format. As mentioned in Chapter 8, the number of viable formats is the product of four conditions. Once a particular combination is selected, there is no guarantee that it will remain valid in the future. If no one is responsible for periodically re-evaluating these configurations, the system will eventually break down.

Sixth, when a model swap is close. Supported formats differ from model to model. The format you picked now may not be usable on the next model. If a model replacement is on the horizon, the verification effort invested in quantization may not be transferable to the next generation.

⚠ Conversely, there are also factors that do not necessarily provide a reason to avoid quantization. A general concern about potential accuracy degradation is not sufficient justification. Unless you are measuring the extent of that potential degradation using your own metrics, it's not a decision, but rather a postponement.

11. A Checklist for Choosing

Review these items in order from top to bottom. If you stop at an item higher up, there is no need to proceed further down.

Step 1. Is quantization needed at all?

• Does the model as it stands fit in the accelerator memory of the instance type you intend to use? If so, the default option is to not quantize.
• If you are experiencing performance issues, have you first identified where the bottleneck is? If it's not limited by memory bandwidth, the impact of quantizing only the weights will be limited.
• Is bitwise identity of the output part of your requirements? If it is, quantization is not compatible with it.

Step 2. What do you reduce?

• Is what you want to reduce the resident memory, the arithmetic path, or the memory that piles up per request? The three are separate axes.
• If you want to buy the arithmetic path, reducing only the weights does not get you there. A configuration that reduces the activations as well is required.
• The quantization of the KV cache should be evaluated separately from the quantization of the weights. If you choose to quantize both, measure the impact of each independently.

Step 3. Does the format you picked satisfy four conditions at once? Settle on a format provisionally, then look up the remaining three.

• Does that format exist in the generation of the accelerator you plan to use?
• Does the serving stack version you intend to use implement that format?
• Does the official documentation state that the format is supported for your model? If it does not, did you separate out whether that means out of scope, or whether it is a matter for a different path?
• If you anticipate changing any of these four factors, have you designated who will re-evaluate the selection at that time?

Step 4. What happens if you do not choose?

• Does the combination of your model and instance type fall under a category where quantization is explicitly required, according to the official documentation? Have you verified this in the official tables?
• If you choose a path that is not explicitly defined, do you have a way to determine what settings will be applied? If not, have you designed your system with that uncertainty in mind?
• If you have operational procedures that involve changing instance types, have you documented in those procedures that such changes could affect the precision of the weights?

Step 5. How do you measure what you paid?

• Are you trying to compare by exact match?
• Did you decide the metric that matters for your own workload before you quantized?
• Are you measuring the kind of degradation that breaks the output format separately from the average?
• Are you using the unquantized model as the baseline for your comparisons?
• Do you understand the relationship between the distribution of your calibration data, the distribution of the input used for evaluation, and the distribution of your production data?

Step 6. What do you record?

• Have you documented the items from Chapter 9? In particular, have you documented any settings that you did not explicitly define?
• Have you recorded the date of your verification?

12. Failure Modes and Anti-Patterns

What follows turns the decisions above into the forms to avoid.

First, comparing based solely on naming conventions. Discussing which is better between INT4-AWQ and FP8 will not yield an answer. One is a name that specifies a method, while the other is simply a data type name. If you are going to compare, ensure you align what is being reduced and the method used for that reduction before doing so.

Second, expecting weight-only quantization to speed up the arithmetic. The values sit at low precision, and the matrix multiply still runs after converting them back to the original precision. To optimize performance, you also need to reduce the precision of activations.

Third, copying the correspondence table directly into design documents. The supported formats, the generations they apply to, and the automatically enabled combinations – all are dynamic. The moment you copy it, it begins to become outdated. Refer to the official documentation, and record the date you consulted it.

Fourth, interpreting the absence of a format in the correspondence table as meaning it is not possible to quantize. A row indicating a missing supported format does not necessarily mean that quantization is impossible. It may simply mean that the specific feature is not supported within that particular pathway.

Fifth, treating an untouched setting as grounds for saying nothing was quantized. Certain combinations are automatically enabled and cannot be disabled or overridden. A conditional clause follows the description of the default.

Sixth, interpreting the absence of information about what has been applied as meaning that nothing has been applied. These are two entirely different concepts. The former represents a state of lacking information, while the latter is a statement about the state itself.

Seventh, treating changes to instance types solely as performance-related changes. Even with the same model, changing the combination can alter the precision of the weights. This is an operation that can potentially affect the model's behavior.

Eighth, borrowing a statement that accuracy does not drop without checking its scope. Take wording written for one model and one format and apply it to another model and another format, and the quote stays accurate while the claim stops holding.

Ninth, trying to detect degradation with exact match. The output does not match even without quantization. Observing mismatches does not allow you to isolate the cause.

Tenth, creating calibration data and performing evaluations using only the target distribution. If you create data that performs well in a specific area and measure performance in that same area, degradation may not be apparent. The input used for evaluation must be independent of the input used for calibration.

Eleventh, performing quantization and failing to record the provenance. It is often difficult to determine, from the final product, which method was used, what bit-depth was used, or what calibration data was used. Without proper records, it will be impossible to trace the reasons for any subsequent changes in behavior.

Twelfth, quantizing without an established evaluation process. Reducing precision without a mechanism to detect degradation is like paying a cost without measuring the consequences.

13. Frequently Asked Questions

Q. Which is better, INT4-AWQ or FP8?

There is no definitive answer without more information. The number of bits alone does not determine whether only weights are quantized, or both weights and activations. The optimal choice also depends on which components are quantized, and whether your hardware accelerator supports the format, whether the serving stack has implemented it, and whether support for that model is documented. Fill in the four conditions in Chapter 8 first.

Q. Does quantization always result in faster performance?

No. It gets faster when memory bandwidth is what the situation is stalled on. As the number of concurrent requests increases, the bottleneck may shift, and the effect of quantization becomes smaller. The AWS machine learning blog also writes that the reduction in movement can be offset by the cost of dequantization.

Q. How much accuracy is lost?

The primary sources do not guarantee a magnitude. The Amazon SageMaker AI Developer Guide says only that the quantized model may be less accurate than the model it was optimized from, and does not touch on how far. The values reported in research papers are the results of experiments conducted on specific models, datasets, and metrics. To determine the degradation in your specific configuration, you will need to measure it yourself.

Q. Why do AWS documents have different ways of describing accuracy?

Because the scope differs. The wider the scope a document covers, the less it can put a magnitude on the drop. The narrower the scope, the more strongly it can write. None of them are incorrect. What the reader needs is to check which scope a given sentence was written for. Chapter 5 lines the four up.

Q. Is there a way to confirm that quantization is not in effect?

Yes on one path, and no on the others. For the Amazon Nova SageMaker inference container, there are official tables listing combinations that require quantization, so you can check if your configuration is on that list. For pre-optimized configurations and inference recommendations, the methods applied are not documented in the Developer Guide, so it's impossible to determine from publicly available information.

Q. Can automatically enabled quantization be disabled?

No. The User Guide explicitly states that for certain combinations, it is not possible to disable or override quantization. If you want to avoid quantization, the only option available to users is to change the combination of models and instance types.

Q. Can KV cache quantization also be enabled simultaneously?

Yes, but treat it as a separate consideration. It affects different components, has different default values, and requires different environment variables. Enabling both simultaneously can make it difficult to isolate the cause of any observed degradation.

Q. What data should be used for calibration?

The primary sources do not name particular data. The original paper on GPTQ notes for itself that the calibration data used in its experiments was drawn from the training side of the evaluation dataset. The original paper on AWQ claims that its method is less prone to overfitting the calibration set. The closer you move to the production distribution, the stronger the model gets on that ground, and nothing is guaranteed outside it. Ensure that the input used for evaluation is independent of the input used for calibration.

Q. Can weight quantization be combined with speculative decoding?

Yes. The Amazon Nova SageMaker inference container documentation states that support for combining FP8 quantization with Eagle3 speculative decoding was added in version v1.4. The methodology on the speculation side is outside the scope of this article. See Speculative Decoding and Draft Models on AWS.

Q. Is weight quantization the same thing as the quantization in vector search?

No. The name is shared and the technology is not. The technology that compresses embedding vectors to reduce the memory footprint of indexes is discussed in Vector Database Selection on AWS. The properties it aims to preserve and the methods used to evaluate it are different.

Q. What should you check when you use a quantized model exactly as it was distributed?

What was reduced, the method, the bit width, and where the calibration data came from. All four are often unreadable from the artifact files themselves, so check what the distributor publishes and record it. If it is not readable, the only remaining option is to run the comparison in Chapter 9 yourself.

14. Summary

Under the single word quantization there are at least three separate targets. Do you reduce only the weights, the weights and the activations, or the KV cache? The three take effect in different places and are chosen independently. AWS documentation sometimes lists these together in the same column, so do not assume that their proximity indicates they are equivalent.

Quantizing only the weights primarily addresses memory usage. By reducing the amount of data read, it can improve performance in areas limited by memory bandwidth. The computations themselves are performed with the original precision. Getting onto the low-precision arithmetic path takes reducing the activations as well.

The difference between different weight-only quantization methods is not about the number of bits. It's about the magnitude of the impact that calibration data has on the results. The original AWQ paper specifically mentions GPTQ, highlighting this as a key differentiator. The choice of what calibration data to use is a design decision that the user must make.

On what the accuracy costs, AWS writes at four different strengths. The Developer Guide mentions a potential drop in accuracy, AWS Neuron states a slight decrease, the Amazon Nova inference container claims the impact on the weights is minimal, and the same guide notes minor numerical differences for the KV cache. All four statements are accurate regarding what they describe. Only the last of the four explicitly states that you should verify this for your specific use case.

The formats you can pick are the product of four conditions. The format itself, the accelerator generation, the serving stack, and the model. Only the range where all four hold at the same time is the range you can actually choose from. Change any one of the four and the other three get looked up again.

There are situations where quantization is not a choice. The Amazon Nova SageMaker inference container, for example, automatically enables FP8 quantization for specific model and instance type combinations, and explicitly states that it cannot be disabled or overridden. The decision is not made by the model alone, nor by the instance type alone; it's a combination of both.

Separately, there are two paths where the documentation does not state what has been applied. These are pre-optimized configurations and inference recommendations. What can be said about these two is not that they are quantized, but rather that the details of what has been applied are not visible to the user. Confusing these two can lead to unsubstantiated claims.

What you paid cannot be measured by exact match. Quantization changes the output, and even without quantization the output does not come back identical. Because of these overlapping factors, observing discrepancies does not necessarily allow you to isolate the cause. What you measure is an aggregated metric, and the metric is decided before you quantize.

Finally, put the option of not quantizing first. If the model performs adequately without it, you will not realize the primary benefits of quantization. You're then left with the costs associated with potential accuracy trade-offs, verification effort, and provenance management. Having no reason to reject quantization is not the same as having a reason to choose it.

15. References

• Inference optimization for Amazon SageMaker AI models - Amazon SageMaker AI Developer Guide
• Create an inference optimization job - Amazon SageMaker AI Developer Guide
• Supported models reference - Amazon SageMaker AI Developer Guide
• Deploy a pre-optimized model - Amazon SageMaker AI Developer Guide
• Optimized generative AI inference recommendations - Amazon SageMaker AI Developer Guide
• OptimizationConfig - Amazon SageMaker AI API Reference
• ModelQuantizationConfig - Amazon SageMaker AI API Reference
• Inference Container Features - Amazon Nova User Guide
• SageMaker Inference - Amazon Nova User Guide
• Amazon SageMaker AI launches optimized generative AI inference recommendations - AWS What's New
• Generative AI Inference Recommendation for Amazon SageMaker now available in the SageMaker AI Studio - AWS What's New
• NxD Inference Features Configuration Guide - AWS Neuron Documentation
• Data Types - AWS Neuron Documentation
• Achieve up to ~2x higher throughput while reducing costs by ~50% for generative AI inference on Amazon SageMaker with the new inference optimization toolkit - Part 1 - AWS Machine Learning Blog
• Quantization - vLLM Documentation
• Quantized KV Cache - vLLM Documentation
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers - arXiv:2210.17323
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models - arXiv:2211.10438
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration - arXiv:2306.00978
• Reproducible LLM Inference - Batch Invariance, the Cost of Determinism, and Where Bitwise Identity Is Actually Required
• Speculative Decoding and Draft Models on AWS - Acceptance Rates, Which Methods Preserve the Output Distribution, and What the Managed Draft Model Takes Away
• Vector Database Selection on AWS - OpenSearch, Aurora pgvector, MemoryDB, and S3 Vectors with Index Algorithm Internals
• Self-Managed LLM Inference on Amazon EKS - Serving Open-Weight Models with vLLM, Neuron/GPU, and Karpenter
• Disaggregated Prefill and Decode for LLM Serving on AWS - The KV Transfer, the Routing Threshold, and What Disaggregation Does Not Fix
• Amazon Bedrock Model Evaluation Practical Guide - Automatic Metrics, LLM-as-a-Judge, RAG Evaluation, and CI/CD Quality Gates
• LLM Output Verification Patterns - Grounding Checks, Self-Verification, Cross-Model Review, and Citation Enforcement
• LLMOps Observability and Evaluation Architecture on AWS - Tracing, Metrics, and Automated Evaluation Gates with CloudWatch and OpenTelemetry
• AWS Custom Silicon History and Timeline - Nitro System, Graviton, Inferentia, Trainium, and Introduction
• Amazon EC2 Instance Types History and Timeline - Instance Families, Generations, and Introduction
• Open-Weights LLM Release History and Timeline - Llama, Mistral, Qwen, DeepSeek, and gpt-oss Model Families, Capability Evolution, and Licenses
• Amazon Bedrock Model Catalog 2026

References:
Tech Blog with curated related content