Quantization Error

Quantization error is deterministic for a given input and fixed quantization parameters (scale, zero-point). However, its magnitude and distribution are highly data-dependent. The error for a specific activation tensor depends entirely on the statistical distribution of its values relative to the chosen quantization range. This is why a calibration dataset is critical for static quantization to estimate these ranges accurately.

• Example: A layer with a narrow, Gaussian distribution of values will have lower error under symmetric quantization than a layer with a long-tailed, asymmetric distribution.

The error for any single value is bounded by half the quantization step size (for rounding-to-nearest). However, the aggregate effect across a network is non-linear and does not simply average to zero. Error can compound through successive layers, as the quantized output of one layer becomes the quantized input to the next. This can lead to error propagation where small per-layer errors cause significant drift in the final model output.

• Key Insight: Bounded per-value error does not guarantee bounded model output error, making techniques like Quantization-Aware Training (QAT) necessary for sensitive models.

The granularity of the quantization scheme dictates how error is distributed and its overall impact.

• Per-Tensor Quantization: Applies one set of parameters to an entire tensor. Efficient but can produce high error if the tensor's value distribution is wide.
• Per-Channel Quantization: Uses unique parameters for each channel (e.g., each output channel of a weight tensor). This accounts for inter-channel variation, significantly reducing error for weight tensors with minimal overhead.
• Block-Wise Quantization: A finer granularity (e.g., per 64 values) that offers a better accuracy/efficiency trade-off, especially for Low-Bit Quantization like INT4.

The choice between symmetric and asymmetric quantization creates a fundamental trade-off between computational simplicity and error minimization.

• Symmetric Quantization: Range is centered on zero. The zero-point is 0, simplifying integer arithmetic (no zero-point addition needed during matrix multiplication). Best for data distributions that are roughly symmetric around zero (e.g., weight tensors after normalization).
• Asymmetric Quantization: Range is not centered on zero. Uses a non-zero zero-point to precisely map the real value zero. This minimizes error for data with an asymmetric distribution (e.g., ReLU activations that are all >=0) but adds computational overhead.

Quantization error fundamentally creates a trade-off between inference efficiency and model fidelity. This manifests in two key ways:

1. Task Accuracy Drop: Measured as the decrease in standard evaluation metrics (e.g., top-1 accuracy on ImageNet). A well-calibrated 8-bit model might see a drop of <1%, while aggressive 4-bit quantization can cause >5% degradation.
2. Reduced Output Robustness: Quantized models can become more sensitive to adversarial examples or noisy inputs, as the error perturbs the model's decision boundaries. This is a critical consideration for safety-critical applications.

Not all layers contribute equally to the total error. Quantization Sensitivity Analysis reveals that layers closer to the output (e.g., classification heads) and certain operations (e.g., attention softmax) are often highly sensitive. Conversely, early convolutional layers may tolerate aggressive quantization. This insight drives advanced strategies:

• Mixed-Precision Quantization: Applying higher precision (FP16) to sensitive layers and lower precision (INT8) to robust layers.
• Selective Quantization: Skipping quantization entirely for one or two critical layers to recover significant accuracy with minimal performance cost.

The quantization scale is a multiplicative factor that maps a range of floating-point values to a quantized integer range. It is calculated as (float_max - float_min) / (quant_max - quant_min). This parameter directly controls the resolution of the quantization grid; a larger scale compresses a wider range of values into the same integer bits, increasing the potential for error. It is a critical parameter determined during calibration.

The quantization zero-point is an integer value that represents the real numerical zero in the quantized domain. It is essential for asymmetric quantization schemes, allowing the quantized range to precisely map an asymmetric distribution of values (e.g., ReLU activations that are all >=0). The formula is zero_point = quant_min - round(float_min / scale). An accurate zero-point ensures that the value zero is quantized without error, which is crucial for operations like padding.

These are two fundamental methods for mapping values to discrete levels:

• Uniform Quantization: Uses evenly spaced quantization levels (constant step size). It is simple and hardware-friendly but may waste representational capacity if the data distribution is not uniform.
• Non-Uniform Quantization: Uses unevenly spaced levels, allowing finer granularity where the data distribution is denser. This can lower mean squared error for a given bitwidth but requires more complex, often lookup-based, hardware support.

A calibration dataset is a small, representative set of unlabeled data (typically 100-1000 samples from the training distribution) used during Post-Training Quantization (PTQ). Its purpose is to observe the dynamic range of model activations in order to calculate optimal quantization scale and zero-point parameters. The choice and quality of this dataset significantly impact the final quantization error, as poor calibration leads to clipping or excessive rounding error.

Dequantization is the reverse process of quantization, converting integer values back into floating-point numbers. The operation is float_value = scale * (int_value - zero_point). This step is necessary to interpret the results of integer arithmetic, especially when feeding outputs into non-quantized layers or for final loss calculation. The quantization error is formally defined as the difference between the original floating-point value and the result of quantizing-then-dequantizing it (QDQ(x) - x).

Fake quantization (or simulated quantization) is an operation that mimics the effects of quantization and dequantization during training or calibration, while keeping the underlying tensors in floating-point format. It applies the rounding, scaling, and clipping logic of the target quantization scheme. This is the core mechanism of Quantization-Aware Training (QAT), allowing the model to learn to compensate for the introduced error, and is used in calibration for Post-Training Quantization (PTQ) to determine ranges.