Quantization Error

Quantization error can be deterministic or stochastic. Deterministic error arises from the fixed rounding scheme (e.g., round-to-nearest) and is reproducible for a given input. Stochastic error is introduced by techniques like dithering, where random noise is added before quantization to break up structured error patterns and prevent artifacts, trading a small increase in noise for a reduction in perceptual distortion.

A fundamental analytical model treats quantization error as additive white noise. This Quantization Noise Model assumes the error is uncorrelated with the signal and has a uniform distribution. Key implications:

• The noise power is proportional to the square of the quantization step size (Δ²/12).
• The Signal-to-Quantization-Noise Ratio (SQNR) increases by approximately 6 dB per additional bit of precision.
• This model breaks down for low-bit quantization or small signals, where error becomes correlated and non-linear.

The quantization step size (Δ) is the fundamental granularity of the discrete representation, defined as (max_value - min_value) / (2^b - 1) for b bits. Characteristics:

• Granular Distortion: Large step sizes cause large jumps between representable values, leading to rounding error.
• Overload Distortion: Occurs when input values exceed the representable range (clipping).
• The choice of clipping thresholds (min/max) directly controls the trade-off between granular error (if range is too wide) and clipping error (if range is too narrow).

Error is not isolated; it propagates and can amplify through successive layers. Key propagation effects:

• Error Accumulation: Small errors in early layers compound through non-linear activations and matrix multiplications in deeper layers.
• Bias Shift: Asymmetric quantization schemes or clipping can introduce a systematic bias that shifts the mean of layer outputs.
• Variance Inflation: Quantization can alter the variance of activations, disrupting the statistics assumed by normalization layers (e.g., BatchNorm), leading to instability during inference.

Not all parts of a model are equally sensitive. Layer-wise sensitivity analysis reveals that:

• Final classification layers are often highly sensitive, as small errors in logits can cause misclassification.
• Attention mechanisms in transformers are extremely sensitive to low-precision Key and Query matrices.
• Weight quantization error is static and present for all inferences, while activation quantization error is dynamic and input-dependent. This non-uniform impact motivates mixed-precision quantization strategies.

Error characteristics are heavily dependent on model design:

• Residual connections can help mitigate error by providing a high-precision bypass, but quantizing the identity path introduces error directly into the skip.
• Depthwise separable convolutions (e.g., in MobileNet) have different error profiles than standard convolutions due to their factorization.
• Models with smooth activation functions (e.g., GELU, Swish) generally quantize better than those with discontinuous functions (e.g., hard sigmoid, step functions) which amplify quantization steps.

The error signal introduced by the quantization process, mathematically modeled as additive noise that perturbs weights and activations. This noise is the direct cause of accuracy degradation in quantized models.

• Statistical Model: Often modeled as uniform or Gaussian noise based on the quantization step size.
• Propagation: Noise from quantized weights and activations propagates through the network, accumulating and potentially amplifying in deeper layers.
• Mitigation: Techniques like Quantization-Aware Training (QAT) simulate this noise during training, allowing the model to learn robust representations.

The degree to which a compressed model's outputs and internal behaviors match those of the original, uncompressed golden model. It is a broader measure than simple task accuracy.

• Measurement Metrics:
  • KL Divergence: Quantifies the difference between output probability distributions.
  • Cosine Similarity: Measures the alignment of activation vectors or embedding spaces.
  • Mean Squared Error (MSE): Calculates the average squared difference between layer outputs.
• High Fidelity indicates the compressed model is functionally equivalent to the original, which is critical for safety-sensitive deployments.

A systematic evaluation to determine which components of a neural network are most sensitive to compression and thus most critical for preserving accuracy. This analysis directly informs heterogeneous compression strategies.

• Layer-Wise Sensitivity: Measures accuracy drop when compressing individual layers in isolation.
• Channel/Filter Sensitivity: Identifies redundant filters within convolutional layers that can be pruned with minimal impact.
• Output: Produces a sensitivity profile, guiding techniques like Mixed-Precision Quantization where sensitive layers are kept at higher precision (e.g., FP16) while robust layers are aggressively quantized (e.g., INT4).

The process of regaining performance lost during compression, typically through post-compression training techniques. This is essential for moving a model along the Pareto frontier toward a better size-accuracy tradeoff.

• Primary Methods:
  • Fine-Tuning After Compression: The compressed model is further trained on the task dataset with a low learning rate.
  • Quantization-Aware Training (QAT): Simulates quantization during training, allowing the model to adapt to lower precision.
  • Knowledge Distillation: Uses soft labels from the original teacher model to guide the recovery of the compressed student model.

In compression-accuracy analysis, the set of optimal points on a tradeoff curve where no other model configuration can improve one metric (e.g., accuracy) without worsening another (e.g., model size or latency).

• Visualization: Plotted with accuracy (or loss) on one axis and a compression metric (size, FLOPs, latency) on the other.
• Engineering Goal: To find compressed models that lie on this frontier, representing the best possible tradeoff for a given constraint.
• Frontier Shifting: Advanced techniques like QAT or novel compression algorithms aim to shift the entire frontier, enabling smaller models at the same accuracy level.

A predefined, application-specific threshold for the maximum allowable accuracy degradation that a compressed model must meet to be considered viable for deployment. This defines the business and technical constraints of the tradeoff.

• Definition: Often expressed as an absolute drop in top-1 accuracy (e.g., max 2% drop).
• Use Case Dependency: A real-time mobile vision app may tolerate a 3% drop for a 4x speed-up, while a medical diagnostic model may allow only a 0.5% drop.
• Validation: The compressed model's validation accuracy must be measured against this threshold during performance profiling and on-device evaluation.