Accuracy Drop

Quantization error is the fundamental numerical discrepancy caused by mapping continuous, high-precision floating-point values (e.g., FP32) to a discrete, lower-precision integer representation (e.g., INT8). This process introduces quantization noise, an additive error that perturbs every weight and activation in the network. The error manifests as:

• Rounding Error: The loss of fractional information when values are rounded to the nearest quantized level.
• Clipping Error: The saturation of outliers beyond the representable range of the quantized format, causing a loss of signal dynamic range.
• Bias in Statistical Estimation: Incorrect estimation of activation ranges during calibration, leading to suboptimal scaling factors that amplify error. This distortion propagates through the computational graph, causing a cumulative deviation in layer outputs and, ultimately, incorrect predictions.

Compression techniques directly reduce a model's parameter count or the information density per parameter, constraining its hypothesis space. This loss of model capacity limits its ability to represent complex functions learned during training.

• Pruning: Removing neurons, channels, or weights creates a sparse architecture. While theoretically preserving function, aggressive pruning can excise critical, non-redundant pathways essential for specific subtasks.
• Low-Rank Factorization: Decomposing weight matrices into smaller factors imposes a low-rank constraint, reducing the matrix's representational power.
• Extreme Quantization (e.g., 1-4 bit): Severely restricts the number of unique values weights can assume, acting as a powerful regularizer that can oversimplify the learned mapping. The compressed model becomes a lower-capacity approximator of the original, high-accuracy function, leading to increased error on complex or edge-case inputs.

Post-training quantization relies on a small calibration dataset to estimate the statistical distribution (min/max values) of layer activations. Inaccuracies in this estimation are a primary source of accuracy drop.

• Out-of-Distribution Calibration: If the calibration data is not representative of real inference data, the calculated quantization ranges will be suboptimal, leading to excessive clipping or wasted precision.
• Dynamic Activation Ranges: Activations with high variance or distributions that change significantly between inputs (e.g., in attention layers) are poorly served by static quantization parameters.
• Cross-Layer Error Amplification: Mismatches compound across layers; a small clipping error in an early layer can distort the input distribution to subsequent layers, leading to cascading inaccuracies. This is why quantization-aware training, which models this noise during training, often yields superior results.

Neural networks learn hierarchical feature representations. Compression can disrupt the delicate alignment and relative importance of these features.

• Layer-Wise Sensitivity Variation: Not all layers contribute equally to the final output. Compression applied uniformly across a network will disproportionately damage sensitive layers (often later, task-specific layers) while leaving robust layers (early convolutional filters) relatively intact. This misalignment degrades the quality of the final feature embedding.
• Destruction of Fine-Grained Features: Low-bit quantization and aggressive pruning tend to eliminate subtle, high-frequency patterns in the data that are critical for distinguishing between semantically similar classes (e.g., different bird species).
• Aliasing in Weight Space: Quantization can cause distinct, well-separated weight values in the original model to collapse to the same quantized value, effectively merging previously distinct feature detectors.

Compression can exacerbate inherent numerical instabilities in certain network operations, turning minor floating-point rounding errors into major functional failures.

• Softmax & Normalization Layers: These layers are highly sensitive to the precision of their input logits. Quantization error before a softmax can drastically alter the resulting probability distribution.
• Residual Connections: In architectures like ResNet, quantization error in the residual branch is added directly to the identity mapping. If not properly handled, this can lead to a growing error signal across the network depth.
• Attention Mechanisms: The dot-product and scaling operations in transformer attention are numerically delicate. Low-precision quantization of queries, keys, and values can lead to significant errors in the attention weights and, consequently, the context vector. These instabilities are often not apparent in full-precision models but become critical failure points after compression.

Accuracy drop can occur at the deployment stage due to mismatches between the theoretical compressed model and its practical execution on target hardware.

• Kernel Implementation Differences: The integer math kernels used for inference (e.g., on a mobile NPU) may implement operations like rounding or saturation differently than the software used during quantization simulation, leading to implementation divergence.
• Non-Linear Function Approximation: Hardware often uses low-precision polynomial or lookup-table approximations for non-linear functions (e.g., sigmoid, GELU). Differences between these approximations and the software emulation used during compression can cause accuracy loss.
• Sparse Pattern Inefficiency: If the hardware lacks efficient support for the specific sparsity pattern induced by pruning, the runtime may execute the model in a dense format, negating the intended benefits and sometimes even slowing down inference without the accuracy recovery of fine-tuning. This underscores the necessity of on-device evaluation as the final validation step.