Compression Artifacts

The fundamental numerical discrepancy introduced when mapping continuous floating-point values to a finite set of discrete integer levels. This process inherently loses information.

• Round-off Error: The difference between the original FP32 value and its nearest representable quantized integer.
• Clipping/Saturation: Values outside the quantization range are clamped to the minimum or maximum, destroying information about the distribution's tails.
• Granularity: Lower bit-widths (e.g., INT4 vs. INT8) increase the step size between discrete levels, amplifying this error. This noise propagates through the computational graph, accumulating in deeper layers.

Compression reduces the model's parameter space, directly limiting its ability to represent complex functions learned during training.

• Parameter Reduction (Pruning): Removing weights reduces the degrees of freedom in the model. While many weights are redundant, aggressive pruning can excise critical, non-redundant connections.
• Precision Reduction (Quantization): Lower bit-widths shrink the number of unique values a weight or activation can take. A 4-bit weight can represent only 16 values, whereas FP32 can represent billions, constraining the model's expressivity.
• Architectural Sparsity: Induced sparsity patterns may not align with the data flow needed for a specific task, creating bottlenecks in information propagation.

Quantization parameters (scale/zero-point) are typically calibrated on a small, static dataset. If the activation statistics of real-world inference data differ significantly, quantization becomes misaligned.

• Out-of-Distribution Inputs: Data unlike the calibration set can produce activations outside the calibrated range, leading to severe clipping and distortion.
• Batch Norm Folding Errors: During post-training quantization, Batch Normalization statistics are folded into preceding layers. If the folding assumes a fixed activation range that doesn't hold during inference, error compounds.
• Dynamic Range Mismatch: Layers with highly variable activation ranges (e.g., attention layers in transformers) are particularly susceptible, as a single scale factor may poorly represent the entire distribution.

Compression, especially when applied post-training, can break the smooth, learned gradient pathways that were optimized during training.

• Non-Differentiable Operations: The quantization function (rounding) has a zero gradient almost everywhere. During quantization-aware training, Straight-Through Estimators (STEs) provide a proxy gradient, but this is an approximation that can lead to suboptimal convergence.
• Pruned Connectivity: Removing weights creates dead pathways. While the remaining weights can compensate, the loss landscape becomes more chaotic and difficult to fine-tune.
• Increased Sensitivity: Compressed models often exhibit higher sensitivity to input perturbations, as the reduced capacity offers less buffering against noise.

Operations that were stable in high precision can become unstable in low precision due to overflow, underflow, and catastrophic cancellation.

• Softmax/Attention in Low Precision: These functions involve exponentiation and summation. In INT8, large values can saturate, and small differences can vanish, flattening the output distribution.
• Accumulation in Integer Arithmetic: Summing many small integer values can lead to overflow if the accumulator bit-width is insufficient, a common issue in hardware without 32-bit integer accumulators.
• Underflow in Activation Functions: In very low precision (e.g., 4-bit), the quantization grid may not have a non-zero value small enough to represent the tails of functions like GELU or SiLU, causing a 'dead zone'.

The theoretical benefits of a compression algorithm (e.g., sparsity) may not materialize due to inefficient execution on the target hardware.

• Inefficient Sparsity Patterns: Unstructured sparsity offers high theoretical compression but often leads to no speed-up on general-purpose hardware (GPUs/CPUs) due to lack of parallelization. Hardware requires structured (e.g., 2:4) or block-wise sparsity for acceleration.
• Kernel Overhead: The runtime cost of decoding sparse formats or dequantizing weights on-the-fly can negate the benefits of reduced memory bandwidth.
• Suboptimal Format Conversion: Transforming a model into a hardware-specific format (e.g., TFLite for mobile NPUs) may involve additional rounding or packing steps that introduce secondary artifacts not present in the original compressed checkpoint.

The fundamental numerical discrepancy introduced when converting continuous floating-point values to discrete integer representations. This is the primary mathematical source of accuracy degradation in post-training quantization.

• Mechanism: Arises from the rounding/clipping operations that map a continuous range of values to a finite set of quantization levels.
• Impact: Propagates through the network, causing output divergence from the original model's predictions.
• Example: An 8-bit integer can represent only 256 discrete values, so a weight of 0.314159 is approximated as the nearest representable integer value, introducing a small error.

A measure of how closely a compressed model's outputs and internal behaviors match those of the original, uncompressed golden model. It is a broader concept than simple task accuracy.

• Quantification: Often measured using statistical metrics like KL Divergence or cosine similarity between output probability distributions.
• Importance: High fidelity is critical for applications where the exact numerical output matters, not just the final classification label.
• Evaluation: Involves comparing activations layer-by-layer, not just the final prediction, to ensure the compressed network's "reasoning" remains intact.

The systematic process of evaluating which components of a neural network are most vulnerable to performance loss from compression. This analysis directly informs strategies to minimize compression artifacts.

• Purpose: Identifies critical layers, channels, or weights that must be preserved at higher precision.
• Method: Often involves compressing individual layers in isolation and measuring the resulting accuracy drop on a validation set.
• Outcome: Enables mixed-precision quantization and guided pruning by creating a sensitivity profile for the entire model.

The set of techniques applied after compression to regain lost model performance, moving a model back towards an acceptable loss threshold.

• Primary Method: Fine-tuning after compression, where the compressed model is trained for a few additional epochs on the task data.
• Advanced Method: Quantization-aware training (QAT), which simulates quantization noise during the original training process, making the model robust to it from the start.
• Goal: To achieve a favorable compression ratio while minimizing the final model degradation.

The graphical and analytical tools used to visualize and select optimal compression configurations.

• Tradeoff Curve: A plot with accuracy (or another quality metric) on one axis and a compression metric (e.g., model size, latency) on the other. Each point represents a differently compressed model variant.
• Pareto Frontier: The set of points on the tradeoff curve where it is impossible to improve one metric without worsening the other. These represent the optimal compression-accuracy tradeoff choices.
• Use Case: Engineers use this analysis to select a model that meets specific degradation thresholds for deployment.

The final, critical stage of tradeoff analysis where a compressed model is benchmarked on the actual target hardware. This reveals real-world compression artifacts that may not be apparent in simulation.

• Measures: Real inference latency, memory bandwidth usage, power consumption, and sustained accuracy under thermal constraints.
• Reality Check: Catches issues like unexpected slowdowns from sparse kernel inefficiencies or accuracy drops due to hardware-specific numerical quirks.
• Outcome: Provides the definitive performance profiling data required for production deployment decisions.