Quantization Sensitivity Analysis

The core of the analysis involves profiling each layer or operation (e.g., convolutional, linear, attention) to measure its contribution to the overall model's accuracy loss when quantized. This is done by:

• Isolating layers: Quantizing a single layer while keeping others at full precision.
• Measuring error: Using a calibration dataset to compute the drop in task-specific metrics (e.g., accuracy, F1 score).
• Ranking sensitivity: Creating a ranked list from most to least sensitive layers. Highly sensitive layers often contain outlier weights or require high dynamic range for activations.

Sensitivity can be analyzed at different levels of granularity, each offering a trade-off between optimization potential and analysis complexity:

• Per-Tensor: A single sensitivity score for an entire weight or activation tensor. Fast but coarse-grained.
• Per-Channel: Sensitivity is measured for each output channel in a weight tensor. This is common for convolutional and linear layers, as channels can have vastly different weight distributions.
• Per-Token/Per-Activation: Analyzing sensitivity across the sequence dimension in transformers, crucial for models with highly variable activation ranges.
• Block-Level: Grouping multiple sequential operations (e.g., a residual block) for a broader optimization view.

Sensitivity is quantified using statistical methods on a representative calibration dataset. Key techniques include:

• Signal-to-Quantization-Noise Ratio (SQNR): Measures the power of the original signal versus the introduced quantization noise for a layer's output.
• Mean Squared Error (MSE): Computes the reconstruction error between the full-precision and quantized outputs.
• Task Loss Gradient: In Quantization-Aware Training (QAT) contexts, the gradient of the loss with respect to the fake quantization nodes indicates sensitivity.
• Data Range Analysis: Observing the min/max or percentile (e.g., 99.9%) ranges of activations to identify layers with large, unpredictable outliers.

The primary deliverable of the analysis is a bit-width configuration map. This blueprint assigns optimal precision to each layer, such as:

• FP16/BF16: Reserved for highly sensitive layers (e.g., certain attention projections, final classifier).
• INT8: Assigned to the majority of robust layers (e.g., many intermediate convolutions).
• INT4 or lower: Applied to extremely robust layers identified in Low-Bit Quantization analysis. This map directly informs the Mixed-Precision Quantization process, maximizing compression and speed while adhering to a target accuracy budget.

Sensitivity analysis is integrated into major ML optimization frameworks:

• PyTorch (FX Graph Mode): Allows hooks for layer-wise observation and manipulation during calibration.
• TensorRT: Uses built-in calibrators (e.g., EntropyCalibrator) that inherently perform sensitivity analysis to select per-layer dynamic ranges for INT8 conversion.
• ONNX Runtime: Provides quantization tools that generate sensitivity reports to guide precision selection.
• Brevitas / AIMET: Research-focused libraries offering advanced sensitivity metrics and automated mixed-precision search algorithms.

Sensitivity analysis is not performed in isolation; it directly enables and is informed by other quantization methods:

• Prerequisite for PTQ: Informs Static Quantization parameter selection and is essential for effective Mixed-Precision Quantization.
• Guides QAT: Identifies which layers require the adaptive recovery provided by Quantization-Aware Training.
• Complementary to Pruning: Often performed alongside Weight Pruning analysis, as sensitivity to sparsity and reduced precision can be correlated.
• Foundation for Hardware-Specific Tuning: Results are refined based on target hardware, as some NPU accelerators have unique precision support for certain operators.

This core technique involves quantizing individual layers or modules in isolation while keeping the rest of the model in full precision. The resulting accuracy drop is measured to create a sensitivity profile. Key methods include:

• One-at-a-time (OAT) analysis: Quantizing a single layer and evaluating the model's performance.
• Granularity selection: Determining whether per-tensor or more precise per-channel quantization is tolerable for each weight tensor.
• Output: A ranked list or heatmap identifying sensitive layers (e.g., attention outputs, final classifier) that require higher precision and robust layers (e.g., certain convolutional filters) that can be aggressively quantized to INT8 or INT4.

Quantifying sensitivity requires precise metrics beyond top-1 accuracy. Common analytical measures include:

• Signal-to-Quantization-Noise Ratio (SQNR): Measures the power of the original signal versus the noise introduced by quantization for a tensor's values.
• Cosine Similarity: Compares the direction of the full-precision and quantized layer outputs.
• Mean Squared Error (MSE) of Activations: Measures the layer-wise distortion.
• Tools like Netron (for visualizing quantization annotations) and AIMET from Qualcomm provide dashboards to visualize these metrics across the model graph, making sensitivity interpretable.

Sensitivity analysis is not just for post-training. During Quantization-Aware Training (QAT), sensitivity guides the fine-tuning process:

• The Straight-Through Estimator (STE) allows gradients to flow through fake quantization nodes.
• Analysis can identify which fake-quantized layers are struggling to adapt, prompting adjustments like increasing their bit-width for the QAT phase or applying layer-wise learning rate scaling.
• This closed-loop process ensures the final quantized model has its capacity focused on preserving accuracy in the most critical regions.

Post-Training Quantization (PTQ) is the foundational compression technique where a pre-trained model's weights and activations are converted to lower precision (e.g., INT8) without retraining. Sensitivity analysis is a mandatory precursor to PTQ, as it identifies which layers can withstand this aggressive compression without a calibration dataset to guide adaptation.

• Primary Use: Rapid deployment optimization for already-trained models.
• Process: Uses a small calibration dataset to observe activation ranges and determine quantization scale and zero-point.
• Risk: Applying PTQ uniformly can cause significant quantization error in sensitive layers, leading to accuracy collapse.

Quantization-Aware Training (QAT) is the proactive counterpart to PTQ. It simulates quantization effects during the training or fine-tuning process, allowing the model to learn robust representations that are inherently tolerant of lower precision.

• Mechanism: Uses fake quantization nodes that apply rounding and scaling in the forward pass but use the Straight-Through Estimator (STE) to approximate gradients during backpropagation.
• Advantage: Typically yields higher accuracy than PTQ, especially for very low bitwidths like INT4.
• Relation to Sensitivity Analysis: Sensitivity analysis can inform which layers to target first for QAT or help decide if QAT is necessary for a given model architecture.

Mixed-Precision Quantization is the strategic application of different quantization bitwidths (e.g., 4-bit, 8-bit, 16-bit) to different layers or tensors within a single model. It is the direct operational outcome of a thorough sensitivity analysis.

• Goal: Optimize the trade-off between compression/performance and accuracy by protecting sensitive layers with higher precision.
• Implementation: A sensitivity profile guides the assignment of bitwidths, creating a "recipe" for the quantized model.
• Challenge: Requires hardware and software runtimes that support executing models with heterogeneous precision.

A Calibration Dataset is a small, representative set of unlabeled data used during PTQ and sensitivity analysis to observe the dynamic range of model activations. The quality and representativeness of this dataset are paramount for effective quantization.

• Function: Used to calculate scale and zero-point parameters for asymmetric quantization or to profile activation ranges for sensitivity metrics.
• Size: Typically 100-1000 samples; must reflect the operational data distribution.
• Critical Role: An unrepresentative calibration set will lead to incorrect quantization parameters and poor model performance, regardless of sensitivity analysis.

Quantization Granularity defines the scope over which a single set of quantization parameters (scale/zero-point) is applied. Choosing the right granularity is a key lever for minimizing error, informed by sensitivity analysis.

• Per-Tensor Quantization: A single scale/zero-point for an entire tensor. Simple but can be inaccurate if the tensor's values have high variance.
• Per-Channel Quantization: Unique parameters for each output channel of a weight tensor (common for convolutions and linear layers). More complex but significantly reduces error, especially for weight tensors.
• Group Quantization: Parameters are shared across small groups of values within a tensor, a trade-off between per-tensor and per-channel methods.

Quantization Error is the fundamental numerical distortion introduced by reducing precision. Sensitivity analysis quantifies this error layer-by-layer using specific metrics to guide optimization decisions.

• Signal-to-Quantization-Noise Ratio (SQNR): Measures the power ratio between the original signal (full-precision tensor) and the introduced quantization noise. Higher SQNR indicates better fidelity.
• Mean Squared Error (MSE): A direct measure of the average squared difference between original and dequantized values for a layer's output.
• Cosine Similarity: Evaluates the directional alignment of activation vectors before and after quantization, crucial for attention-based models.
• Task Loss Impact: The ultimate metric—the change in validation accuracy or loss when a specific layer is quantized.