Calibration (Quantization)

A representative dataset is the cornerstone of effective calibration. It is a small, unlabeled subset of the model's training or validation data used to observe the dynamic range of activations.

• Purpose: To capture the statistical distribution (min, max, mean, standard deviation) of each layer's output.
• Size: Typically 100-1000 samples are sufficient; more data yields more stable statistics but increases calibration time.
• Criticality: Using non-representative data (e.g., all zeros, out-of-distribution images) leads to poorly estimated ranges, causing severe quantization error and accuracy loss.

These algorithms analyze the calibration data to compute the scale and zero-point parameters. The choice of algorithm directly impacts the accuracy-efficiency trade-off.

• Min-Max: Sets the range using the absolute minimum and maximum values observed. Simple but highly sensitive to outliers.
• Entropy / KL-Divergence: Minimizes the information loss between the original and quantized distributions. Used in TensorRT, it often provides the best accuracy for activations.
• Percentile (e.g., 99.9%): Ignores extreme outliers by using a percentile threshold (e.g., the 99.9th percentile value) as the range maximum, providing robustness.
• Moving Average: Tracks range statistics over multiple batches, useful for dynamic quantization scenarios.

This distinction defines when calibration parameters are computed and whether they are fixed for inference.

• Static Calibration: Parameters are calculated once during the calibration step and baked into the model. This is the standard for Post-Training Quantization (PTQ), enabling maximum inference speed and deterministic latency. It requires a representative dataset.
• Dynamic Calibration: Scale and zero-point for activations are recomputed at runtime for each input. This adapts to varying input distributions but introduces computational overhead. It's used when a single static range is insufficient (e.g., for models with highly variable activation ranges).

Granularity defines the scope over which a single set of quantization parameters is shared.

• Per-Tensor Calibration: A single scale and zero-point is calculated for an entire tensor. This is simpler and widely supported but can be suboptimal if the tensor's channels have divergent ranges.
• Per-Channel Calibration: A unique scale and zero-point is calculated for each output channel of a weight tensor (commonly for convolutions and linear layers). This provides a tighter fit to the data distribution, significantly reducing error, especially for weight tensors. It is a standard best practice for modern quantization.

The primary outputs of calibration are the parameters that define the affine quantization mapping: quantized_value = round(float_value / scale) + zero_point.

• Scale (Δ): A floating-point number representing the size of each quantization step. Calculated as: (float_max - float_min) / (quant_max - quant_min).
• Zero-Point (Z): An integer that aligns the zero of the quantized range with the zero of the real range. Crucial for efficient integer-only arithmetic.
• Quantization Range: The minimum and maximum integer values representable (e.g., [-128, 127] for 8-bit signed). Calibration determines how the float range is mapped onto this integer grid.

These are the two fundamental parameters determined during calibration. The scale is a floating-point multiplier that defines the step size between integer values. The zero-point is an integer offset that maps the quantized value '0' to a specific floating-point number, enabling asymmetric quantization.

• Purpose: They define the affine transformation: float_value = scale * (int_value - zero_point).
• Calibration's Role: The calibration dataset is analyzed to calculate the optimal scale and zero-point that minimize the clipping of the tensor's value range.

A Post-Training Quantization (PTQ) method where the quantization parameters for activations are fixed after a one-time calibration step. This is the most common target of calibration processes.

• Process: A representative dataset is fed through the model, and the observed ranges of all activation tensors are recorded to determine their final scale/zero-point.
• Benefit: Eliminates runtime overhead of calculating activation ranges, leading to faster and more deterministic inference.
• Contrast with Dynamic Quantization: In dynamic quantization, activation ranges are computed on-the-fly for each inference batch.

The numerical distortion introduced when mapping a continuous range of floating-point values to a finite set of discrete integer levels. Calibration directly influences the magnitude of this error.

• Sources: Error arises from clipping (values outside the calibrated range are saturated) and rounding (values within the range are mapped to the nearest integer level).
• Calibration's Goal: To choose a range (min/max) that optimally balances clipping error and rounding resolution. A poorly calibrated range can lead to significant accuracy loss.
• Measurement: Often analyzed using metrics like Signal-to-Quantization-Noise Ratio (SQNR).

These are granularity schemes that define how many sets of scale/zero-point parameters are used. Calibration must be performed accordingly.

• Per-Tensor: A single scale and zero-point is calculated for an entire tensor. This is simpler but less accurate if the tensor's distribution varies significantly across channels.
• Per-Channel: A unique scale and zero-point is calculated for each output channel of a weight tensor (common for convolutions and linear layers). Calibration must analyze each channel's distribution independently.
• Impact: Per-channel quantization typically yields higher accuracy post-quantization but requires more calibration data and storage for the parameters.

The small, representative set of unlabeled input data used to estimate activation ranges or distributions for static quantization. Its quality is paramount.

• Characteristics: Typically 100-1000 samples from the model's target domain. It does not require labels, as the goal is to observe activation statistics, not perform training.
• Requirement: Must be statistically representative of real inference data. Using a non-representative set (e.g., all zeros, out-of-domain images) will lead to poorly calibrated ranges and high quantization error.
• Best Practice: Often a random subset of the training or validation set.

A training-time alternative to post-training calibration. QAT simulates quantization during the forward pass, allowing the model to learn to compensate for the quantization error.

• Relationship to Calibration: In QAT, the quantization parameters (scale/zero-point) are still determined via a calibration-like process, but this process is often integrated into the training loop and can be learned or updated via gradient descent.
• Advantage: Typically achieves higher accuracy than Post-Training Quantization (PTQ) with calibration, as the model weights are adapted to the quantized regime.
• Trade-off: Requires retraining resources (time, compute), whereas PTQ with calibration is a fast, data-only process.