Asymmetric Quantization

The zero-point is an integer value that corresponds exactly to the real value zero in the floating-point domain. It is the critical component that enables asymmetric quantization to handle data ranges that are not symmetric around zero.

• Purpose: It ensures that the integer value 0 has a meaningful, non-zero floating-point equivalent, allowing for accurate representation of unsigned data (e.g., ReLU activations) or biased distributions.
• Calculation: Derived during calibration, it is typically the rounded integer result of -min / scale. This shifts the quantized integer range to align with the floating-point range.
• Impact: While it adds a small computational overhead (an extra integer addition during dequantization), it often results in lower quantization error compared to symmetric schemes for real-world, non-zero-centered data.

The scale is a positive floating-point number that defines the ratio between a quantized integer step and the corresponding step in the floating-point range. It determines the resolution of the quantization.

• Function: Acts as the multiplicative factor to convert between integer and floating-point representations: float_value ≈ scale * (int_value - zero_point).
• Determination: Calculated from the observed min and max values of a tensor (weights or activations) during calibration: scale = (max - min) / (quant_max - quant_min). For 8-bit integers, quant_max=127 and quant_min=-128.
• Precision Trade-off: A larger scale (from a wide dynamic range) reduces resolution and increases quantization error. Per-channel quantization uses independent scale values for each output channel to mitigate this.

The quantization grid is the set of discrete integer values (e.g., -128 to 127 for INT8) to which the continuous floating-point range is mapped. The dynamic range is the span of floating-point values being represented.

• Mapping: The process projects the observed dynamic range [min, max] onto the fixed integer grid [quant_min, quant_max]. Values outside the calibrated range are clipped.
• Clipping Error: A primary source of quantization error. If calibration underestimates the true runtime range of activations, outliers are lost, degrading accuracy.
• Calibration Strategy: The choice of how to determine min and max (e.g., using moving averages, percentile-based methods like 99.99%, or entropy minimization) directly controls the trade-off between clipping and rounding error.

Dequantization is the process of reconstructing an approximate floating-point value from its quantized integer representation. In asymmetric quantization, this requires both the scale and zero-point.

• Formula: float_value = scale * (int_value - zero_point). This operation is typically required at the interface between quantized layers and non-quantized operations or for final outputs.
• Integer-Only Inference: For full efficiency, the network graph is transformed to keep computations in the integer domain. This involves fusing the scale and zero-point into the weights of subsequent layers, often eliminating explicit dequantization ops during the main compute graph.
• Hardware Consideration: While the core integer arithmetic (convolutions, matrix multiplies) is fast, the final dequantization or requantization steps between layers must be optimized to avoid becoming a bottleneck.

A calibration dataset is a small, representative subset of the training or validation data used to estimate the dynamic range (min/max) of activation tensors for determining scale and zero-point parameters.

• Requirement: Essential for Post-Training Quantization (PTQ). The dataset must be statistically representative of real inference data to avoid distribution shift and miscalibration.
• Size: Typically 100-1000 unlabeled samples are sufficient. There is no backward pass or gradient calculation; it is a forward-pass-only analysis.
• Methodology: Common calibration algorithms include:
  • Min/Max: Uses absolute minimum and maximum values observed. Simple but sensitive to outliers.
  • Entropy Minimization (e.g., KL Divergence): Selects a threshold that minimizes the information loss between float and quantized distributions.
  • Percentile: Uses the 99.99th percentile value to exclude extreme outliers, providing a more robust range.

The ultimate goal of asymmetric quantization is to enable efficient integer-only inference on target hardware like NPUs, DSPs, or mobile CPUs.

• Core Operation: The fundamental quantized matrix multiplication or convolution is expressed as: S3*(i3 - Z3) = S1*(i1 - Z1) * S2*(i2 - Z2) This is rearranged to perform pure integer arithmetic on i1, i2, with the scales and zero-points folded into pre-computed terms.
• Zero-Point Overhead: The asymmetry introduced by the zero-point requires additional integer additions and multiplications. Hardware-aware compilers (like TensorRT, TFLite) optimize these operations, often fusing them with bias addition.
• Benefit vs. Symmetric: While symmetric quantization (zero_point = 0) is computationally simpler, asymmetric is often more accurate for real-world models, as it utilizes the full integer range. The choice is a key hardware-software co-design decision.

Convolutional Neural Networks (CNNs) for vision tasks ubiquitously use ReLU or ReLU6 activation functions, which clamp outputs to a range of [0, +n]. Symmetric quantization wastes half its dynamic range on negative values that never occur. Asymmetric quantization aligns the quantized integer range [0, 255] perfectly with the actual output range, achieving much lower quantization error for the same bit-width.

• Example: Quantizing a MobileNetV2 layer's output after ReLU6 (range [0, 6]).

This technique is essential for layers where weight or activation distributions are inherently skewed. Common cases include:

• Final layer outputs before a softmax (logits).
• Embedding layers where values are not centered.
• Models using certain activation functions like sigmoid (outputs [0,1]). By using a learned zero-point, the quantization grid can be shifted to cover the actual min/max of the data, minimizing clipping error and rounding error compared to forcing a symmetric range.

Asymmetric quantization is not just for post-training. During Quantization-Aware Training, the forward pass simulates the effects of asymmetric quantization (including scale and zero-point calculation) so the model can adapt its parameters. This results in higher accuracy for low-bit-width models (e.g., INT4). The fake quantization nodes inserted during QAT mimic the exact asymmetric quantization scheme that will be used during deployment.

Symmetric quantization is a scheme where the quantization range is symmetric around zero. This eliminates the need for a separate zero-point parameter, simplifying the dequantization formula to: real_value = scale * quantized_value. Its key characteristics are:

• Simpler Hardware Implementation: Symmetry often allows for more efficient integer arithmetic circuits.
• Potential Accuracy Trade-off: It can be less optimal for data distributions not centered at zero, as the quantized range may be underutilized compared to asymmetric quantization.

Quantization-aware training is a process where a neural network is trained or fine-tuned with simulated quantization operations in the forward pass. This allows the model's parameters to adapt to the precision loss that will occur during deployment. Key aspects include:

• Simulated Quantization: Fake quantization nodes are inserted into the training graph to mimic integer rounding and clamping.
• Robust Parameter Learning: The optimizer adjusts weights to minimize error under the quantized regime, typically yielding higher accuracy than post-training quantization for aggressive bit-widths like INT4.
• Training Overhead: Requires a full or partial retraining cycle, increasing development time but improving final model quality.

Post-training quantization is a compression technique that converts a pre-trained floating-point model to a lower-precision format (e.g., INT8) using a small, representative calibration dataset, without requiring retraining. Its workflow involves:

• Dynamic Range Calibration: Running calibration data through the model to record the min/max ranges of activations.
• Parameter Calculation: Using these statistics to compute scale and (for asymmetric) zero-point values for each tensor.
• Model Conversion: Transforming weights and graph operations to the quantized format. PTQ is faster than QAT but may incur greater accuracy loss, especially for models with high activation variance.

Integer-only inference is an execution paradigm where all operations of a quantized neural network are performed using integer arithmetic, completely eliminating floating-point calculations. This is critical for:

• Power-Efficient Edge Deployment: Integer arithmetic units consume significantly less power and silicon area than FPUs.
• Latency Reduction: Integer operations are typically faster on dedicated hardware like NPUs and DSPs.
• Achieving Full Quantization: Requires designing or using layers (e.g., integer batch normalization, activations) that avoid floating-point rescaling during runtime. Asymmetric quantization's zero-point is essential for handling signed integer arithmetic in this context.

Dynamic range calibration is the process of analyzing the statistical distribution of a model's activations over a representative dataset to determine optimal quantization parameters. For asymmetric quantization, this specifically involves finding:

• Effective Min/Max Values: The actual range of floating-point values observed in each activation tensor.
• Zero-Point Calculation: The integer value that maps directly to the real value zero, calculated as zero_point = round(-min / scale).
• Calibration Strategies: Methods include MinMax (using observed min/max), Entropy (minimizing KL divergence), or Percentile (to mitigate outlier effects). The choice of strategy directly impacts the final accuracy of the quantized model.

Per-channel quantization is a granular method where scaling factors (and zero-points for asymmetric schemes) are calculated independently for each output channel of a weight tensor, typically in convolutional or fully connected layers. This contrasts with per-tensor quantization, which uses one set of parameters for the entire tensor. Its advantages include:

• Reduced Quantization Error: By accounting for varying ranges across channels, it preserves more information.
• Hardware Considerations: While more accurate, it requires hardware support for channel-wise scaling operations, which is now common in modern NPUs and AI accelerators.
• Common Application: Frequently applied to weights, while activations often use per-tensor quantization for simpler implementation.