Symmetric vs. Asymmetric Quantization

Symmetric quantization constrains the quantization range to be symmetric around zero (e.g., [-α, +α]). This often allows the zero-point to be exactly 0, simplifying the integer arithmetic. Asymmetric quantization uses a range [β, γ] that is not centered on zero, requiring a non-zero integer zero-point to map the real value zero, which better accommodates skewed data distributions (e.g., ReLU outputs that are all non-negative).

Both methods use an affine transformation: Q = round(r / S) + Z, where r is the real value, S is the scale, and Z is the zero-point.

• Symmetric: Z is typically 0. The scale S is derived from max(abs(min), abs(max)).
• Asymmetric: Z is calculated as round(-min / S). The scale S is derived from (max - min) / (2^b - 1), where b is the bit-width. The integer Q is then clamped to the target integer range (e.g., [-128, 127] for INT8).

Symmetric quantization leads to simpler, faster computation. With a zero-point of 0, the core integer matrix multiplication (INT8 x INT8) avoids the overhead of zero-point addition. This is highly efficient on NPUs with dedicated integer arithmetic units. Asymmetric quantization requires additional instructions to handle the non-zero zero-point during accumulation, adding computational overhead. However, modern hardware like the Qualcomm Hexagon NPU or Google Edge TPU often includes optimized instructions for this affine operation.

The choice impacts model accuracy based on activation statistics.

• Symmetric is optimal for data roughly symmetric around zero (e.g., weight distributions post-BatchNorm, certain activation functions like Tanh). It can waste representational range if data is skewed.
• Asymmetric is superior for handling skewed distributions, such as the output of a ReLU layer (all values ≥ 0). By aligning the quantization range with the actual min/max, it reduces clipping error and utilizes the full integer dynamic range more effectively.

Common frameworks implement both schemes:

• TensorFlow / TFLite: Uses asymmetric quantization for activations by default in its post-training quantization tool. Symmetric quantization can be specified for performance.
• PyTorch: torch.ao.quantization supports both. The QConfig allows selection of symmetric (per_tensor_symmetric) or asymmetric (per_tensor_affine) observers.
• ONNX Runtime: Supports both through its quantization tools, with symmetric often used for weights and asymmetric for activations in static quantization workflows.

Use Symmetric Quantization when:

• Targeting maximum inference speed on integer-only hardware.
• The tensor's distribution is approximately symmetric.
• Quantizing weights (which often have a symmetric distribution).

Use Asymmetric Quantization when:

• Activating tensors (e.g., from ReLU) have a strongly skewed, non-negative distribution.
• Accuracy preservation is the primary concern and the hardware supports the overhead.
• Performing post-training quantization (PTQ) without fine-tuning, as it often yields better accuracy by minimizing clipping.

Quantization is the process of mapping continuous, high-precision floating-point values (e.g., FP32) to a discrete, finite set of lower-bit integer representations (e.g., INT8). This reduces the computational and memory footprint of a neural network, enabling faster inference and deployment on resource-constrained hardware like NPUs and edge devices. The core trade-off is between model size/speed and potential accuracy loss.

• Primary Goal: Reduce model size and accelerate inference.
• Key Mechanism: Uses scale and zero-point parameters to define a linear mapping between float and integer domains.
• Impact: Can achieve ~4x model size reduction and 2-4x latency improvement with INT8 vs. FP32.

Post-Training Quantization (PTQ) is a model compression technique applied after a model is fully trained. It converts weights and activations to lower precision (e.g., INT8) without retraining, using a small calibration dataset to determine optimal quantization parameters (scale/zero-point). It's fast and requires no labeled data for fine-tuning, but may incur higher accuracy loss compared to quantization-aware training.

• Process: Calibrate on unlabeled data → determine ranges → quantize model.
• Use Case: Rapid deployment of pre-trained models where retraining is impractical.
• Variants: Includes static quantization (fixed calibration) and dynamic quantization (runtime calibration).

Quantization-Aware Training (QAT) simulates quantization effects during the training or fine-tuning phase. Fake quantization nodes are inserted into the forward pass, rounding and clipping values as if they were quantized, while backward passes use high-precision gradients. This allows the model to adapt its weights to mitigate the accuracy loss inherent in quantization, typically yielding higher accuracy than PTQ.

• Mechanism: Forward pass simulates INT8; backward pass uses FP32 gradients.
• Advantage: Higher recovered accuracy, especially for complex models.
• Cost: Requires additional training time and computational resources.

Quantization granularity defines the scope over which a single set of quantization parameters (scale and zero-point) is shared. The choice significantly impacts model accuracy and hardware efficiency.

• Per-Tensor: One scale/zero-point for an entire tensor. Simplest, most efficient, but can lose accuracy.
• Per-Channel: Unique parameters for each output channel of a weight tensor (common for convolutions). Higher accuracy, slightly more overhead.
• Per-Token/Per-Axis: For activations in transformers, parameters can vary per sequence token or specific axis.
• Group-Wise: Parameters shared across small groups of values (e.g., 128 weights). A balance between per-tensor and per-channel.

Integer-Only Inference is an execution paradigm where the entire neural network forward pass uses integer arithmetic, eliminating floating-point operations. This is enabled by symmetric quantization (where zero-point is 0) and the use of integer-friendly activation functions (e.g., ReLU). It is critical for deployment on ultra-low-power microcontrollers, dedicated NPUs, and hardware without FPUs.

• Requirement: All layers (MatMul, Conv, Add, ReLU) must have integer implementations.
• Benefit: Maximum power efficiency and speed on integer-optimized silicon.
• Challenge: Requires careful quantization scheme design to avoid accuracy degradation from integer-only approximations of non-linear functions.

Quantization error is the numerical distortion introduced when converting values from high to low precision. It is the primary source of accuracy loss in quantized models and consists of two main components:

• Rounding Error: Caused by mapping multiple float values to the same integer. Governed by the quantization scale.
• Clipping Error: Occurs when float values outside the representable range are saturated to the minimum or maximum integer value. This is a function of how the range [min, max] is chosen.

Managing this error involves selecting appropriate quantization parameters (symmetric vs. asymmetric, granularity) and sometimes using techniques like Quantization-Aware Training to let the model learn to be robust to it.