Symmetric Quantization

The defining characteristic of symmetric quantization is that the representable range is symmetric around zero. This is expressed as [-α, +α], where α is the absolute maximum value (or a calibrated scale factor) to be represented. This contrasts with asymmetric quantization, which uses a range [β, γ] not centered on zero.

• Key Property: The real-valued zero maps directly to the integer zero.
• Consequence: The zero-point parameter (z) is fixed at 0, simplifying the quantization and dequantization formulas significantly.

The process of converting a floating-point value x (within the range [-α, +α]) to an integer q is governed by a single scale factor (S).

Formula: q = clamp(round(x / S), -Q_max, Q_max)

Where:

• S (Scale): α / Q_max. This is the step size between integer values.
• Q_max: The maximum positive integer representable. For INT8, Q_max = 2^(8-1) - 1 = 127.
• clamp and round: Ensure the result is a valid integer within the symmetric range.

The absence of a zero-point term (z) is what makes this formula simpler than its asymmetric counterpart.

To recover an approximate floating-point value x' from the quantized integer q, a single multiplication is performed.

Formula: x' = q * S

This operation is computationally cheap, often just an integer multiplication or a bit shift. The simplicity of this dequantization step is a major advantage for high-performance inference kernels on hardware like NPUs and GPUs, where fused multiply-add operations are optimized.

Quantization Error: The error introduced is x - x' = x - (q * S). Because the range is symmetric, this error is uniformly distributed around zero for data that is also symmetrically distributed.

The accuracy of symmetric quantization hinges on choosing an appropriate scale S. The scale is derived from the range α.

Common Methods:

• Max: α = max(abs(min(X)), abs(max(X))). Uses the absolute maximum value from tensor X. Simple but can be sensitive to outliers.
• Calibrated (e.g., Percentile): α is set to the 99.99th percentile of abs(X) observed on a calibration dataset. This clips extreme outliers, reducing the range and improving precision for most common values.

For weights, the range is static. For activations in static quantization, this range is determined once during calibration.

The mathematical simplicity of symmetric quantization translates directly to hardware efficiency.

• Simplified Kernels: Integer matrix multiplication kernels (e.g., INT8 GEMM) do not need to account for a zero-point offset, leading to cleaner, faster code.
• Zero-Point Free: Eliminating the zero-point removes a broadcast addition operation in the compute graph, reducing instruction count.
• Wide Support: This scheme is natively and optimally supported by inference engines like TensorRT, ONNX Runtime, and TFLite for INT8 inference on CPUs, GPUs, and NPUs.

It is the preferred scheme for linear layers (Conv, MatMul) where the data distribution is roughly symmetric.

Symmetric quantization is not optimal for all data distributions.

Key Limitation: It wastes representational capacity if the data is not centered on zero. For example, an activation function like ReLU, which produces only non-negative values, has an effective range of [0, γ]. Using a symmetric range [-γ, γ] dedicates half the quantization bins to negative values that never occur.

Contrast with Asymmetric Quantization:

• Asymmetric uses: q = round(x / S) + z and x' = (q - z) * S.
• The zero-point z allows the quantized range [0, 255] for INT8 to map to any real range [β, γ], providing higher precision for asymmetric data at the cost of slightly more complex computation.

The symmetric scheme is defined by a scale factor (S) and a zero-point (z) fixed at 0. The quantization and dequantization formulas simplify dramatically:

• Quantize: q = round(r / S)
• Dequantize: r' = q * S Because z = 0, there is no additive term. The scale is calculated as S = |max(abs(r_min), abs(r_max))| / (2^(b-1) - 1), where b is the bit-width (e.g., 8). This ensures the range [-max_abs, max_abs] maps symmetrically to integer values like [-127, 127] for INT8.

TensorFlow Lite provides built-in symmetric quantization for INT8 inference. Key components:

• TFLite Converter: The tf.lite.TFLiteConverter can apply post-training integer quantization using a representative dataset for calibration.
• Full Integer Deployment: Weights and activations are converted to INT8. The inference_input_type and inference_output_type are set to tf.int8.
• Kernel Support: TFLite includes optimized kernels for symmetric INT8 operations like convolutions and fully connected layers, leveraging hardware acceleration on mobile and edge devices.
• Model Compatibility: Primarily applied to convolutional and dense layers. Dynamic-range quantization may be used for layers with highly variable activation ranges.

PyTorch's quantization API supports symmetric quantization through its QConfig settings.

• QConfig Selection: A symmetric configuration is defined using torch.ao.quantization.qconfig.QConfig with a symmetric activation observer (e.g., MinMaxObserver with qscheme=torch.per_tensor_symmetric).
• Observer for Calibration: Observers like MinMaxObserver or HistogramObserver collect tensor statistics to determine the scale factor.
• Fusion and Preparation: Common patterns like Conv+ReLU are fused before inserting fake quantization modules (prepare_qat).
• Integer Model Export: The convert function produces a model where weights and activations are represented as quantized integers, with floating-point scales stored for dequantization at specific points in the graph.

ONNX Runtime (ORT) uses symmetric quantization for cross-platform performance.

• Quantization Tool: The onnxruntime.quantization module provides a QuantFormat.QOperator path for symmetric quantization, generating an ONNX model with QuantizeLinear and DequantizeLinear nodes.
• Static Quantization: The quantize_static function requires a calibration dataset to determine the symmetric ranges for activations.
• Execution Provider Support: Quantized models run efficiently across providers like CPU, CUDA, and TensorRT via ORT, as the symmetric integer operations are a standard target for hardware accelerators.
• Model Portability: The quantized ONNX model serves as an interchange format between training frameworks and diverse inference runtimes.

Symmetric quantization is favored in hardware design due to its efficiency but has specific constraints.

• Computational Benefit: With z=0, the core operation r' = q * S is a simple integer multiplication, avoiding an extra addition. This simplifies hardware datapaths and reduces circuit complexity.
• Efficient Matrix Multiplication: For a layer operation Y = XW, symmetric quantization allows the integer computation Y_q = X_q * W_q, followed by a fused dequantization scaling, which is highly optimized.
• Range Limitation: It is less efficient for tensors where the data distribution is not symmetric around zero (e.g., ReLU outputs which are all non-negative). Here, asymmetric quantization can use the available integer range more precisely.
• Common Use Case: It is predominantly used for weight tensors, which often have a symmetric distribution, and for activations following layers like nn.Linear or nn.Conv2d without a subsequent ReLU.

Asymmetric Quantization is a scheme where the quantization range is not centered on zero. It uses a separate zero-point value to map the real numerical zero to an integer, allowing for a more precise representation of data distributions that are not symmetric around zero.

• Key Difference: Unlike symmetric quantization, the zero-point is not fixed at 0.
• Formula: quantized_value = round(real_value / scale) + zero_point
• Use Case: Often provides higher accuracy for activations following ReLU, which have a range of [0, +r], by minimizing clipping error.

The Quantization Scale is a multiplicative factor used to map floating-point values to a quantized integer range. It is defined as the ratio between the original value range and the quantized range.

• Calculation: scale = (float_max - float_min) / (quant_max - quant_min)
• Role: Determines the resolution of quantization. A larger scale compresses a wider range of values, increasing potential quantization error.
• In Symmetric Quantization: The scale is calculated based on the maximum absolute value: scale = max(|tensor|) / (2^(b-1) - 1) for a signed b-bit integer.

INT8 Inference is the execution of a neural network using 8-bit integer precision for weights and activations. It is the primary target for symmetric and asymmetric quantization techniques.

• Benefit: Reduces model memory footprint by 4x compared to FP32 and enables the use of high-throughput integer arithmetic units on modern GPUs and CPUs.
• Process: Requires quantization of weights/activations to INT8, integer matrix multiplication, and then dequantization of outputs.
• Performance: Can provide 2-4x latency and throughput improvements over FP16/FP32 inference on supported hardware.

Quantization-Aware Training is a model optimization technique that simulates the effects of lower numerical precision during the training process. This allows the model's weights and activations to adapt to the quantization error.

• Purpose: To recover accuracy lost during Post-Training Quantization (PTQ).
• Mechanism: Uses fake quantization nodes that apply rounding and clipping during the forward pass but maintain full precision gradients for the backward pass, often using a Straight-Through Estimator (STE).
• Outcome: Produces models that are more robust to quantization, often achieving near-FP32 accuracy with INT8 inference.

Post-Training Quantization is a compression technique that reduces the numerical precision of a pre-trained model's weights and activations without requiring retraining.

• Advantage: Fast and requires no labeled data or GPU resources for retraining.
• Process: Involves feeding a calibration dataset through the model to observe activation ranges and determine optimal scale and zero-point parameters.
• Types: Includes static quantization (fixed scales) and dynamic quantization (per-batch scale calculation). Symmetric quantization is commonly applied as a PTQ method.

These terms define the granularity at which quantization parameters are applied.

• Per-Tensor Quantization: A single set of scale and zero-point parameters is applied to all values within an entire tensor. This is simpler but can be less accurate if the tensor's distribution varies significantly.
• Per-Channel Quantization: A unique set of parameters is applied to each channel (e.g., each output channel of a convolutional filter). This accounts for per-channel variation and typically yields higher accuracy, especially for weight tensors.
• Symmetric Context: Symmetric quantization can be applied with either granularity. Per-channel symmetric quantization of weights is a standard practice for INT8 inference.