INT8 Quantization

INT8 quantization typically uses an affine (linear) mapping defined by a scale factor (S) and a zero-point (Z). The formula q = round(r / S) + Z converts a real value r to an integer q. The zero-point allows the integer range to align with the real value distribution, which is crucial for accurately representing asymmetric data like ReLU activations. This scheme enables lossy but controlled compression.

A key design choice is the quantization range.

• Symmetric Quantization: Sets the zero-point to 0 and uses a range symmetric around zero (e.g., [-127, 127]). This simplifies arithmetic by eliminating zero-point addition in matrix multiplication but can be inefficient if the data distribution is not symmetric.
• Asymmetric Quantization: Allows the zero-point to shift, fitting the range [min, max] of the actual data. This better utilizes the 8-bit dynamic range for skewed distributions (common in activations) but adds computational overhead for the zero-point correction.

Quantization parameters can be shared at different granularities, affecting accuracy and hardware support.

• Per-Tensor: A single scale and zero-point for an entire tensor. This is simple and widely supported but can lead to higher error if the tensor's values have wide variance.
• Per-Channel: Applies unique scale/zero-point to each output channel of a weight tensor (common for convolutional and linear layers). This finer granularity preserves accuracy much better, as it accounts for varying weight distributions across channels. It is the default for weight quantization in frameworks like TensorRT and TFLite.

A core goal of INT8 quantization is to enable integer-only inference. After quantization, the core operation of a layer (e.g., matrix multiplication, convolution) can be performed using 8-bit integer inputs and 32-bit integer accumulators. The output is then re-quantized to 8-bit. This eliminates floating-point units from the compute path, enabling deployment on low-power edge devices, NPUs, and DSPs that have dedicated integer arithmetic logic units (ALUs), yielding significant latency and power benefits.

For static quantization, optimal scale and zero-point values must be determined before deployment via a calibration process. A small, representative calibration dataset is passed through the model in FP32 to collect the statistical distribution (min/max, histogram) of activation tensors. Algorithms like Entropy Minimization or Mean Squared Error Minimization then analyze these histograms to select quantization parameters that minimize the information loss or distortion, balancing clipping and rounding error.

Pure post-training quantization (PTQ) to INT8 often causes accuracy drops. Two primary techniques mitigate this:

• Quantization-Aware Training (QAT): During fine-tuning, fake quantization nodes simulate INT8 rounding and clipping in the forward pass, while gradients flow in FP32. This allows the model to adapt its weights to the quantization error.
• Advanced PTQ Algorithms: Methods like Layer-Wise Equalization and Bias Correction adjust weights and activations post-training to reduce quantization-induced bias, often recovering most accuracy loss without retraining.

Modern NVIDIA GPUs (Ampere, Hopper architectures) feature Tensor Cores that perform mixed-precision matrix operations. For INT8, these cores execute integer matrix multiply-accumulate (IMMA) operations, providing a 4x theoretical throughput increase over FP16 on the same hardware. This makes data center GPUs like the A100 and H100, as well as edge GPUs like the Jetson Orin, highly effective platforms for INT8 inference, especially when paired with TensorRT.

Neural Processing Units (NPUs) and AI accelerators (e.g., Google TPU, Apple Neural Engine, Qualcomm Hexagon) are designed with native integer arithmetic logic units (ALUs). These specialized cores execute 8-bit integer operations with extreme power efficiency, making them ideal for mobile and edge devices. Their instruction sets and memory hierarchies are optimized for the predictable dataflow of quantized neural networks, enabling integer-only inference without any floating-point units.

Quantization is the foundational process of mapping a continuous set of high-precision values (like FP32) to a discrete, finite set of lower-bit integer representations. Its primary goal is to reduce the computational and memory footprint of neural networks. Key aspects include:

• Affine Mapping: Typically uses a linear transformation defined by a scale and zero-point.
• Trade-offs: Inevitably introduces quantization error, a trade-off between model size/speed and accuracy.
• Workflow Stages: Encompasses both post-training quantization (PTQ) and quantization-aware training (QAT).

Post-Training Quantization (PTQ) is a model compression technique that converts a pre-trained model's weights and activations to lower-bit integers without retraining. It is fast and requires no labeled data for fine-tuning. The process involves:

• Calibration: Running a small, representative calibration dataset through the model to observe activation ranges.
• Parameter Calculation: Determining optimal quantization scale and zero-point values.
• Conversion: Transforming the model graph for integer execution. PTQ is ideal for rapid deployment but may incur higher accuracy loss than QAT.

Quantization-Aware Training (QAT) is a process that simulates quantization effects during the training or fine-tuning phase. This allows the model to learn to compensate for the precision loss, typically yielding higher accuracy than PTQ. Core mechanisms include:

• Fake Quantization: Inserting operations that mimic rounding and clipping in the forward pass, while maintaining high-precision gradients for the backward pass.
• Model Adaptation: The network's weights adjust to the quantized representation.
• Use Case: Essential for models where PTQ causes unacceptable accuracy degradation, providing a more robust quantized model.

Mixed-precision training is a technique that uses multiple numerical precisions (e.g., FP16 for tensors and FP32 for master weights) during model training to accelerate computation and reduce memory usage. It is distinct from, but complementary to, integer quantization. Key components are:

• Automatic Mixed Precision (AMP): Libraries like NVIDIA's Apex or PyTorch AMP that automate precision casting.
• Loss Scaling: A critical technique to prevent numerical underflow of small gradient values in FP16.
• Hardware Acceleration: Leverages tensor cores in modern GPUs for faster FP16 matrix operations, reducing training time.

FP16 and BF16 are 16-bit floating-point formats central to mixed-precision workflows, often used as an intermediate step before INT8 quantization.

• FP16 (Half-Precision): Defined by IEEE 754, uses a 5-bit exponent and 10-bit mantissa. Halves memory bandwidth but has a limited dynamic range, risking overflow/underflow.
• BF16 (Brain Floating-Point): Uses an 8-bit exponent (like FP32) and a 7-bit mantissa. It better preserves the dynamic range of FP32, making it more robust for training, especially in deep networks.
• Role in Quantization: Models are often trained or fine-tuned in FP16/BF16 before being quantized to INT8 for ultimate inference efficiency.

Integer-only inference is the ultimate goal of INT8 quantization: executing an entire neural network using integer arithmetic, eliminating floating-point operations. This is critical for deployment on low-power NPUs and edge devices. It requires:

• Full Quantization: Both weights and activations must be quantized to integers (e.g., INT8).
• Quantized Kernels: Hardware or software backends (like TensorRT Lite or TFLite) that provide optimized integer operations for layers like convolution and matrix multiplication.
• Fused Operations: Combining operations like convolution, bias add, and activation (e.g., ReLU) into a single integer kernel to minimize data movement and latency.