INT8 Quantization

INT8 quantization maps 32-bit floating-point (FP32) values to 8-bit integers. This yields a 4x reduction in model size and a corresponding 4x reduction in memory bandwidth requirements. For example, a 1GB FP32 model becomes approximately 250MB in INT8. This is critical for deploying large models on memory-constrained devices like mobile phones or edge accelerators. The process involves determining a scale factor and, optionally, a zero-point to map the floating-point range to the integer range [-128, 127] or [0, 255].

The core transformation is defined by: Q = round(R / S) + Z, where R is the real (FP32) value, S is the scale factor, Z is the zero-point, and Q is the quantized INT8 value. Dequantization reconstructs an approximate float: R' = S * (Q - Z).

• Symmetric Quantization: Sets Z = 0, simplifying computation. The range is symmetric around zero (e.g., [-127, 127]).
• Asymmetric Quantization: Uses a non-zero Z to better fit asymmetric data distributions (e.g., ReLU activations that are all non-negative), often yielding higher accuracy.

INT8 operations are natively supported by modern AI accelerators like NVIDIA Tensor Cores (Ampere+), Intel AMX, and ARM DOT instructions. These units perform integer matrix multiplications (INT8 GEMM) with significantly higher throughput and lower power consumption than equivalent FP32 operations. This hardware support is the primary driver for latency reduction, often achieving 2-4x speedup for compute-bound layers. The benefit is most pronounced in linear and convolutional layers where weights and activations are both quantized.

Calibration determines the optimal scale (S) and zero-point (Z) for each tensor.

• Static Quantization: Uses a representative calibration dataset to profile activation ranges offline. Parameters are fixed post-calibration, resulting in zero runtime overhead. Used by TensorRT, TFLite.
• Dynamic Quantization: Calculates quantization parameters for activations on-the-fly per inference. This adapts to varying inputs but adds computational overhead. Often applied to layers with highly variable activation ranges (e.g., attention layers in transformers).

This defines the granularity at which quantization parameters are applied.

• Per-Tensor Quantization: A single scale and zero-point is used for an entire tensor. This is simpler but can be suboptimal if the tensor's distribution varies significantly across channels.
• Per-Channel Quantization: Applied primarily to weight tensors in convolutional and linear layers. Each output channel gets its own scale and zero-point. This finer granularity accounts for varying weight distributions across channels, typically preserving more accuracy with minimal overhead. It is the standard for weight quantization in frameworks like PyTorch and TensorRT.

Quantization introduces quantization error from rounding and clipping. To recover accuracy:

• Quantization-Aware Training (QAT): The model is trained or fine-tuned with fake quantization nodes simulating INT8 rounding during forward passes. The optimizer learns to compensate for the error, yielding the highest accuracy.
• Post-Training Quantization (PTQ): Uses calibration and advanced algorithms like percentile calibration or entropy minimization to find optimal ranges without retraining. Faster but may have higher accuracy drop.
• Mixed-Precision Layers: Critical layers (e.g., final classifier) may be kept in higher precision (FP16) to preserve accuracy, creating a hybrid INT8/FP16 model.

Starting with the Ampere architecture (e.g., A100, A10, A2), NVIDIA's Tensor Cores added dedicated INT8 compute capability. They can perform matrix multiply-accumulate operations on INT8 data, delivering up to 4x the peak throughput compared to FP16 on the same hardware. This is a key driver for INT8 adoption in data centers. The Hopper architecture (H100) further enhances this with the Transformer Engine which dynamically manages FP8 and INT8 precision.

Modern CPUs include instruction set extensions for accelerating INT8 inference:

• Intel DL Boost (AVX-512 VNNI): Vector Neural Network Instructions on Xeon Scalable and Core processors combine multiply and add on INT8 vectors in one instruction, reducing latency and power.
• ARMv8.2-A Dot Product: The SDOT and UDOT instructions provide similar acceleration for INT8 operations on ARM Cortex-A CPUs, powering most mobile and edge devices. These instructions are leveraged by frameworks like ONNX Runtime and PyTorch (FBGEMM).

Quantization is the foundational model compression technique that reduces the numerical precision of a neural network's weights and activations. This process decreases model size, memory bandwidth requirements, and computational cost. Key forms include:

• Post-Training Quantization (PTQ): Converts a pre-trained model using a calibration dataset, no retraining needed.
• Quantization-Aware Training (QAT): Trains the model with simulated quantization to learn compensation for precision loss. INT8 is a specific, aggressive form of quantization to 8-bit integers.

BFloat16 (BF16) is a 16-bit floating-point format designed for deep learning. Unlike FP16, it preserves the 8-bit exponent range of a standard FP32 float, minimizing the risk of overflow/underflow during training. This makes it ideal for:

• Mixed Precision Training: Often used alongside FP32 master weights.
• Modern Hardware: Efficiently supported by TPUs and recent NVIDIA/AMD GPUs. While INT8 offers greater compression, BF16 provides a better balance of speed and accuracy for many layers, especially on hardware without dedicated INT8 tensor cores.

Calibration is the critical data-driven step in static quantization that determines how to map floating-point values to integers. A representative calibration dataset is passed through the model to observe the statistical range (min/max) of activation tensors. This process calculates:

• Scale Factor: The ratio between the floating-point and integer ranges.
• Zero-Point: In asymmetric quantization, this aligns the integer range with the tensor's distribution. Poor calibration (e.g., using an unrepresentative dataset) is a primary source of quantization error and accuracy degradation.

Per-Channel Quantization is a granular approach where separate quantization parameters (scale/zero-point) are calculated for each output channel of a weight tensor (e.g., in a convolutional layer). This contrasts with per-tensor quantization, which uses one set of parameters for the entire tensor.

• Advantage: Provides finer control, often leading to significantly higher accuracy for INT8 weight quantization because it accounts for varying ranges across channels.
• Hardware Support: Requires support from the inference engine (e.g., TensorRT, ONNX Runtime) and underlying kernels.

Dequantization is the inverse operation that converts low-precision integer values back into higher-precision floating-point numbers. In an INT8 quantized model, it is not a single end-step but is strategically applied within the computational graph. For example:

• After loading INT8 weights, they may be dequantized to FP16/BF16 for computation if the hardware lacks fast INT8 arithmetic.
• For operations that are sensitive to precision (e.g., certain accumulations), activations may be dequantized, computed in higher precision, and then re-quantized. This process manages the latency-accuracy trade-off within a mixed precision graph.