INT8 Inference

Converting model weights and activations from 32-bit floating-point (FP32) to 8-bit integers (INT8) reduces the memory required to store the model by approximately 75%. This compression enables:

• Deployment of larger models within the same GPU memory budget.
• Batch size increases for higher throughput.
• Feasibility for on-device and edge inference where memory is severely constrained (e.g., mobile phones, microcontrollers).

Integer arithmetic operations (INT8) are fundamentally faster than floating-point operations (FP32) on modern hardware. AI accelerators like NVIDIA Tensor Cores, Google TPUs, and Intel DL Boost include dedicated silicon for low-precision math, offering:

• 2x to 4x higher peak theoretical compute for matrix multiplications (GEMM).
• Reduced data movement bandwidth requirements between memory and compute units.
• The ability to serve more queries per second (QPS) on the same hardware, directly lowering inference cost per token.

Reduced precision computation consumes significantly less energy per operation. This is critical for:

• Battery-powered edge devices where power efficiency dictates product viability.
• Data center sustainability goals, reducing the total cost of ownership (TCO).
• Achieving lower latency for real-time applications (e.g., autonomous driving, live translation) because smaller, faster operations complete more quickly, reducing end-to-end response time.

Early quantization often caused significant accuracy drops. Modern methods like Quantization-Aware Training (QAT) and advanced Post-Training Quantization (PTQ) with fine-grained per-channel quantization and careful calibration can maintain accuracy within <1% of the FP32 baseline for many models. This makes INT8 a practical default for production without sacrificing model quality.

INT8 quantization is not an endpoint but a foundation that unlocks further performance optimizations:

• Operator/Kernel Fusion: Compilers like TVM and XLA can fuse quantized linear, convolution, and activation layers into single, highly optimized kernels, eliminating intermediate memory writes.
• Model Serving Density: The reduced memory footprint allows more model instances or larger batches to be packed onto a single server, improving GPU utilization and overall throughput for serving platforms like Triton Inference Server.

Vendor libraries that provide the lowest-level kernels for INT8 math on specialized silicon.

• cuDNN & cuBLAS (NVIDIA): Provide optimized INT8 convolution and matrix multiplication routines used by frameworks like TensorRT.
• oneDNN (Intel): Features highly optimized INT8 primitives for CPUs, supporting advanced instructions like AVX-512 VNNI and AMX.
• ARM Compute Library: Delivers optimized INT8 functions for ARM Cortex-A CPUs and Mali GPUs, crucial for Android deployment.
• Qualcomm AI Engine Direct: Allows direct execution of quantized models on Hexagon DSPs and Adreno GPUs in Snapdragon platforms.

These libraries are typically invoked indirectly through higher-level frameworks.

A training-time optimization where the model learns to compensate for quantization error. Fake quantization nodes simulate INT8 precision during forward passes, while the Straight-Through Estimator (STE) allows gradients to flow backward. This results in higher accuracy compared to post-training methods but requires retraining.

• Key Benefit: Higher final accuracy for low-bit quantization.
• Use Case: Critical applications where post-training quantization causes unacceptable accuracy drops.

The process of converting a pre-trained FP32 model to a lower precision (like INT8) without retraining. It uses a small calibration dataset to determine optimal quantization scale and zero-point values. Methods include:

• Static Quantization: Scales are fixed after calibration.
• Dynamic Quantization: Activation scales are computed at runtime.

Primary Advantage: Fast, no training overhead. Trade-off: Potential for greater accuracy loss versus QAT.

Defines the scope for which a single set of quantization parameters (scale, zero-point) is applied. This choice impacts accuracy and hardware efficiency.

• Per-Tensor Quantization: One scale/zero-point per entire tensor. Simple but less accurate.
• Per-Channel Quantization: Unique parameters for each channel in a weight tensor. Accounts for variance across channels, preserving accuracy, especially for convolutional and linear layers. Common for weight tensors in INT8 inference.

The mathematical method for mapping floating-point values to integers.

• Symmetric Quantization: The quantized range is symmetric around zero (zero-point = 0). Simplifies computation but can waste precision if the data range is not symmetric.
• Asymmetric Quantization: The quantized range is mapped to the min/max of the data, using a non-zero zero-point. More accurately represents real data distributions (e.g., ReLU activations that are all non-negative).

Hardware Note: Some NPUs are optimized for symmetric quantization.

An advanced optimization where different layers or operations within a model use different numerical precisions. While INT8 is the target for most compute-heavy layers, sensitive layers may remain in FP16 or BF16 to preserve accuracy.

• Driven by: Quantization sensitivity analysis, which identifies layers where precision reduction causes significant accuracy loss.
• Goal: Achieve near-FP32 accuracy with inference latency closer to full INT8.
• Implementation: Requires sophisticated compilers (e.g., TensorRT) to manage different data types and kernel calls.

The process of converting computed integer values back to floating-point for downstream operations or final output. It is the inverse of quantization: float_value = scale * (int_value - zero_point).

• In INT8 Pipelines: Often occurs at the boundaries between INT8 and higher-precision layers, or before final softmax/activation functions.
• Performance Consideration: While a necessary step, modern inference engines fuse dequantization with subsequent floating-point operations to minimize overhead.