Quantization

The core mechanism involves mapping continuous 32-bit floating-point (FP32) values to discrete 8-bit integer (INT8) or lower representations. This is achieved through a linear mapping defined by a scale factor and a zero point. The scale factor determines the step size between quantized levels, while the zero point ensures that the value zero in the original domain is represented exactly in the quantized domain. This mapping is critical for maintaining numerical stability, especially around zero, which is a common value for padded or inactive neurons.

PTQ is a one-shot calibration process applied to a fully trained FP32 model. It requires a small, representative calibration dataset to estimate the dynamic range of activations. The process involves:

• Collecting activation statistics by running inference on the calibration data.
• Computing optimal scale factors and zero points per tensor or per channel.
• Converting weights and activations to INT8. PTQ is the most common approach because it is fast, requires no retraining, and typically results in minimal accuracy loss for convolutional and transformer architectures.

QAT simulates the effects of quantization during the forward pass of training. Fake quantization nodes are inserted into the computational graph, which round values to lower precision but maintain gradients in full precision for the backward pass. This allows the model to learn parameters that are robust to quantization error. QAT consistently achieves higher accuracy than PTQ, especially for aggressive targets like INT4 or binary quantization, but at the cost of full retraining cycles.

The granularity of quantization determines how scale factors are shared. Per-tensor quantization uses a single scale and zero point for an entire weight tensor, which is simple but can cause significant error if the dynamic range varies across channels. Per-channel quantization assigns a unique scale factor to each output channel of a weight tensor, preserving more fine-grained information. Per-channel is standard for weight quantization in modern inference engines like TensorRT, while activations are typically quantized per-tensor due to hardware efficiency constraints.

The latency gains from quantization are realized through specialized INT8 matrix multiplication kernels on modern hardware. NVIDIA GPUs with Tensor Cores, Intel CPUs with VNNI instructions, and ARM processors with NEON all provide dedicated instructions for 8-bit integer operations. These kernels can execute multiple INT8 operations in a single clock cycle, delivering up to 4x higher throughput compared to FP32. The memory bandwidth savings are equally significant, as INT8 weights occupy 4x less space, reducing pressure on memory-bound layers.

Dynamic quantization computes the activation scale factor on-the-fly for each inference request, quantizing activations just before computation. This eliminates the need for a calibration dataset and is common in PyTorch for LSTM and transformer models. Static quantization pre-computes activation scales offline using calibration data, avoiding runtime overhead. Static quantization is preferred for production serving because it enables full graph optimization and fusion of quantization operations with adjacent layers, yielding the lowest possible latency.

A percentile metric indicating that 99% of inference requests are served faster than this threshold. While average latency may appear healthy, P99 exposes tail latency outliers that degrade user experience in real-time personalization.

• A 100ms average with a 2-second P99 means 1 in 100 users waits 2+ seconds
• Tail latencies are often caused by garbage collection pauses or resource contention
• Quantization reduces P99 by shrinking memory footprint and eliminating cache misses