Post-Training Quantization (PTQ)

PTQ requires a calibration dataset—a small, representative sample of the model's inference data—to determine optimal quantization parameters. These parameters, the scale and zero-point, are calculated by observing the statistical range (min/max values) of the model's activations during a forward pass. This process tailors the quantization to the specific data distribution the model will encounter, unlike a theoretical range. Common calibration methods include Min-Max and Entropy-based approaches.

The scope of applied quantization parameters defines granularity, a key accuracy/efficiency trade-off.

• Per-Tensor Quantization: A single scale and zero-point is applied to an entire tensor. This is simple and highly efficient but can introduce significant error if the tensor's values have a wide dynamic range.
• Per-Channel Quantization: Unique parameters are calculated for each output channel of a weight tensor (common in convolutional and linear layers). This accounts for inter-channel variation, preserving accuracy at the cost of slightly more complex computation and metadata storage.

This defines how the quantized integer range maps to the original floating-point range.

• Symmetric Quantization: The range is centered on zero. The zero-point is fixed at 0, simplifying the arithmetic (no zero-point addition needed during integer matrix multiplication). It is efficient but can waste representation bins if the data distribution is not symmetric.
• Asymmetric Quantization: The range is not centered on zero. A non-zero zero-point allows the full integer range to precisely map an asymmetric data distribution (e.g., all-positive activations like ReLU outputs). This minimizes quantization error but adds computational overhead.

PTQ methods are categorized by when activation ranges are determined.

• Static Quantization: The most common PTQ method. Activation ranges are fixed during calibration using a representative dataset. All scaling factors are pre-computed and constant during inference, enabling aggressive kernel fusion and optimization. Used in frameworks like TensorRT and TFLite.
• Dynamic Quantization: Activation ranges are computed on-the-fly for each input during inference. This adapts to varying inputs but introduces runtime overhead. It is often applied to layers with highly variable activation ranges, like attention mechanisms in transformers.

PTQ is a lossy compression technique. The core trade-off is between model efficiency and task accuracy.

• Efficiency Gains: Reducing precision from FP32 to INT8 cuts the model's memory footprint by ~75% and can increase inference speed by 2-4x on hardware with optimized integer units.
• Accuracy Loss: The introduction of quantization error can degrade model performance. The magnitude of loss depends on model architecture, task complexity, and quantization strategy. Quantization Sensitivity Analysis is used to identify and protect vulnerable layers.

A key goal of PTQ is to enable integer-only inference, where all computations use low-precision integers. This allows the model to leverage specialized hardware instructions (e.g., INT8 Tensor Cores on NVIDIA GPUs, NEON on ARM CPUs, or NPU integer units) for peak throughput and power efficiency. The final quantized model bypasses floating-point arithmetic entirely after the initial dequantization of inputs.

A training-time optimization technique that simulates the effects of lower numerical precision during the forward and backward passes. Unlike Post-Training Quantization (PTQ), QAT allows the model's weights to adapt to the introduced quantization error, typically resulting in higher final accuracy at the cost of requiring a retraining cycle.

• Key Mechanism: Uses fake quantization nodes that apply rounding and scaling during training but maintain full-precision weights for gradient updates.
• Primary Use Case: Deploying models to ultra-low precision (e.g., INT4) where PTQ would cause unacceptable accuracy degradation.

A small, representative set of unlabeled data used during static post-training quantization to determine the optimal quantization scale and zero-point for activation tensors. The model performs a forward pass on this dataset to observe the dynamic range (min/max values) of its activations.

• Purpose: Enables the quantization algorithm to set parameters that minimize clipping and rounding error.
• Size: Typically only a few hundred samples are sufficient.
• Criticality: The representativeness of this dataset directly impacts the quantized model's accuracy; a poor calibration set leads to suboptimal quantization parameters.

The execution of a neural network using 8-bit integer precision for both weights and activations. This is the most common target precision for Post-Training Quantization due to widespread hardware support and a favorable accuracy-efficiency trade-off.

• Benefits: Reduces model memory footprint by 4x compared to FP32. Enables the use of high-throughput integer arithmetic units (e.g., Tensor Cores on NVIDIA GPUs, NPUs).
• Hardware Support: Natively accelerated by inference engines like TensorRT, OpenVINO, and TFLite.
• Typical Accuracy Loss: For many vision and language models, INT8 PTQ results in less than a 1% drop in task accuracy.

A granularity scheme where a unique set of quantization parameters (scale and zero-point) is calculated and applied to each output channel of a weight tensor (e.g., each filter in a convolutional layer). This contrasts with per-tensor quantization, which uses one set of parameters for the entire tensor.

• Advantage: Accounts for variation in weight distributions across channels, leading to lower quantization error and higher accuracy, especially for weights.
• Computational Overhead: Adds minimal overhead during inference as the per-channel scaling can be fused into the preceding bias addition.
• Common Practice: Standard for weight quantization in frameworks like PyTorch and TensorRT.

The process of converting computed integer values back into floating-point numbers in the higher-precision format expected by subsequent layers or for final output. It is the inverse operation of quantization and is defined by the formula: float_value = scale * (int_value - zero_point).

• Runtime Role: In many inference engines, dequantization is fused with a following floating-point operation or postponed until necessary to maintain performance.
• Mathematical Necessity: Required because operations like addition or non-linear activation functions (e.g., SiLU, GELU) are often performed in higher precision (FP16/FP32) even in quantized pipelines.

The numerical distortion introduced when converting a value from a higher-precision format (e.g., FP32) to a lower-precision format (e.g., INT8). It is the primary trade-off in any quantization process and is composed of:

• Rounding Error: Caused by mapping a continuous value to the nearest discrete quantization level.
• Clipping Error: Occurs when values outside the quantized range are clamped to the minimum or maximum representable value.

The goal of PTQ algorithms is to choose scale and zero-point parameters that minimize the overall error across all model tensors, often by analyzing the distribution of weights and activations.