Post-Training Quantization

PTQ requires no access to the original training pipeline or loss function. Instead, it uses a small, unlabeled calibration dataset—typically a few hundred representative samples—to collect statistical information about the range of activations flowing through each layer. This data is used to compute optimal scaling factors and zero-points that map floating-point tensors to integer grids with minimal information loss.

The standard target for PTQ is 8-bit integer (INT8) representation, reducing model size by 4× compared to FP32. Key mechanisms include:

• Weight quantization: Static, layer-wise conversion of kernel values
• Activation quantization: Dynamic or static quantization of feature maps
• Per-channel vs. per-tensor: Fine-grained scaling along output channels preserves accuracy better than global tensor scaling Modern toolchains also support INT4 and FP8 for aggressive compression.

Advanced PTQ pipelines go beyond simple min-max range estimation. Techniques include:

• Histogram entropy calibration: Minimizing Kullback-Leibler divergence between original and quantized distributions
• Percentile clipping: Trimming outlier values to tighten the representable range
• Bias correction: Compensating for systematic errors introduced by weight quantization in each layer's output distribution These methods recover significant accuracy lost by naive quantization.

PTQ is tightly coupled to target inference hardware. Different accelerators impose distinct constraints:

• NPUs and DSPs: Often require symmetric quantization with per-axis scaling
• Desktop GPUs: Support mixed-precision via Tensor Cores with INT8 and FP16 modes
• Edge TPUs: Mandate fully integer-only execution, including quantization of non-linear operations like activations A PTQ pipeline must respect the operator set and quantization scheme of the target runtime.

When direct PTQ causes unacceptable accuracy degradation, lightweight recovery techniques are applied without full retraining:

• AdaRound: Learns optimal rounding decisions (up or down) for each weight value rather than simple nearest rounding
• AdaQuant: Performs per-layer optimization of quantized weights to minimize output error
• Cross-layer equalization: Exploits scale-equivariance in activation functions like ReLU to redistribute dynamic range across consecutive layers These methods often close the gap to floating-point accuracy within 1%.

PTQ is supported across major inference optimization frameworks:

• TensorRT: NVIDIA's closed-source optimizer with INT8 calibration and kernel auto-tuning
• ONNX Runtime: Cross-platform execution with quantization tools for CPU, GPU, and edge targets
• PyTorch Quantization API: Native support for static, dynamic, and quantization-aware training modes
• OpenVINO: Intel's toolkit with post-training optimization for CPU and VPU inference Each toolchain provides calibration utilities and hardware-specific kernel libraries.

Unlike PTQ, Quantization-Aware Training simulates the effects of lower precision during the forward pass of training. This allows the model to learn parameters that are inherently robust to quantization error. While QAT typically yields higher accuracy than PTQ, especially for aggressive compression targets like INT4, it requires access to the full training pipeline and labeled data, making it computationally more expensive and less flexible than post-training methods.

Weight pruning is a complementary compression technique that removes individual weights or entire structures (neurons/channels) from a network based on their magnitude. The core idea is to eliminate parameters with near-zero values that contribute negligibly to the output. Unstructured pruning leads to sparse matrices requiring specialized hardware for acceleration, while structured pruning produces dense, smaller models directly compatible with standard inference engines.

Knowledge distillation trains a compact 'student' model to mimic the output distribution of a larger, more accurate 'teacher' model. Instead of compressing an existing model directly, it transfers the generalization capability. The student is trained on the soft logits of the teacher, capturing inter-class relationships. This is often combined with PTQ to create extremely compact, high-performance models for edge deployment.

INT8 calibration is the specific process of determining the optimal scaling factors and zero-points for mapping 32-bit floating-point tensors to 8-bit integers. This involves running a representative dataset through the model to collect the dynamic range of activations. Common calibration algorithms include:

• MinMax: Uses the absolute min/max values.
• Entropy: Minimizes information loss measured by KL divergence.
• Percentile: Clips extreme outliers to reduce quantization error for the majority of values.

The granularity of quantization parameters significantly impacts accuracy. Per-tensor quantization uses a single scale and zero-point for an entire weight tensor, which is fast but can be inaccurate if the distribution of weights varies significantly across channels. Per-channel quantization assigns distinct scaling factors to each output channel, preserving finer weight distributions and often recovering substantial accuracy lost by per-tensor methods, at a slight computational overhead.