Post-Training Quantization

PTQ relies on a representative calibration dataset—typically 100-1000 unlabeled samples—to determine the optimal clipping range for each tensor. The process records the dynamic range of activations during a forward pass and selects quantization parameters (scale and zero-point) that minimize information loss:

• Min-max calibration: Uses the absolute min/max values observed, sensitive to outliers
• Percentile calibration: Discards extreme outliers (e.g., 99.99th percentile) for tighter ranges
• KL divergence calibration: Minimizes the information loss between the original FP32 distribution and the quantized INT8 representation
• Mean Squared Error calibration: Directly minimizes the MSE between original and quantized outputs The choice of calibration algorithm directly impacts the accuracy-quantization trade-off and must be validated against the target domain's data distribution.

PTQ supports two fundamental mapping strategies that determine how floating-point values are projected onto integer grids:

• Symmetric quantization: Maps the floating-point range symmetrically around zero, using a single scale factor. This simplifies hardware implementation because zero-point is always zero, making it the default for weight quantization in frameworks like TensorRT
• Asymmetric quantization: Uses both a scale factor and a non-zero zero-point to map a min/max range that is not centered on zero. This better captures activation distributions (e.g., ReLU outputs are always positive) but adds computational overhead during matrix multiplication
• Per-tensor quantization: Applies a single scale/zero-point to an entire tensor, maximizing throughput
• Per-channel quantization: Assigns unique quantization parameters to each channel dimension, preserving finer granularity at the cost of slightly more complex dequantization logic Most production deployments use symmetric per-tensor for weights and asymmetric per-tensor for activations.

Quantization introduces systematic bias in layer outputs because the rounding operation shifts the mean of the distribution. Bias correction is a lightweight post-hoc fix that absorbs this shift into the layer's bias term without retraining:

• After calibration, compute the expected quantization error for each channel: E[Wx] - E[Q(W)Q(x)]
• Absorb this error into the existing bias parameter: bias_corrected = bias + error
• This technique is particularly effective for depthwise-separable convolutions and transformer attention layers where small per-channel shifts compound across the network
• Advanced implementations also apply AdaRound (Adaptive Rounding), which learns whether to round each weight up or down during quantization rather than using nearest-neighbor rounding, recovering up to 1-2% absolute accuracy on challenging models like MobileNetV2 and EfficientNet.

Not all INT8 implementations are equal. PTQ must target the specific quantized operator set supported by the inference accelerator:

• NVIDIA TensorRT: Supports INT8 with per-channel weight quantization and per-tensor activation quantization, leveraging Tensor Cores for 2x throughput over FP16
• Intel OpenVINO: Uses symmetric INT8 for convolutions and fully-connected layers, optimized for DL Boost (VNNI) instructions on Xeon processors
• ARM CMSIS-NN: Targets Cortex-M microcontrollers with INT8 symmetric per-tensor quantization, requiring explicit requantization steps between layers
• Qualcomm Hexagon DSP: Supports asymmetric INT8 with per-channel weights, optimized for the HVX vector extensions
• Cross-platform ONNX Runtime: Provides a unified quantization API that maps to backend-specific implementations, enabling write-once-deploy-anywhere workflows Selecting the wrong quantization profile can result in a model that is mathematically correct but fails to accelerate on the target silicon.

The choice between PTQ and quantization-aware training (QAT) depends on the model's sensitivity to precision loss:

• PTQ is sufficient when: The model is over-parameterized (e.g., ResNet-50 on ImageNet), the target precision is INT8, and the calibration dataset is representative. PTQ achieves near-lossless compression for most convolutional architectures
• QAT is required when: The model is highly optimized and compact (e.g., MobileNetV3, EfficientNet-EdgeTPU), the target precision is INT4 or lower, or the task requires fine-grained predictions (e.g., small object detection, medical image segmentation)
• Hybrid approach: Apply PTQ to the backbone and QAT to the sensitive head layers, balancing development cost with accuracy
• Sensitivity analysis: Tools like NVIDIA's polygraphy or Qualcomm's AIMET can identify which layers are most vulnerable to quantization error, guiding where to apply QAT selectively For diagnostic imaging models deployed at the edge, PTQ with per-channel quantization and bias correction is the default starting point before escalating to QAT.

The calibration dataset must faithfully represent the inference-time data distribution, or the quantized model will suffer from silent accuracy degradation:

• Domain shift failure mode: A model calibrated on natural images and deployed on medical CT scans will have mismatched activation ranges, causing clipping artifacts that manifest as reduced sensitivity to subtle lesions
• Batch normalization fusion: During PTQ, batch norm parameters are folded into the preceding convolution weights. If calibration data statistics differ from training data, the fused weights will encode incorrect normalization, compounding quantization error
• Mitigation strategies: Use a calibration set drawn from the target deployment population, apply Hounsfield Unit normalization for CT data before calibration, and validate quantized model outputs against the FP32 baseline on a held-out test set
• Monitoring in production: Deploy runtime range observers that track activation statistics post-deployment, triggering recalibration if the distribution drifts beyond acceptable thresholds For regulated medical device software, the calibration dataset and its provenance must be documented as part of the FDA submission package.

A runtime strategy that executes most operations in lower precision (FP16 or INT8) while retaining FP32 for numerically sensitive layers. This balances the speed and memory benefits of quantization with the stability required for critical computations like softmax or normalization.

• Often used alongside PTQ for optimal speed-accuracy trade-off
• Supported natively by TensorRT and ONNX Runtime
• Critical for layers where small errors can propagate

A small, representative subset of unlabeled data used during PTQ to determine optimal scaling factors for activations. The quality of this dataset directly impacts post-quantization accuracy. For medical imaging, it must capture the full range of anatomical variation and scanner parameters.

• Typically 100-500 representative samples
• Must match the target deployment distribution
• Poor calibration leads to significant accuracy degradation

The total RAM and storage required to load and execute a model. PTQ directly reduces this by converting 32-bit floats to 8-bit integers, achieving a 4x reduction in model size. This is the primary constraint for deploying diagnostic AI on embedded medical devices with limited resources.

• FP32 model: 100 MB → INT8 model: 25 MB
• Enables deployment on microcontrollers and low-power SoCs
• Reduces flash storage and RAM requirements simultaneously

A key efficiency metric measuring the electrical energy consumed per forward pass, typically in millijoules. INT8 quantization dramatically reduces this by using simpler integer arithmetic. This directly extends battery life and reduces thermal output in fanless medical devices.

• Integer math consumes significantly less power than floating-point
• Critical for battery-powered portable diagnostic tools
• Enables sustained operation without active cooling