Post-Training Quantization (PTQ)

PTQ relies on a representative calibration dataset—typically 100-1000 unlabeled samples—to collect activation statistics. These statistics determine optimal clipping ranges and scaling factors for each tensor.

• Min-Max Calibration: Sets the quantization range to the observed minimum and maximum activation values
• Percentile Calibration: Discards extreme outliers to reduce quantization error for the majority of values
• Mean Squared Error (MSE) Calibration: Minimizes the L2 distance between original and quantized tensor distributions

The quality of calibration data directly determines accuracy retention. Mismatched data distributions cause significant degradation.

PTQ supports two fundamental mapping schemes that trade off hardware efficiency against representational fidelity:

• Symmetric Quantization: Maps floating-point values to a signed integer range centered on zero (e.g., -128 to 127 for INT8). This eliminates the zero-point offset, simplifying hardware multiply-accumulate operations
• Asymmetric Quantization: Uses a non-zero zero-point to map the full floating-point range into the integer domain. This better preserves distributions with non-zero means, such as ReLU activations, at the cost of additional subtraction operations

Most edge NPUs and DSPs optimize for symmetric per-tensor quantization due to its computational simplicity.

The granularity of quantization parameters dramatically impacts accuracy retention:

• Per-Tensor Quantization: Applies a single scale and zero-point to an entire weight tensor. Fastest execution but struggles with wide weight distributions across output channels
• Per-Channel Quantization: Assigns independent quantization parameters to each output channel of a convolutional or fully-connected layer. This preserves accuracy significantly better for models with high variance in per-channel weight ranges

Per-channel quantization is the default recommendation for convolutional architectures. Per-tensor is preferred for fully-connected layers and transformer attention projections where channel variance is lower.

The end goal of PTQ is full-integer quantization, where all arithmetic—including activations, weights, biases, and intermediate tensors—occurs in the integer domain.

• Weight Quantization: Static; weights are quantized once offline after calibration
• Activation Quantization: Dynamic or static; quantization parameters are either computed at runtime or fixed during calibration
• Bias Quantization: Typically stored as INT32 to preserve accumulation precision before requantization to INT8 output

Full-integer execution eliminates floating-point units entirely, enabling deployment on microcontrollers, fixed-point DSPs, and integer-only NPU architectures like Qualcomm Hexagon and Google Edge TPU.

Modern PTQ toolchains implement automatic mixed-precision selection to balance accuracy and latency:

• Sensitivity Analysis: Measures each layer's contribution to total accuracy degradation when quantized. Layers with high sensitivity are kept at higher precision (INT16 or FP16)
• Layer-wise Fallback: Operators that exhibit unacceptable error—often the first and last layers, depthwise convolutions, or LayerNorm—are selectively excluded from quantization
• Quantization Simulation: The quantized graph is executed on calibration data to measure layer-wise and end-to-end accuracy drop before hardware deployment

This approach achieves near-floating-point accuracy while still realizing 2-4x throughput improvements on target accelerators.

PTQ pipelines apply graph-level optimizations that fuse multiple operators into single quantized kernels to minimize memory traffic and kernel launch overhead:

• Conv-BatchNorm-ReLU Fusion: Folds BatchNorm parameters into the preceding convolution weights and biases, then applies ReLU in-place during the convolution output write
• Add-ReLU Fusion: Combines residual addition and activation into a single fused kernel, avoiding an intermediate tensor allocation
• Requantization Insertion: Strategically places requantization nodes between integer operations to prevent accumulator overflow while maintaining precision

These fusions are critical for achieving the theoretical TOPS/Watt of edge accelerators by reducing DRAM bandwidth pressure.

The systematic removal of redundant or low-magnitude connections in a neural network. Pruning creates sparse weight matrices that reduce model size and computational complexity.

• Unstructured pruning removes individual weights, producing irregular sparsity
• Structured pruning removes entire channels or filters for direct hardware acceleration
• N:M sparsity enforces exactly N non-zero values per block of M weights for predictable GPU tensor core speedup
• Often combined with PTQ in the Deep Compression Pipeline

A compression method where a compact student model is trained to replicate the output distribution of a larger, high-capacity teacher model. The student learns from both hard labels and the teacher's softened probability outputs, transferring dark knowledge.

• Does not require specialized hardware support
• Student architecture can differ completely from teacher
• Effective for RF tasks where teacher models capture subtle signal features
• Often combined with PTQ for multiplicative compression gains

A critical pre-processing step before PTQ that mathematically absorbs batch normalization parameters into the preceding convolutional or fully-connected layer's weights and biases. This eliminates redundant runtime operations and prevents quantization error amplification.

• Reduces the number of operations per inference
• Prevents mismatch between training-time BN statistics and inference-time fixed-point arithmetic
• Standard step in TensorFlow Lite and ONNX Runtime quantization toolchains

The most widely deployed PTQ precision target, mapping 32-bit floating-point weights and activations to 8-bit integers. INT8 models achieve approximately 4x memory reduction and 2-4x inference speedup on hardware with vectorized integer instructions.

• Supported natively on ARM Cortex-M CMSIS-NN kernels
• Calibration requires representative data samples to determine optimal scale factors and zero points
• Per-channel quantization provides finer granularity than per-tensor for weight distributions

A specialized PTQ variant that performs precision reduction without access to the original training dataset. DFQ relies on synthetic data generation from batch normalization statistics or generative models to calibrate quantization parameters.

• Critical for defense and proprietary RF applications where training data cannot be shared
• Typically achieves lower accuracy than data-dependent PTQ
• Useful when deploying third-party models on classified signal processing hardware