Post-Training Quantization (PTQ)

The ultimate output of PTQ is a model where all arithmetic—convolutions, matrix multiplications, and element-wise operations—executes using integer-only operations. This eliminates the need for floating-point DSP slices on FPGAs, allowing the entire computation to map efficiently to DSP48 blocks configured as INT8 multiply-accumulate units. Combined with operator fusion and data packing, integer-only inference maximizes throughput and minimizes power consumption, enabling real-time modulation classification on devices like the Xilinx RFSoC.

Unlike PTQ, QAT simulates quantization effects during training by inserting fake quantization nodes into the computational graph. The forward pass uses quantized weights and activations, while the backward pass uses the Straight-Through Estimator (STE) to propagate gradients through the non-differentiable rounding operation. This allows the network to adapt its parameters to minimize quantization error, typically yielding 1-3% higher accuracy than PTQ for aggressive 4-bit or lower precision targets. QAT is preferred when PTQ accuracy drop exceeds deployment requirements.

A pre-quantization optimization that addresses per-channel weight imbalance across consecutive layers. The technique exploits the scale-equivariance property of ReLU activations to redistribute weight magnitudes between layers without changing the network's mathematical output:

• Equalizes dynamic ranges across channels
• Reduces quantization error from per-tensor quantization
• Particularly effective for MobileNet and EfficientNet architectures
• Often applied as a preprocessing step before PTQ to recover accuracy lost to aggressive bit-width reduction

A mandatory pre-quantization step that mathematically absorbs batch normalization parameters into the preceding convolutional or fully-connected layer's weights and biases. During inference, batch normalization is a fixed affine transformation:

• y = γ * (x - μ) / σ + β
• Folding computes new weights: W_folded = W * γ/σ
• New bias: b_folded = β - γ*μ/σ This eliminates redundant runtime computation and reduces memory access, making it essential before applying PTQ to any network with batch normalization layers.

A complementary compression technique that removes redundant connections rather than reducing their precision. Pruning creates sparse weight matrices where many elements are exactly zero:

• Unstructured pruning: Removes individual weights, achieving high compression but requiring sparse hardware support
• Structured pruning: Removes entire channels or filters, producing dense sub-networks compatible with standard hardware
• Combined with PTQ, pruning can achieve 10-50x total model compression
• Magnitude-based pruning removes weights with the smallest absolute values, assuming they contribute least to the output

A training-based compression approach where a compact student network learns to mimic the behavior of a larger teacher network. The student is trained on the teacher's softened output probabilities rather than hard labels:

• The temperature parameter controls softness of the teacher's output distribution
• Student learns inter-class relationships captured by the teacher
• Can be combined with PTQ: distill first, then quantize the student
• Often produces smaller models with higher accuracy than training the compact architecture from scratch on ground-truth labels alone

The deployment target that PTQ enables. In this mode, all arithmetic operations use integer math, eliminating floating-point units entirely:

• Weights stored as INT8 or INT4
• Activations computed with integer multiply-accumulate
• Bias and scale factors applied as integer shifts and additions
• Enables execution on fixed-point DSP slices in FPGA fabric
• Reduces power consumption by 3-5x compared to FP32 inference
• Critical for battery-powered edge devices and high-throughput streaming pipelines