Model Quantization

Quantization maps continuous high-precision values to a discrete set of lower-precision levels. The fundamental operation is affine mapping, where a floating-point tensor r is converted to an integer tensor q using a scale factor and zero-point offset: r = S(q − Z). The scale S is a floating-point value representing the step size between quantized levels, while the zero-point Z is the integer value corresponding to real zero. This mapping must be carefully calibrated to preserve the dynamic range of each tensor.

• Symmetric quantization: Zero-point is fixed at 0, simplifying computation but wasting one quantization bin for signed values
• Asymmetric quantization: Zero-point is calibrated to minimize clipping error, better utilizing the full integer range
• Per-tensor quantization: A single scale and zero-point for an entire weight tensor
• Per-channel quantization: Separate scale factors for each output channel, critical for preserving accuracy in convolutional layers with high variance across filters

PTQ applies quantization to a fully trained neural network without any retraining or fine-tuning. A small calibration dataset—typically a few hundred representative input samples—is passed through the model to collect activation statistics. These statistics determine optimal scale factors and zero-points for each layer by observing the min/max ranges or using more sophisticated methods like KL divergence minimization or mean squared error optimization.

• Calibration: The critical step where activation ranges are measured; insufficient or unrepresentative calibration data leads to severe accuracy degradation
• Layer fusion: Batch normalization, convolution, and activation layers are merged into a single quantized operation before calibration to reduce quantization error propagation
• Advantage: Zero training cost and minimal data requirements, making PTQ the fastest path to deployment
• Limitation: Accuracy drops become significant below 8-bit precision, especially for compact or heavily regularized models

QAT inserts fake quantization nodes into the neural network graph during training. These nodes simulate the quantization and dequantization operations in the forward pass—rounding values to integer precision and then converting back to floating-point—while the backward pass uses the straight-through estimator (STE) to approximate gradients through the non-differentiable rounding function. This allows the model to learn parameters that are inherently robust to quantization error.

• Fake quantization: Simulates INT8 arithmetic using FP32 operations during training, exposing the optimizer to quantization effects
• Straight-through estimator: Approximates the gradient of the round() function as 1, enabling backpropagation through the quantization barrier
• Batch normalization folding: BN parameters are absorbed into preceding convolution weights before QAT begins to prevent training-inference mismatch
• Advantage: Recovers accuracy to within 0.5-1% of the FP32 baseline, even for aggressive 4-bit or mixed-precision schemes

The timing of scale factor computation distinguishes two quantization regimes. Dynamic quantization computes activation ranges on-the-fly for each input during inference, quantizing weights statically but activations dynamically. This eliminates calibration but adds per-inference overhead. Static quantization pre-computes both weight and activation scales offline using a calibration dataset, enabling fully integer-arithmetic inference with zero runtime overhead.

• Dynamic quantization: Best suited for models dominated by matrix multiplications with small activation memory footprints, such as LSTM and Transformer layers
• Static quantization: Required for convolutional neural networks where activation memory dominates and per-layer calibration is feasible
• Integer-arithmetic-only inference: Static quantization enables execution on integer-only hardware accelerators, eliminating floating-point units entirely
• Hybrid approaches: Some frameworks apply static quantization to convolutions and dynamic quantization to recurrent layers within the same model

Mixed-precision quantization assigns different bit-widths to different layers based on their sensitivity to quantization error. Layers with high parameter redundancy or smooth activation distributions can tolerate aggressive 4-bit or even 2-bit quantization, while sensitive layers—such as the first convolutional layer processing raw input or the final classification layer—retain higher precision like INT8 or FP16. This is often automated through neural architecture search (NAS) or Hessian-based sensitivity analysis.

• Sensitivity analysis: Measures the impact of quantizing each layer on final loss using the Hessian matrix's top eigenvalues or by direct perturbation experiments
• Hardware-aware search: Optimizes bit-width assignments subject to latency, energy, and memory constraints of the target FPGA or ASIC
• Pareto-optimal frontier: Mixed-precision produces a family of models trading off accuracy against resource consumption
• Granularity levels: Bit-widths can be assigned per-layer, per-channel, or even per-filter group for maximum flexibility

FPGA deployment imposes unique constraints beyond simple bit-width reduction. Quantization schemes must align with the DSP slice architecture, which typically implements 18×18 or 27×18 multiply-accumulate operations. Power-of-two quantization—where scale factors are restricted to powers of two—eliminates the need for hardware multipliers during rescaling, replacing them with bit-shift operations. Additionally, channel-wise quantization must be balanced against the FPGA's routing resources, as per-channel scales require additional multipliers in the datapath.

• DSP packing: Multiple low-precision multiplications (e.g., two INT8 operations) can be packed into a single DSP48 slice, doubling throughput
• Power-of-two scales: Convert multiplication by scale factors into simple barrel shifter operations, saving LUTs and DSPs
• Activation bit-width alignment: Matching activation precision to on-chip memory word widths minimizes wasted BRAM capacity
• Streaming architectures: Quantized models enable line-buffer-free streaming inference where activations flow directly between layers without external DRAM access

The strategy for setting initial neural network parameters before quantization-aware training begins. Proper initialization is critical because quantization amplifies poor initial conditions—weights that start in saturated regions produce zero gradients after discretization.

• Xavier/Glorot initialization scales weights by the square root of fan-in, suitable for tanh activations
• He initialization uses a factor of 2 for ReLU networks, preventing variance collapse
• For quantized networks, initial weight distributions should be tightly centered to minimize clipping error
• Some practitioners initialize from a pre-trained FP32 model's weights to preserve learned structure before fine-tuning with fake quantization nodes

The systematic optimization of architectural and training parameters that directly influence a quantized predistorter's linearization performance. Quantization introduces additional hyperparameters beyond standard neural network design.

• Bit-width selection: Trading off between INT4, INT8, and mixed-precision for different layers
• Quantization scheme: Choosing between symmetric vs. asymmetric, per-tensor vs. per-channel quantization
• Calibration dataset size: Determining how many representative samples are needed to compute optimal scaling factors
• Learning rate scheduling: Quantized networks often require lower learning rates and longer training to converge within discrete weight spaces
• Automated methods like Bayesian optimization or grid search explore this expanded parameter space

A layer that normalizes activations to zero mean and unit variance, inserted between linear operations and activation functions. During quantization, batch normalization presents a fusion opportunity that reduces inference overhead.

• The learned scale (γ) and shift (β) parameters can be folded into the preceding convolutional or fully-connected layer's weights
• This folding eliminates a separate normalization step during inference, reducing latency
• Post-folding, the combined weights must be re-quantized to the target integer representation
• For predistorter networks, batch normalization stabilizes training but must be carefully handled during the quantization-aware training to deployment pipeline to avoid accuracy degradation

An adaptive training paradigm where quantized neural network predistorter coefficients are continuously updated during live transmission. Quantization complicates online learning because gradient updates must respect discrete weight constraints.

• Straight-through estimator (STE) approximates gradients through quantization functions during backpropagation
• Online adaptation tracks time-varying PA characteristics from temperature drift and component aging
• Quantized online learning often uses stochastic rounding to allow small weight updates to accumulate
• FPGA implementations may maintain a high-precision shadow copy of weights for accumulation while using quantized values for forward inference
• This approach enables lifelong adaptation without requiring full retraining cycles