Model Quantization

A one-shot compression technique applied to a fully trained model without retraining. PTQ converts 32-bit floating-point weights to 8-bit integers using calibration data to determine optimal scaling factors. Key characteristics:

• Minimal data requirements (a few hundred representative samples)
• Fastest path to deployment
• Typically achieves 4x model size reduction
• May incur 1-3% accuracy degradation on complex modulation classification tasks
• Ideal for RadioML and GNU Radio pre-trained models

A training methodology that simulates quantization effects during the forward pass, allowing the model to learn parameters robust to reduced precision. How it works:

• Inserts fake quantization nodes into the computational graph
• Maintains full-precision weights during backpropagation
• Models learn to compensate for rounding errors
• Preserves accuracy within 0.5% of the full-precision baseline
• Essential for complex-valued neural networks processing I/Q data where phase information is sensitive to precision loss

A lightweight PTQ variant that quantizes only weights to 8-bit integers while activations remain in floating-point during inference. Trade-offs:

• 2-3x latency improvement vs. 4x for full integer quantization
• No calibration dataset required
• Weights are quantized offline; activations are quantized dynamically at runtime
• Suitable for blind modulation recognition models deployed on CPUs without dedicated neural processing units
• Lower memory bandwidth reduction compared to static quantization

Two granularity strategies for determining scaling factors. Per-tensor uses a single scale for an entire weight tensor, while per-channel assigns independent scales to each output channel. Comparison:

• Per-channel preserves accuracy better for models with wide weight distribution variance
• Per-tensor offers simpler hardware implementation and faster inference
• Convolutional AMC models benefit significantly from per-channel quantization due to filter diversity
• Per-channel is the default in TensorFlow Lite and PyTorch Mobile quantization pipelines

An advanced strategy that assigns different numerical precisions to different layers based on their sensitivity to quantization error. Implementation approach:

• First layer and last layer often kept at higher precision (16-bit or 32-bit)
• Intermediate feature extraction layers aggressively quantized to 4-bit or 8-bit
• Sensitivity analysis identifies layers critical to modulation confidence score accuracy
• Can achieve 6-8x compression while maintaining classification performance
• Requires automated search algorithms or manual layer profiling

A lightweight compression method that applies numerical precision reduction to a pre-trained model without retraining. PTQ uses a small calibration dataset to determine optimal clipping ranges and scaling factors for weights and activations. It is the most common entry point for deploying AMC models on edge hardware.

• Requires no access to the original training pipeline
• Calibration typically uses a few hundred unlabeled I/Q samples
• Can introduce accuracy degradation at very low bit-widths (e.g., 4-bit)

A technique that simulates the effects of low-precision arithmetic during the forward and backward passes of training. By modeling quantization noise as part of the optimization process, the network learns parameters that are inherently robust to precision loss. QAT consistently outperforms PTQ for aggressive compression targets.

• Inserts fake quantization nodes into the computational graph
• Recovers accuracy lost by PTQ at 4-bit and below
• Essential for deploying high-order QAM classifiers on NPUs

A scheme where weights are statically quantized to 8-bit integers, but activations are quantized dynamically at runtime based on their observed min/max range. This offers a balance between ease of implementation and performance, as it eliminates the need for activation calibration data.

• Weights are fixed at conversion time
• Activation ranges are computed per-inference call
• Slightly higher latency than static quantization due to runtime scaling

A complementary compression strategy where a compact student network is trained to replicate the output distribution of a large, high-precision teacher model. When combined with quantization, the student learns smoother decision boundaries that are less susceptible to quantization noise.

• Student mimics the teacher's soft labels (probability vectors)
• Often used alongside QAT for extreme compression
• Improves generalization for few-shot modulation classes

A structural or unstructured compression method that removes redundant or near-zero weights from a neural network. When combined with quantization, pruning creates sparse, low-precision models that exploit both sparsity acceleration and reduced memory bandwidth on specialized hardware.

• Unstructured pruning zeroes out individual weights
• Structured pruning removes entire channels or filters
• Pruning followed by QAT yields the highest compression ratios

A specialized quantization scheme designed for complex-valued neural networks that natively process I/Q data. Unlike real-valued quantization that treats I and Q components independently, complex-valued quantization preserves the phase relationship by quantizing magnitude and phase or using joint complex scaling factors.

• Critical for preserving phase information in AMC tasks
• Prevents constellation rotation artifacts at low bit-widths
• Requires custom quantization operators in the inference runtime