Model Quantization

Simulates quantization effects during the forward pass of training, allowing the model to learn parameters robust to reduced precision. The backward pass still uses full-precision gradients.

• Fake quantization nodes inserted into the computational graph
• Model learns to compensate for rounding errors and clipping
• Recovers accuracy that PTQ might lose, especially for aggressive bit-widths
• Essential for INT4 or lower precision targets

QAT is the standard approach when deploying to Edge TPU or bare-metal inference engines where every bit counts.

Quantizes only the weights to INT8 while activations remain in floating-point during inference. The activations are quantized dynamically based on their observed range at runtime.

• Simplest quantization method — no calibration dataset needed
• Weights are statically quantized post-training
• Activations quantized on-the-fly with minimal overhead
• Achieves 2-4x speedup on CPUs with SIMD instruction support

Commonly used in TensorFlow Lite for rapid prototyping on mobile and embedded CPUs.

Two granularity strategies for assigning scale and zero-point parameters. Per-tensor uses a single scale for an entire weight tensor, while per-channel assigns independent scales to each output channel.

• Per-tensor: Simpler, faster, but sensitive to outlier channels
• Per-channel: Higher accuracy, especially for depthwise convolutions
• Per-channel adds minimal compute overhead on modern hardware
• Critical for preserving accuracy in CNN-based modulation classifiers

Per-channel quantization is the default in TensorRT and recommended for vision and signal processing models.

Assigns different numerical precisions to different layers based on their sensitivity to quantization error. Sensitive layers retain FP16 or INT16, while robust layers drop to INT8 or INT4.

• Layer-wise sensitivity analysis guides precision assignment
• First and last layers often kept at higher precision
• Balances model size, latency, and accuracy trade-offs
• Automated tools like NVIDIA's TAO Toolkit perform this analysis

Essential for complex architectures where uniform quantization causes unacceptable degradation in critical feature extraction layers.

The end-goal of quantization: executing the entire forward pass using only integer operations. This eliminates floating-point units entirely, enabling deployment on bare-metal microcontrollers and FPGAs.

• All operations — convolutions, activations, batch norm — are integer-based
• Requires careful calibration of zero-point and scale for each tensor
• Enables execution on hardware without FPUs, like ARM Cortex-M
• Used in TensorFlow Lite Micro for true edge deployment

For real-time spectrum classification, integer-only inference on an FPGA offload engine achieves deterministic, low-latency classification within the inference latency budget.

A compression technique that removes redundant or low-magnitude connections from a neural network. Unstructured pruning zeros out individual weights, creating sparse matrices, while structured pruning removes entire neurons, channels, or filters.

• Magnitude-based pruning eliminates weights below a threshold
• Iterative pruning with fine-tuning recovers accuracy between sparsification rounds
• Can achieve 50-90% model size reduction with minimal accuracy loss
• Often combined with quantization for compound compression effects

A training paradigm where a smaller student model learns to mimic the behavior of a larger, high-performing teacher model. The student is trained on the teacher's softened output probabilities (logits) rather than hard labels.

• Captures inter-class relationships that one-hot labels miss
• Temperature scaling controls softness of teacher probability distribution
• Enables deployment of compact models with performance approaching the teacher
• Particularly effective for modulation classifiers where the teacher was trained on abundant synthetic data

An automated methodology for discovering optimal network architectures under specific hardware constraints. Hardware-aware NAS incorporates latency, memory, and power budgets directly into the search objective.

• Search spaces include convolutional kernel sizes, channel counts, and layer depth
• Reinforcement learning or evolutionary algorithms guide architecture exploration
• Produces bespoke models that are inherently efficient for target silicon
• Eliminates manual trial-and-error in designing compact modulation classifiers

A graph-level optimization that combines multiple consecutive operations into a single kernel launch. This eliminates intermediate memory reads and writes between layers.

• Fuses convolution + batch normalization + activation into one operation
• Reduces memory bandwidth bottlenecks on edge accelerators
• Implemented automatically by inference optimizers like TensorRT and ONNX Runtime
• Critical for achieving real-time inference on FPGA and NPU targets

The specific quantization workflow that converts a pre-trained floating-point model to integer precision without retraining. A small calibration dataset is used to determine optimal scaling factors and zero points for each tensor.

• Requires only a few hundred representative samples for calibration
• Min-max calibration sets range based on observed tensor extremes
• KL divergence calibration minimizes information loss between distributions
• Significantly faster than quantization-aware training with acceptable accuracy trade-offs

An approach that assigns different numerical precisions to different layers of a network based on their sensitivity to quantization error. Sensitive layers (first and last) often retain higher precision while robust intermediate layers use aggressive INT8 or INT4 formats.

• Balances accuracy preservation with maximum throughput
• Hardware must support mixed-precision execution natively
• Sensitivity analysis identifies layers that disproportionately impact output
• Common in modulation classifiers where feature extraction layers tolerate lower precision