Model Compression

The systematic removal of redundant or low-magnitude parameters from a neural network to create a sparse architecture. Unstructured pruning zeroes out individual weights, leading to irregular sparsity that requires specialized hardware or software for acceleration. Structured pruning removes entire neurons, channels, or filters, producing a physically smaller model that executes efficiently on standard hardware. Iterative magnitude-based pruning, combined with fine-tuning cycles, can reduce parameter counts by 50-90% with negligible accuracy degradation. The lottery ticket hypothesis suggests that dense networks contain sparse subnetworks capable of matching the original model's performance when trained in isolation.

A compression technique that reduces the numerical precision of a model's weights and activations from 32-bit floating-point to lower bit-width representations such as 8-bit integers. INT8 quantization is the most common scheme, mapping floating-point ranges to integer values using scale and zero-point parameters. This conversion reduces memory bandwidth requirements by up to 4x and enables the use of fast integer arithmetic units on CPUs and edge accelerators. Dynamic quantization computes the scaling factors at runtime for activations, while static quantization pre-computes them using a calibration dataset. Quantization-aware training simulates low-precision effects during the training process to recover accuracy lost during direct post-training conversion.

A training paradigm where a compact student model learns to replicate the behavior of a larger, high-capacity teacher model. The student is trained on the softened probability distributions produced by the teacher's final layer, which contain rich inter-class similarity information absent from hard labels. A temperature parameter controls the softness of these distributions, revealing dark knowledge about the teacher's internal representations. Distillation can be combined with pruning and quantization for compound compression effects. This technique is particularly effective for compressing transformer-based architectures used in signal identification, where the student can learn to attend to the same discriminative RF features as the teacher.

A compression approach that decomposes large weight matrices into the product of two or more smaller matrices using techniques like Singular Value Decomposition or tensor decomposition. For a fully connected layer, a weight matrix of dimensions m×n can be approximated by two matrices of sizes m×r and r×n, where r is a small rank. This reduces the parameter count from m×n to r×(m+n). In convolutional layers, spatial filters can be decomposed into separable depthwise and pointwise operations, forming the basis of architectures like MobileNet. Low-rank methods are particularly effective for compressing the large linear layers in transformer attention mechanisms used for sequence-based signal analysis.

An automated methodology that uses reinforcement learning, evolutionary algorithms, or gradient-based optimization to discover compact model architectures optimized for specific hardware constraints. The search space includes kernel sizes, channel counts, layer depths, and connectivity patterns. Hardware-aware NAS incorporates latency and energy consumption feedback from the target deployment platform directly into the search objective. This approach has produced highly efficient architectures like EfficientNet and MnasNet. For RF fingerprinting applications, NAS can discover specialized architectures that balance the need for high-frequency feature extraction with the strict latency requirements of real-time spectrum monitoring on FPGA or SDR platforms.

A compression technique that reduces the effective number of unique parameters by forcing multiple weights to share the same value. K-means clustering is applied to the weight matrices, and each weight is replaced by the centroid of its assigned cluster. Only the cluster indices and a small codebook of centroid values need to be stored, dramatically reducing the model's storage footprint. Huffman coding can further compress the index matrix by assigning shorter codes to frequently occurring indices. This method is particularly synergistic with quantization, as the clustered centroids can themselves be stored at reduced precision. Weight sharing is a foundational technique in extreme compression scenarios targeting microcontroller-class devices.

The systematic removal of redundant or low-magnitude connections from a neural network. By zeroing out parameters that contribute minimally to the output, the model becomes sparse and computationally cheaper.

• Unstructured Pruning: Removes individual weights, leading to irregular sparsity that requires specialized hardware or software to accelerate.
• Structured Pruning: Removes entire neurons, channels, or filters, resulting in a smaller, dense model that runs efficiently on standard hardware.
• Magnitude-based Pruning: A simple heuristic that removes weights with the smallest absolute values, often followed by fine-tuning to recover accuracy.
• Iterative Pruning: Repeats the prune-train cycle, gradually increasing sparsity while allowing the network to adapt.

A compression technique that reduces the numerical precision of a model's weights and activations after training is complete. Converting from 32-bit floating-point (FP32) to 8-bit integers (INT8) drastically reduces memory footprint and accelerates inference.

• Dynamic Quantization: Quantizes weights ahead of time but dynamically computes activation ranges during inference, offering a good balance of ease and performance.
• Static Quantization: Calibrates both weight and activation ranges using a representative dataset, enabling fully integer-arithmetic execution for maximum speed.
• Quantization-Aware Training (QAT): Simulates quantization noise during training, allowing the model to learn parameters robust to lower precision, yielding higher final accuracy than post-training methods.

A training paradigm where a compact student model is trained to mimic the behavior of a larger, high-performance teacher model. Instead of learning directly from hard labels, the student learns from the teacher's softened output probabilities, capturing inter-class relationships.

• Response-based Distillation: The student minimizes the divergence between its output logits and the teacher's logits, often using a temperature-scaled softmax.
• Feature-based Distillation: The student is trained to replicate the intermediate feature representations of the teacher, transferring richer structural knowledge.
• Relation-based Distillation: Focuses on preserving the mutual relationships between data samples as learned by the teacher, capturing higher-order structural information.

A technique that decomposes large weight matrices into the product of two or more smaller matrices. This exploits the inherent low-rank structure often present in over-parameterized layers, significantly reducing the number of parameters and multiply-accumulate operations.

• Singular Value Decomposition (SVD): Factorizes a weight matrix W into USV^T, allowing the smallest singular values to be discarded for compression.
• Tensor Decomposition: Extends matrix factorization to higher-dimensional tensors, useful for compressing convolutional kernels in CNNs.
• Depthwise Separable Convolutions: A structural factorization that splits a standard convolution into a depthwise convolution followed by a pointwise convolution, forming the basis of efficient architectures like MobileNet.

An automated methodology for discovering optimal, compact network architectures tailored to specific hardware constraints. NAS algorithms explore a defined search space of operations and connections to find models that maximize accuracy under strict latency and memory budgets.

• Hardware-Aware NAS: Incorporates direct feedback from the target deployment platform into the search loop, optimizing for real-world inference speed rather than theoretical FLOPs.
• Differentiable NAS (DARTS): Relaxes the discrete search space into a continuous one, enabling gradient-based optimization for dramatically faster architecture discovery.
• Once-for-All Networks: A method that trains a single large network containing many sub-networks, allowing specialized, compressed models to be extracted for different hardware targets without retraining.

A compression method that groups the scalar weight values of a model into a discrete number of clusters, storing only the cluster centroids and a small index per weight. This reduces the effective number of bits required to store the model.

• K-Means Clustering: The most common algorithm, which partitions weights into k clusters by minimizing the within-cluster sum of squares.
• Shared Weights: All weights assigned to the same cluster share a single centroid value, dramatically reducing the model's memory footprint.
• Huffman Coding: Often applied after clustering to further compress the index matrix by assigning variable-length codes based on cluster frequency.