Weight Pruning

The most fundamental pruning criterion removes weights with the smallest absolute values, operating on the assumption that near-zero parameters contribute minimally to network output. Global magnitude pruning ranks all weights across the entire network simultaneously, while layer-wise pruning applies independent thresholds per layer. After each pruning step, the model typically undergoes fine-tuning to recover accuracy before the next sparsity iteration. This iterative prune-retrain cycle can achieve 90%+ sparsity on overparameterized architectures like VGG-16 with negligible accuracy degradation.

Unstructured pruning removes individual scalar weights, producing irregular sparsity patterns that achieve high compression ratios but require specialized sparse linear algebra libraries for acceleration. Structured pruning removes entire channels, filters, or attention heads, producing dense sub-networks that run efficiently on commodity hardware without custom sparse kernels. Structured approaches directly reduce FLOPs and latency on standard GPUs and NPUs, while unstructured methods excel at maximizing parameter reduction for storage-constrained deployments. The choice depends on whether the target hardware supports N:M sparsity patterns.

Rather than removing weights in a single aggressive step, gradual pruning follows a sparsification schedule that progressively increases sparsity over training iterations. The polynomial decay schedule starts with zero sparsity and reaches the target sparsity at a specified step, allowing the network to continuously adapt its remaining connections. This approach outperforms one-shot pruning because surviving weights have time to compensate for removed connections. Automated gradual pruning dynamically adjusts the pruning rate based on validation loss, slowing removal when accuracy begins to degrade.

Modern pruning frameworks incorporate weight regrowth mechanisms that allow previously pruned connections to reactivate if they become important later in training. This dynamic sparse training paradigm, exemplified by RigL and SET algorithms, periodically drops low-magnitude weights while regrowing new connections at random locations or based on gradient magnitude. The result is a sparse network discovered during training rather than post-hoc, often matching or exceeding the accuracy of dense baselines while maintaining 80-95% sparsity throughout the entire training process.

Empirically demonstrated by Frankle and Carbin in 2019, this hypothesis reveals that dense, randomly-initialized networks contain winning tickets—sparse subnetworks that, when trained in isolation with their original initialization, achieve accuracy comparable to the full model. Finding these tickets requires iterative magnitude pruning with weight rewinding to early training states. The discovery has profound implications: it suggests overparameterization primarily serves to increase the probability that a successful sparse subnetwork exists at initialization, rather than being necessary for learning capacity itself.

Advanced pruning frameworks incorporate hardware latency and energy models directly into the pruning objective, optimizing for real inference speed rather than abstract sparsity percentages. Tools like TensorFlow Model Optimization Toolkit and NVIDIA ASP allow developers to specify target latency budgets, and the pruning algorithm automatically determines per-layer sparsity ratios that maximize accuracy within those constraints. This approach accounts for hardware-specific characteristics including memory bandwidth, cache hierarchy, and tensor core utilization, producing models that achieve measured speedups rather than theoretical FLOP reductions.

A three-stage framework that sequentially applies pruning, trained quantization, and Huffman coding to achieve state-of-the-art compression rates. The pipeline first removes redundant connections via pruning, then reduces the bit-width of remaining weights, and finally applies lossless data compression to the storage format. This combined approach often yields 30-50x model size reduction without significant accuracy degradation, making it foundational for deploying neural receivers on resource-constrained FPGAs and microcontrollers.

A fine-grained structured pruning pattern that enforces exactly N non-zero values in every contiguous block of M weights. The common 2:4 pattern delivers predictable 2x acceleration on NVIDIA Ampere GPU tensor cores with minimal accuracy impact. Unlike unstructured pruning, N:M sparsity maps directly to hardware vectorization, eliminating the irregular memory access patterns that typically limit real-world speedup from sparse models.

A family of mathematical techniques that approximate high-dimensional weight tensors with smaller constituent factors to reduce parameter count. Methods include:

• Low-rank factorization of fully-connected layers
• CP decomposition for convolutional kernels
• Tensor train format for extreme compression Unlike pruning, decomposition restructures the entire layer rather than zeroing individual weights, often producing more hardware-friendly dense matrices.

A training method that simulates low-precision inference during the forward pass, enabling the model to learn parameters robust to quantization error. QAT is the natural complement to pruning: pruning removes redundant connections, while QAT reduces the bit-width of those that remain. Together they form the core of the deep compression pipeline, with QAT typically applied after pruning to recover any accuracy lost during sparsification.

A compression method where a compact student model is trained to replicate the output distribution of a larger teacher model. Rather than pruning an existing network, distillation transfers the teacher's learned representations to an architecturally simpler student. For RF signal classification, a lightweight student can learn to mimic the soft labels and attention maps of a high-capacity transformer-based receiver, achieving comparable accuracy with 10x fewer parameters.