Sparse Training

Unlike static pruning, sparse training allows the network's connectivity graph to evolve during training. Weights that are pruned early can regrow later if the optimizer identifies a need for new connections.

• RigL (Rigged Lottery): Periodically prunes low-magnitude weights and grows new ones at random locations with high gradient momentum.
• SET (Sparse Evolutionary Training): Replaces pruned weights with random new connections, mimicking evolutionary selection.
• This dynamic process explores a wider combinatorial space of subnetworks, often finding superior sparse topologies compared to one-shot pruning.

Sparse training directly reduces the multiply-accumulate (MAC) operations required for both forward and backward passes. By maintaining a fixed sparsity ratio (e.g., 90%) from initialization, the computational cost is proportional to the number of non-zero weights.

• A 90% sparse network requires only ~10% of the dense network's FLOPs per iteration.
• This enables training of large models on resource-constrained hardware without a dense pre-training phase.
• The savings compound: sparse forward pass, sparse backward pass, and sparse weight updates.

Sparse training maintains the model in a compressed sparse row (CSR) or coordinate list (COO) format throughout the entire training lifecycle. Only non-zero weights and their indices are stored.

• Storage: A 90% sparse matrix stores only 10% of the weight values plus integer indices.
• Optimizer States: Momentum and variance buffers in Adam are also stored sparsely, dramatically reducing total memory consumption.
• This allows training of models that would otherwise exceed GPU memory limits, critical for high-dimensional RF signal processing.

Sparse training provides a practical realization of the Lottery Ticket Hypothesis, which posits that dense networks contain sparse, trainable subnetworks that can match the full model's accuracy.

• Instead of searching for winning tickets post-hoc via iterative pruning, sparse training finds and trains the winning ticket simultaneously.
• The learned sparse mask is the winning ticket, discovered without ever training the dense network.
• This validates the hypothesis that over-parameterization is not strictly necessary for effective learning, only for finding the right connectivity.

Modern sparse training frameworks enforce structured sparsity patterns (e.g., 2:4 sparsity) that map efficiently to hardware accelerators like NVIDIA's Tensor Cores or FPGA systolic arrays.

• 2:4 Structured Sparsity: Exactly 2 non-zero values in every contiguous block of 4 weights, enabling direct hardware acceleration.
• Block Sparsity: Pruning entire blocks of weights to maintain regular memory access patterns.
• This co-design between the training algorithm and the target hardware ensures that theoretical FLOPs reductions translate to real wall-clock speedups on the deployment platform.

Sparse training uses the Straight-Through Estimator (STE) to propagate gradients through the discrete binary mask that enforces sparsity.

• The forward pass uses the masked weights: w_eff = w * mask.
• The backward pass ignores the mask's non-differentiability and passes gradients directly to the underlying dense weights.
• This allows the optimizer to update both the active weights and the dormant weights (which may regrow), maintaining a full gradient signal for potential future connections.

The conjecture that dense, randomly-initialized networks contain sparse subnetworks ('winning tickets') that can be trained in isolation to match the original network's accuracy. This provides the theoretical basis for why sparse training works: algorithms like iterative magnitude pruning identify these winning tickets early in training, allowing the network to learn a sparse connectivity pattern from scratch without sacrificing representational capacity for modulation classification tasks.

Unlike static pruning which applies a fixed mask post-training, dynamic sparsity allows the connectivity pattern to evolve during training. Techniques like Sparse Evolutionary Training (SET) and RigL periodically drop low-magnitude weights and regrow new connections based on gradient information. This exploration-exploitation cycle discovers optimal sparse topologies that are often superior to those found by pruning dense networks, yielding higher accuracy for IQ sample classifiers at extreme sparsity levels.

The process of removing redundant or low-magnitude connections from a neural network to reduce its memory footprint and computational complexity. Sparse training is distinct from post-training pruning because it learns the sparse structure during the training process itself. Key pruning paradigms include:

• Unstructured pruning: Removes individual weights, yielding irregular sparsity
• Structured pruning: Removes entire channels or filters, enabling direct hardware acceleration
• Magnitude-based pruning: The simplest heuristic, removing weights with the smallest absolute values

The process of minimizing the total number of floating-point operations required for a single forward pass. Sparse training directly targets FLOPs reduction by learning a network where a large fraction of weights are zero, effectively skipping those multiply-accumulate operations. For modulation classifiers deployed on FPGAs, FLOPs reduction translates directly to lower power consumption and higher throughput. A sparse model with 90% weight sparsity can achieve up to 10x theoretical speedup with appropriate hardware support.

A neural architecture search methodology that incorporates hardware feedback—such as latency, energy consumption, or memory constraints from a lookup table—directly into the search objective. When combined with sparse training, hardware-aware NAS can discover architectures that are inherently amenable to sparsity on specific FPGA targets. This co-design approach finds Pareto-optimal models that balance modulation classification accuracy against the real-world inference cost on resource-constrained edge devices.

A critical distinction in sparse training outcomes:

• Unstructured sparsity: Zero weights are distributed randomly throughout the tensor. Achieves higher compression ratios but requires specialized sparse matrix multiplication hardware to realize speedups.
• Structured sparsity: Entire blocks, channels, or filters are zeroed. The N:M sparsity pattern (e.g., 2:4) enforced by NVIDIA's Ampere architecture requires exactly 2 non-zero values per block of 4, enabling direct hardware acceleration on tensor cores. Sparse training algorithms can be constrained to learn these hardware-friendly patterns.