Lottery Ticket Hypothesis

A winning ticket is defined by a binary mask applied to a dense network's initialization. The mask identifies which connections to retain and which to prune. The critical insight is that the specific initialization values of the surviving weights are essential; re-initializing them randomly destroys the ticket's efficacy. The mask is typically discovered through iterative magnitude pruning (IMP) : train the network, prune a percentage of the smallest-magnitude weights, and reset the remaining weights to their original initial values. This process is repeated until the desired sparsity is reached.

The hypothesis reveals a profound dependency on the random seed used at initialization. A subnetwork that trains successfully from one random initialization will fail if its weights are re-initialized, even if the mask (structure) is preserved. This implies that winning tickets are not just architectures, but specific combinations of structure and initial weight values. This sensitivity has driven research into why certain initializations produce viable tickets and others do not, linking the phenomenon to the loss landscape and optimization dynamics.

Remarkably, sufficiently large networks contain subnetworks that achieve significantly better than chance accuracy without any training at all. These are called 'supermasks.' The existence of supermasks demonstrates that a randomly initialized network can encode a functional solution purely through the selection of a subset of its weights. This finding suggests that gradient descent may primarily serve to identify and amplify these pre-existing, fortuitous weight configurations rather than constructing a solution from scratch.

The probability of finding a winning ticket increases with the size of the original dense network. In smaller networks, iterative magnitude pruning often fails to find a trainable subnetwork, but as the parent network's width increases, winning tickets become more abundant. This has led to the Lottery Ticket Conjecture: any sufficiently over-parameterized network contains a subnetwork that can achieve the full model's performance. This provides a theoretical basis for why over-parameterization aids training and justifies large-scale pruning.

Winning tickets discovered on one dataset or task often serve as powerful initializations for other, related tasks. A subnetwork pruned for modulation classification on a high-SNR dataset, for example, can be transferred and fine-tuned for a low-SNR variant, converging faster and to higher accuracy than a randomly initialized sparse network. This cross-task generalization indicates that winning tickets capture fundamental, reusable features, making them valuable for few-shot learning and domain adaptation in RF signal processing.

The original Lottery Ticket Hypothesis required resetting weights to their initial values (iteration 0). Subsequent research showed that for large-scale models, this is unstable. A practical modification, late rewinding, resets weights to their values from an early point in training (e.g., epoch 5) rather than initialization. This stabilizes the ticket-finding process in deep networks like ResNets and transformers, making the hypothesis applicable to the large architectures used for complex IQ sample classification.

The process of removing redundant or low-magnitude connections from a neural network to reduce its memory footprint and computational complexity. Unstructured pruning zeroes out individual weights, leading to sparse matrices, while structured pruning removes entire channels or filters for direct hardware acceleration.

• Magnitude-based pruning removes weights with the smallest absolute values
• Iterative pruning alternates between pruning and fine-tuning to recover accuracy
• Directly validates the Lottery Ticket Hypothesis by isolating winning tickets

A training paradigm that dynamically learns and updates the sparse connectivity pattern of a neural network from scratch, rather than pruning a dense model post-hoc. This natively compressed model eliminates the need for a separate pruning step.

• Dynamic sparse training evolves the sparsity mask during training using prune-and-regrow cycles
• RigL (Rigged Lottery) algorithms periodically reactivate dormant weights based on gradient magnitude
• Reduces the computational cost of finding winning tickets compared to iterative magnitude pruning

The process of minimizing the total number of floating-point operations required for a single forward pass. This serves as the primary metric for evaluating the computational efficiency of a compressed modulation classifier.

• Directly proportional to inference latency and energy consumption on FPGA fabric
• Structured pruning achieves FLOPs reduction by removing entire convolutional filters
• Unstructured sparsity requires specialized sparse hardware or software libraries to translate into actual speedups

An automated methodology for discovering optimal neural network topologies that are inherently efficient for specific hardware targets. NAS can be guided by the Lottery Ticket Hypothesis to search for architectures that naturally contain high-quality sparse subnetworks.

• Hardware-aware NAS incorporates latency and energy feedback directly into the search objective
• One-shot NAS trains a single supernet from which winning tickets can be extracted
• Eliminates manual trial-and-error in designing compact modulation classifiers for FPGA deployment

A compression technique that decomposes large weight matrices into the product of smaller matrices using methods like Singular Value Decomposition (SVD). This reduces the computational load of matrix multiplications by approximating the original dense layer.

• Complements pruning by targeting different structural redundancies in the network
• Particularly effective for fully-connected layers common in classifier heads
• Provides a mathematically grounded alternative to the empirical search for winning tickets

A model compression technique where a smaller, efficient student network is trained to replicate the output distribution of a larger, high-performance teacher network. The student learns from both hard labels and the teacher's softened probability outputs.

• Can be combined with pruning to transfer knowledge into a sparse student architecture
• The teacher's soft labels provide richer supervisory signals than one-hot ground truth
• Enables deployment of compact modulation classifiers that approach the accuracy of ensemble models