Lottery Ticket Hypothesis

A winning ticket is a subnetwork identified by a specific pruning and resetting process. After training a dense network, a fraction of weights with the smallest magnitudes are pruned. Crucially, the surviving weights are then reset to their original random initialization and retrained in isolation. This subnetwork, often 50-90% sparse, can achieve accuracy comparable to the full model, proving the original initialization contained a highly effective, compact architecture.

The standard method for finding winning tickets is Iterative Magnitude Pruning (IMP). This process involves a cycle of:

• Training the dense network to completion.
• Pruning a percentage (e.g., 20%) of the remaining weights with the smallest absolute magnitudes.
• Resetting the surviving weights to their initial values from epoch 0. This cycle repeats until the desired sparsity is reached. The final, untrained subnetwork is the winning ticket, which is then trained one final time.

The hypothesis hinges on the original random initialization. A winning ticket only works when its weights are rewound to their specific starting values. Randomly reinitializing the same sparse architecture leads to drastically worse performance. This suggests that the initial weights encode a favorable inductive bias for the task, and the pruning mask simply isolates the most effective combination of these pre-existing, randomly-assigned connections.

For very deep networks or high learning rates, winning tickets found by resetting to epoch 0 can fail. Late rewinding is a robust variant where weights are reset to an earlier point in training (e.g., epoch 5) rather than initialization. This preserves some learned structure, making the subnetwork trainable in isolation. This phenomenon is linked to loss landscape stability, where the initial optimization trajectory must find a region where a sparse mask can effectively define a viable subnetwork.

For resource-constrained edge devices running neural receivers, the Lottery Ticket Hypothesis offers a powerful one-shot pruning strategy. Instead of designing a compact model from scratch, an engineer can:

• Train a large, high-capacity RF model (e.g., for channel estimation).
• Apply IMP to discover a sparse winning ticket with drastically fewer MAC operations.
• Deploy this compact, high-accuracy subnetwork directly to an FPGA or NPU, bypassing the need for a separate, hand-crafted efficient architecture like MobileNet.

The structure of the sparse mask is critical. A randomly generated sparse mask applied to the same initialized weights will not train to high accuracy. The IMP process discovers a specific, non-random connectivity pattern that is uniquely suited to the task and the underlying weight initialization. This demonstrates that the 'winning' property is a combination of both the mask structure and the initial weight values, not just sparsity alone.