Lottery Ticket Hypothesis

A winning ticket is defined by a binary mask applied to a dense network's initialization. Only weights with the highest final magnitudes after initial training are retained, typically resulting in 80-99% sparsity. The mask is identified through iterative magnitude pruning (IMP) : train the network, prune the smallest-magnitude weights, and reset the surviving weights to their original initialization values. This sparse architecture is structurally identical to the original network but computationally far cheaper.

The specific random initialization is critical. A winning ticket's mask is coupled to its initial weights; applying the same mask to a different random initialization or shuffling the surviving weights destroys its efficacy. This phenomenon is explained by the Lottery Ticket Conjecture: SGD seeks out and trains a sparse subnetwork whose initialization makes the loss landscape particularly amenable to optimization. Random reinitialization breaks this fortuitous alignment.

IMP is the canonical algorithm for identifying winning tickets. The process:

• Phase 1: Train the dense network to completion.
• Phase 2: Prune a fraction (e.g., 20%) of weights with the smallest L1 norms.
• Phase 3: Reset remaining weights to their original initialization.
• Phase 4: Repeat until target sparsity is reached. This iterative retraining and pruning is essential; one-shot pruning at initialization fails to find performant subnetworks in deep architectures.

Winning tickets are not static throughout training. Research shows that the sparse connectivity pattern undergoes significant rewiring in early epochs before stabilizing. If pruning is applied too late in training, the mask becomes brittle and fails to generalize. The optimal pruning schedule targets the early phase of training when the network is still exploring the loss landscape, allowing the mask to capture a robust, generalizable subnetwork structure.

In very deep networks (e.g., ResNet-50, transformers), standard IMP struggles to find winning tickets at high sparsity without a learning rate warmup or late resetting modification. The instability of deep networks at initialization causes gradient explosion in sparse subnetworks. Techniques like gradient clipping and layer-wise pruning rates are required. However, once found, winning tickets in large models often train faster and generalize better than the original dense network.

Winning tickets discovered on one dataset or task often transfer effectively to related tasks when used as a fixed initialization. A ticket found on ImageNet can accelerate training on CIFAR-10 or fine-tuning on medical imaging. This suggests winning tickets capture universal, task-agnostic features in their connectivity patterns. The transferred mask provides a powerful inductive bias, reducing the sample complexity required to learn the new task.