Rewinding

The defining mechanism of rewinding is the selective reset of model parameters. After a pruning step, weights are not reinitialized to random values. Instead, they are restored to the values from an earlier training checkpoint, typically from an epoch early in the initial dense training phase (e.g., epoch 1-5). This checkpoint represents a point where the network has learned useful, low-level features but before it has over-specialized. This provides a superior starting point for retraining the sparse subnetwork compared to random initialization.

Rewinding is the practical algorithm that empirically validated the Lottery Ticket Hypothesis. The hypothesis posits the existence of sparse, trainable subnetworks within a dense network. The rewinding procedure (prune, then rewind weights to the early checkpoint, then retrain) is the method used to identify these 'winning ticket' subnetworks. Without rewinding, simply pruning and retraining from scratch often fails to match the original accuracy, underscoring rewinding's critical role in uncovering trainable sparse architectures.

Rewinding is applied within a multi-cycle iterative pruning schedule, not as a one-shot operation. A standard cycle involves:

• Train the dense network for a few epochs to reach the rewind checkpoint.
• Prune a target percentage (e.g., 20%) of the smallest-magnitude weights.
• Rewind the remaining unpruned weights back to their values at the checkpoint.
• Retrain the now-sparse network to recover accuracy. This cycle repeats, incrementally increasing sparsity, until the target compression level is met. Rewinding is applied at the start of each retraining phase.

Pruning alters the model's loss landscape. Retraining the pruned network from its final, pre-pruned state can be suboptimal because the optimizer's state (e.g., momentum buffers in SGD) is tuned for the dense architecture. Rewinding addresses this by:

• Resetting the optimizer state along with the weights.
• Providing a smoother optimization path from a known-good point in weight space.
• Allowing the sparse network to re-learn appropriate feature representations from a stable foundation, which is more effective than fine-tuning the already-specialized final weights.

Rewinding is distinct from standard fine-tuning after pruning.

• Fine-Tuning: Continues training the pruned network from its final, pre-pruned weights and optimizer state. Often used in post-training pruning where a pre-trained model is pruned once and then lightly tuned.
• Rewinding: Resets to an earlier checkpoint before retraining. It is integral to iterative pruning during training and is crucial for finding high-performance sparse networks at extreme compression levels (>90% sparsity). Fine-tuning alone often fails to recover accuracy in these aggressive regimes.

A key hyperparameter in rewinding is the rewind epoch—the specific training checkpoint to which weights are reset. Its selection involves a trade-off:

• Early Epochs (e.g., epoch 1-3): Provide a more generic, foundational starting point. Often yields the most robust winning tickets, especially for high sparsity.
• Later Epochs (e.g., epoch 5-10): The network has learned more task-specific features. May allow for faster retraining convergence at moderate sparsity levels. The optimal rewind epoch is dataset and architecture-dependent and must be determined empirically.

Iterative Magnitude Pruning (IMP) is the foundational algorithm within which rewinding is typically applied. It is a multi-cycle process:

• Train a dense network to convergence.
• Prune a fraction (e.g., 20%) of the smallest-magnitude weights.
• Rewind the remaining weights to their values from an earlier training checkpoint.
• Retrain the sparse network from this rewound state. This cycle repeats until the target sparsity is achieved. Rewinding is the critical step that resets weights to a robust, pre-pruning state, allowing the sparse architecture to recover accuracy efficiently.

The Lottery Ticket Hypothesis is the theoretical foundation that motivates rewinding. It posits that within a dense, randomly-initialized network, there exist sparse subnetworks ("winning tickets") that, when trained in isolation from the original initialization, can match the original network's performance. Rewinding is the practical technique for finding these tickets:

• IMP with Rewinding is the standard algorithm for identifying winning tickets.
• The "rewound" state (early in training) serves as a proxy for the beneficial initial conditions the hypothesis describes.
• This contrasts with resetting to the original random initialization, which typically fails, highlighting that the early training trajectory is crucial.

Pruning-Aware Training is a complementary paradigm to post-hoc pruning methods like IMP. Instead of pruning a fully-trained model, it incorporates sparsity constraints during the initial training. Key techniques include:

• Sparse Training: Training from scratch with a fixed, randomly-initialized sparse mask.
• Regularization: Applying L1 or group sparsity penalties to encourage weights toward zero.
• Dynamic Sparse Training: Allowing the sparse connectivity pattern to evolve during training. Rewinding operates in the post-hoc pruning regime. The choice between pruning-aware training and IMP with rewinding is a trade-off between upfront training cost and final compressed model quality.

A Pruning Schedule dictates the when and how much of parameter removal during iterative pruning. It is a policy that defines:

• Initial Sparsity: Often 0% at the start of pruning.
• Final Sparsity: The target compression level (e.g., 90%).
• Pruning Frequency: How often to prune (e.g., every 1,000 training steps).
• Pruning Function: How the sparsity increases (e.g., cubic, linear). Gradual Magnitude Pruning (GMP) is a common schedule that increases sparsity slowly. Rewinding is the reset step that occurs according to this schedule. The schedule determines the checkpoint from which weights are rewound, making the two concepts tightly coupled in implementation.

This distinction defines the granularity of removal, which impacts how rewinding is applied.

• Unstructured Pruning: Removes individual weights anywhere in the network. This is the primary domain for rewinding, as it creates irregular sparsity that benefits greatly from the recovery phase.
• Structured Pruning: Removes entire structural units like filters, channels, or neurons. Rewinding can also be applied here, but the reset is applied to the remaining structural elements. The hardware-friendly nature of structured sparsity may change the trade-offs and necessity of aggressive rewinding.

Sparse Model Inference refers to the execution engines and optimizations required to realize the theoretical speedups from pruning. Rewinding produces a sparse model, but its efficiency depends on:

• Sparse Kernels: Specialized linear algebra routines that skip multiplications with zero weights.
• Hardware Support: Modern AI accelerators (e.g., NVIDIA Ampere A100's sparse tensor cores) that exploit structured N:M sparsity patterns (e.g., 2:4).
• Runtime Formats: Storage formats like CSR (Compressed Sparse Row) that reduce memory footprint. The ultimate goal of rewinding is to feed a high-accuracy, sparse model into these efficient inference pipelines.