Rewinding

The core mechanism of rewinding involves rolling back the network's parameters to a saved state from an earlier epoch in the initial training run, rather than using the final trained weights. This checkpoint, often called the rewinding point, is typically chosen after the network has learned useful representations but before over-specialization. The process is: 1) Train the dense network to completion and save intermediate checkpoints. 2) Apply the pruning mask to remove weights. 3) Reset the remaining weights to their values at the chosen rewinding point. 4) Resume fine-tuning from that point. This reset is believed to preserve the favorable optimization landscape present earlier in training.

Rewinding is the practical procedure that enables the isolation of winning tickets as described by the Lottery Ticket Hypothesis. The hypothesis states that dense networks contain sparse, trainable subnetworks that can reach original performance. The original iterative magnitude pruning (IMP) algorithm found these tickets by rewinding to initialization, but later work showed rewinding to an early training epoch (late rewinding) is more effective and stable. This connection establishes rewinding not as an arbitrary step, but as the method to return the pruned subnetwork to a state from which it can be successfully re-trained, validating the existence of the lottery ticket.

Fine-tuning a pruned network starting from its final, heavily optimized weights can be unstable and lead to suboptimal convergence. Rewinding mitigates this by providing a more robust starting point. The early-training checkpoint has lower-magnitude weights and resides in a region of the loss landscape with wider, flatter minima, which is more amenable to the altered connectivity of the pruned network. This stabilization is crucial for achieving high accuracy at extreme sparsity levels (e.g., >90%). Without rewinding, the pruned network may struggle to recover, suffering from a larger pruning-induced accuracy drop.

Rewinding is inherently tied to iterative pruning schedules. A standard loop is: Prune (e.g., remove 20% of smallest-magnitude weights) → Rewind → Fine-tune for a few epochs → Repeat. The rewinding step is applied at the start of each fine-tuning phase within the cycle. This contrasts with one-shot pruning, which removes weights once and fine-tunes from the final state. The iterative-rewind approach allows the network to gradually adapt its sparsity pattern while maintaining a trainable parameter state, leading to better final performance than one-shot methods at comparable sparsity.

The choice of epoch for the rewinding point is a hyperparameter that significantly impacts results. Key findings include:

• Early Rewinding (Near Initialization): Often used in original IMP; can work but may be less stable.
• Late Rewinding (e.g., epoch 10-30 in a 100-epoch training): Empirically shown to be more effective for modern architectures. The network has learned basic features but retains plasticity.
• Task-Dependent: The optimal point can vary based on network architecture, dataset, and pruning target. It is typically determined via ablation study. The principle is to rewind to a point where the optimization trajectory is still generalizable for the sparse subnetwork.

Rewinding differs fundamentally from standard sparse fine-tuning or post-training pruning. In standard fine-tuning, the pruned network's remaining weights are initialized with their final, pre-pruned values. Rewinding explicitly rejects this, positing that these final values are over-fitted to the dense connectivity pattern. By resetting, rewinding gives the sparse architecture a fresh, but informed, start. This makes it a form of pruning-aware training, as the training process explicitly accounts for the future pruning step by saving intermediate states for potential reuse.

The foundational algorithm where rewinding is most commonly applied. IMP follows a repeated cycle:

• Train the network to convergence.
• Prune a small percentage of weights with the smallest magnitude.
• Rewind the remaining weights to their values from an earlier training checkpoint.
• Fine-tune the rewound, sparse network to recover accuracy. This cycle repeats until the target sparsity is achieved. Rewinding to a robust, early-state network is critical for the success of IMP, as it prevents the accumulation of pruning damage.

A seminal theory that provides the conceptual foundation for rewinding. It posits that within a dense, randomly-initialized network, there exist sparse subnetworks ('winning tickets') that, if trained from that initial state, can match the original network's performance. Rewinding is the practical technique used to isolate and retrain these hypothesized winning tickets. Instead of fine-tuning from the final, potentially overfitted weights, rewinding resets the subnetwork to its early, generalizable state for optimal recovery.

A pruning paradigm where entire, structurally coherent groups of weights are removed, such as filters, channels, or attention heads. This results in a smaller, dense model that maintains hardware-friendly execution patterns. Rewinding is often more effective in structured pruning scenarios because:

• The damage from removing entire structures is more significant.
• Rewinding to a robust initialization helps the network reorganize functionality into the remaining structures more effectively than fine-tuning from a damaged final state.

The retraining phase that follows a pruning (and often rewinding) step. The sparsity pattern is held fixed, and only the remaining non-zero weights are updated. Rewinding directly influences this phase by determining the starting point:

• Fine-tuning from final weights: The network must recover from a state optimized for a larger, now-altered architecture.
• Fine-tuning from rewound weights: The network starts from a more general, robust state, often leading to faster convergence and higher final accuracy. This is the recovery mechanism that makes aggressive iterative pruning feasible.

A training methodology designed to produce networks inherently robust to pruning, potentially reducing the need for aggressive rewinding. Techniques include:

• Regularization (e.g., L1) that encourages small, prunable weights.
• Progressive pruning during training.
• Dynamic Network Surgery, which cuts and splices connections in real-time. The goal is to bake resilience into the model throughout its lifecycle, making the post-pruning recovery phase (where rewinding is applied) less drastic and more predictable.

A hardware-friendly, fine-grained structured sparsity pattern. In every block of M consecutive weights, at most N are non-zero (e.g., 2:4 sparsity). This pattern is natively supported by modern NVIDIA Ampere/Ada/Hopper GPUs via the Sparse Tensor Core feature, enabling significant speedups. Rewinding is highly relevant for achieving high-accuracy models with N:M sparsity, as the strict pattern constraint makes weight selection critical. Rewinding provides a superior starting point for fine-tuning after this constrained pruning step.