Pruning at Initialization

SNIP is a foundational PaI method that scores each parameter's importance based on its effect on the loss function before training begins. It computes the gradient of the loss with respect to each weight and uses the magnitude of this gradient-times-weight product as a saliency score.

• Mechanism: Saliency is calculated as the absolute value of |gradient * weight|. Weights with the smallest scores are pruned.
• Advantage: Provides a principled, one-shot pruning criterion grounded in the training objective.
• Limitation: The saliency score is computed on a single, small batch of data, which may not be fully representative.

GraSP selects weights for pruning based on their impact on the gradient flow through the network. Unlike SNIP, which focuses on the loss, GraSP aims to preserve the flow of information needed for learning.

• Mechanism: It computes the Hessian-gradient product to estimate how removing a weight affects the gradient norm. Weights whose removal most hurts the gradient flow are kept.
• Objective: To maintain the network's trainability by preserving the directional alignment of gradients, often leading to better performance after training the sparse subnetwork.

SynFlow is a PaI algorithm designed to be data-agnostic. It identifies important connections by analyzing the network's structure and initialization, without requiring any training data.

• Mechanism: It performs a single forward pass using a vector of all ones as input and weights replaced by their absolute values. It then iteratively computes scores that measure synaptic strength.
• Key Benefit: Its data-independent nature makes it useful for privacy-sensitive applications or when representative data is unavailable.
• Result: Tends to preserve weights that form connected paths across the network, avoiding isolated, dead neurons.

PaI methods are directly inspired by the Lottery Ticket Hypothesis, which posits that dense networks contain sparse, trainable subnetworks ('winning tickets').

• PaI as Ticket Finding: These algorithms attempt to identify a promising subnetwork at initialization, before the costly process of training the dense network.
• Goal: To find a strong ticket—a sparse mask—that, when trained from scratch, matches or exceeds the performance of the dense model.
• Significance: Successful PaI validates the hypothesis and provides a practical path to efficient training from the start.

PaI is often contrasted with the canonical Iterative Magnitude Pruning (IMP) method, which prunes after or during training.

• IMP Workflow: Train → Prune smallest-magnitude weights → Retrain. This cycle repeats.
• PaI Workflow: Score weights at init → Prune → Train sparse network once.
• Key Difference: PaI aims to avoid training the dense model altogether, offering massive computational savings. However, the found subnetworks have historically struggled to match the accuracy of those found by IMP, especially at high sparsity levels (>90%).

The practical utility of PaI depends heavily on the induced sparsity pattern and target hardware.

• Unstructured Sparsity: Most PaI methods produce irregular, fine-grained sparsity. This requires specialized libraries (e.g., DeepSpeed) or hardware (e.g., sparse tensor cores) for speedups.
• Structured PaI: Emerging techniques aim to impose N:M or block-structured sparsity at initialization to leverage modern GPU acceleration.
• Deployment Reality: A 90% sparse model only provides inference speedups if the hardware and software stack can efficiently skip zero-valued computations.

SNIP is a foundational pruning-at-initialization algorithm. It scores the importance of each weight connection based on its estimated effect on the loss function using a single gradient computation before any training. The method approximates the change in loss if a weight were removed, allowing for a one-shot pruning of the network based on initial gradients. This contrasts with iterative methods that require multiple training/pruning cycles.

The Lottery Ticket Hypothesis posits that within a dense, randomly-initialized neural network, there exist sparse subnetworks ('winning tickets') that, when trained in isolation from their initial weights, can match the performance of the original dense network. Pruning-at-initialization techniques like SNIP and GraSP are often used to identify these subnetworks early, providing empirical support for this hypothesis by finding trainable sparse structures before extensive training.

Iterative Magnitude Pruning (IMP) is the canonical post-training pruning algorithm, forming a key contrast to pruning-at-initialization. Its standard procedure is:

• Train a dense network to completion.
• Prune a small percentage of weights with the smallest magnitude.
• Retrain the remaining sparse network to recover accuracy.
• Repeat the prune/retrain cycle. Unlike pruning-at-initialization, IMP requires a fully trained model as a starting point and involves computationally expensive iterative retraining.

Pruning-aware training integrates sparsity induction directly into the training loop. Techniques include:

• Adding L0 or L1 regularization to encourage weights toward zero.
• Using straight-through estimators to train with discrete masks.
• Progressively increasing sparsity during training via a pruning schedule. This differs from pruning-at-initialization, which makes a one-time decision before training, and from post-training pruning, which acts after training. The goal is to produce a network inherently robust to parameter removal.

This distinction defines the granularity of removal, applicable to all pruning paradigms:

• Unstructured Pruning: Removes individual weights, creating an irregular, fine-grained sparsity pattern. This achieves high theoretical compression but requires specialized software/hardware (e.g., sparse kernels) for speedup.
• Structured Pruning: Removes entire structural units like filters, channels, or attention heads. This results in a smaller, dense model that runs efficiently on standard hardware but offers less granular compression. Pruning-at-initialization methods can be designed for either granularity.

GraSP is a pruning-at-initialization method that selects weights to preserve the gradient flow through the network. Instead of scoring individual weights, it aims to prune so that the gradient norm of the pruned subnet's loss is maximally preserved relative to the dense network. The criterion is based on the Hessian-gradient product, identifying weights that are critical for the optimization trajectory rather than just their instantaneous saliency. This often finds subnetworks that train more effectively than those found by magnitude or SNIP-based criteria.