Magnitude-Based Pruning

The technique operates on a simple, empirically validated heuristic: weights with smaller absolute magnitudes contribute less to the model's final output. It assumes that setting these near-zero values to exactly zero will have a minimal impact on accuracy. This makes it a parameter-agnostic method; it does not require evaluating the effect of removal on the loss function, unlike second-order methods like Optimal Brain Damage (OBD).

Magnitude-based pruning is most commonly applied in an unstructured manner, removing individual weights anywhere in the network. This creates irregular sparsity patterns that offer high theoretical compression but require specialized software libraries (e.g., those supporting sparse tensor operations) for efficient inference.

It can also be applied in a structured fashion by ranking and removing entire structural units:

• Filter/Channel Pruning: Removing entire convolutional filters based on their L1/L2 norm.
• Neuron Pruning: Removing neurons from fully-connected layers based on the magnitude of incoming/outgoing weights.

Applying a high pruning rate in a single step (one-shot pruning) typically causes severe accuracy loss. Therefore, magnitude-based pruning is usually implemented iteratively.

A standard algorithm is Iterative Magnitude Pruning (IMP) or Gradual Magnitude Pruning (GMP):

1. Train a dense model to convergence.
2. Prune a small percentage (e.g., 20%) of the smallest-magnitude weights.
3. Retrain the remaining sparse network to recover accuracy.
4. Repeat steps 2-3 until the target sparsity is reached. Rewinding is a related technique where, after pruning, weights are reset to values from an earlier training checkpoint before retraining, often improving recovery.

This pruning methodology led to the seminal Lottery Ticket Hypothesis. Research found that the sparse subnetworks (winning tickets) identified by iterative magnitude pruning, when trained from their original initializations, could match the accuracy of the full network. This suggests magnitude-based pruning doesn't just compress but can identify trainable, efficient architectures within larger networks. The hypothesis underscores the importance of the pruning mask (the pattern of zeros) combined with the original weight initialization.

The central challenge is managing the sparsity-accuracy tradeoff. The pruning schedule is critical—it defines how the sparsity level increases over time. An aggressive schedule hurts final accuracy, while a too-gradual one increases training cost.

Key schedule parameters include:

• Initial Sparsity: Often 0%.
• Final Sparsity: The target (e.g., 90%).
• Pruning Frequency: How often to prune (e.g., every 100 steps).
• Pruning Function: How the rate increases (e.g., cubic, linear). The goal is to allow the network to adapt to its thinning architecture gradually.

While simple, unstructured magnitude pruning has a key limitation: hardware inefficiency. The irregular memory access patterns of sparse matrices often don't achieve speedups on standard GPUs/CPUs designed for dense matrix multiplication. This has driven the development of:

• Hardware-aware pruning: Targeting structured patterns like N:M sparsity (e.g., 2:4) that are directly supported by modern sparse tensor cores.
• Pruning for inference: Focusing on patterns that maximize latency reduction on specific deployment hardware (e.g., mobile NPUs). Thus, the choice of pruning granularity is often dictated by the target inference engine.

The most common scheduling method, GMP slowly increases the sparsity of the network over many training steps according to a predefined schedule. Instead of a one-shot removal, it allows the network to adapt.

• Key Mechanism: Starts from an initial sparsity (often 0%) and ramps up to a final target sparsity (e.g., 90%) over a specified number of training steps, following a cubic or polynomial schedule.
• Benefit: By pruning gradually, the network can recover accuracy more effectively than aggressive one-shot pruning. The remaining weights have time to adjust and compensate for the removed connections.
• Example: A schedule might prune 2% of remaining weights every 1000 steps until the target 80% sparsity is reached.

Also known as prune-retrain, this is a cyclic algorithm that alternates between pruning a portion of the weights and retraining the network to recover accuracy.

• Key Mechanism: The classic IMP loop: 1) Train a dense network to convergence, 2) Prune a fixed percentage (e.g., 20%) of the smallest-magnitude weights, 3) Retrain the pruned network from its current weights, 4) Repeat steps 2-3 until the target sparsity is met.
• Connection to LTH: IMP is the experimental procedure used to discover Lottery Ticket Hypothesis subnetworks. A critical variant is rewinding, where weights are reset to an early training checkpoint (not the initial random values) after pruning, before retraining.
• Benefit: Often achieves higher accuracy at high sparsity levels compared to one-shot pruning, as retraining repeatedly recovers performance.

This variant defines the scope used for ranking weights for removal.

• Global Pruning: All weights across the entire network are pooled and ranked by a single, unified criterion (e.g., absolute value). The smallest weights globally are pruned, regardless of which layer they belong to.
  • Effect: Can lead to uneven sparsity distribution across layers, as layers with naturally smaller weight distributions may be pruned more aggressively.
• Local (Layer-wise) Pruning: Weights are ranked and pruned independently within each layer or group. A target sparsity (e.g., 50%) is applied per layer.
  • Effect: Provides precise control over the compression of each layer, which is useful for hardware-aware optimization. It prevents the complete stripping of any single layer.

This defines the aggressiveness and timing of the pruning operation relative to training.

• One-Shot Pruning: The model is trained to convergence first. Then, in a single step, weights below a threshold are pruned (set to zero). This is often followed by a short fine-tuning phase.
  • Use Case: Post-training pruning for rapid model compression. It is fast but typically incurs higher accuracy loss unless the model is highly redundant.
• Progressive Pruning: Pruning is interleaved with the training process itself. This encompasses both GMP and IMP.
  • Use Case: Standard practice for achieving high sparsity with minimal accuracy degradation. The model learns to operate with an increasingly sparse connectivity pattern during training.

A more advanced, gradient-aware variant that challenges the pure magnitude heuristic. It posits that a weight's importance is better indicated by how much it moves during fine-tuning, not just its final magnitude.

• Key Mechanism: Scores weights based on the product of their absolute value and the absolute value of their accumulated gradient during training. Weights that are small and have small gradient movement are pruned.
• Benefit: Particularly effective in task-specific fine-tuning scenarios (e.g., pruning a pre-trained BERT model for a downstream NLP task). It can identify and preserve weights that are small in magnitude but crucial for the new task, as indicated by their active gradient signals.
• Contrast: Unlike static magnitude pruning applied after training, movement pruning is a dynamic, training-aware process.

Applies the magnitude criterion to groups of weights to induce hardware-friendly sparsity patterns, bridging unstructured and structured pruning.

• Key Mechanism: Weights are grouped by a structural unit (e.g., an entire channel, filter, or neuron). The importance of the group is calculated by aggregating the magnitudes of its constituent weights (e.g., L2 norm). The least important groups are pruned entirely.
• Common Variants:
  • Channel Pruning: Removes entire output channels from a convolutional layer based on the summed magnitude of the filters in that channel.
  • Filter Pruning: Removes entire 2D convolutional filters.
• Benefit: Produces a dense, smaller model that runs efficiently on standard hardware (CPUs/GPUs) without requiring specialized sparse kernels, unlike unstructured pruning.

Unstructured pruning removes individual weights anywhere in a network based on a saliency criterion, creating an irregular, non-zero sparsity pattern. This fine-grained approach can achieve very high compression ratios but results in sparse tensors that require specialized libraries (e.g., cuSPARSE) or hardware support (sparse tensor cores) for efficient inference, as standard dense matrix multiplication is no longer optimal.

• Key Characteristic: Irregular sparsity.
• Primary Challenge: Requires sparse compute kernels to realize speedup.
• Contrast with Structured Pruning: Does not maintain regular matrix shapes.

Structured pruning removes entire structural components of a network, such as filters, channels, or neurons. This coarse-grained approach maintains regular, dense matrix structures, allowing the pruned model to run efficiently on standard hardware and deep learning frameworks without specialized sparse kernels.

• Common Targets: Convolutional filters, attention heads, or full layers.
• Hardware Advantage: Results in smaller, dense models compatible with all accelerators.
• Trade-off: Often less aggressive compression than unstructured pruning for a given accuracy target.

Iterative Magnitude Pruning is an algorithm that interleaves training and pruning phases. Starting from a trained dense network, it repeatedly:

1. Prunes a fraction of the smallest-magnitude weights.
2. Retrains the remaining sparse network to recover accuracy. This cycle continues until a target sparsity is reached. IMP is foundational to the Lottery Ticket Hypothesis, which posits that winning sparse subnetworks exist within the initial dense network.

A pruning criterion is the heuristic or metric used to score parameters for removal. While magnitude uses absolute weight value, other criteria include:

• Gradient-based: Saliency estimated from gradient information (e.g., Movement Pruning).
• Second-order: Approximating parameter importance via the Hessian matrix (e.g., Optimal Brain Damage).
• Activation-based: Using feature map statistics to prune unimportant channels. The choice of criterion directly impacts which parameters are removed and the final accuracy of the sparse network.

The sparsity pattern defines the specific locations of zero elements within weight tensors. Patterns range from unstructured (random distribution of zeros) to highly structured.

A notable engineered pattern is N:M Sparsity, where in every block of M consecutive weights, at most N are non-zero. This pattern is directly supported by NVIDIA's Ampere/Ada/Hopper GPU sparse tensor cores, enabling predictable 2x speedups for matrix operations. The pattern is a constraint applied during the pruning process.

This is a training paradigm where pruning considerations are integrated from the start, rather than applied post-hoc. Techniques include:

• Regularization: Applying L1 or group Lasso regularization to encourage weights toward zero.
• Sparse Training: Training a network with a fixed sparse connectivity pattern from initialization.
• Gradual Pruning: Slowly increasing sparsity during training (e.g., Gradual Magnitude Pruning). The goal is to steer the optimization process to learn representations that are inherently robust to the subsequent removal of parameters.