Model Pruning

This is the most common heuristic technique. It operates on the principle that parameters with smaller absolute values contribute less to the model's output. The process involves:

• Ranking all weights (or weights within a layer) by their absolute magnitude.
• Removing the smallest k% of weights, setting them to zero.
• The resulting sparsity pattern is unstructured, as zeros are scattered irregularly throughout the tensor. While simple and effective, it assumes weight magnitude is a direct proxy for importance, which isn't always true, especially after ReLU activations where negative weights can be important.

This technique removes entire structural components of the network to maintain hardware-friendly, dense operations. Unlike unstructured pruning, it directly reduces the size of weight matrices and feature maps. Key forms include:

• Filter Pruning: Removes entire convolutional filters, reducing the number of output channels.
• Channel Pruning: Removes input channels to a layer, reducing the width of incoming feature maps.
• Neuron Pruning: Removes entire neurons (units) in fully connected layers. The benefit is immediate speed-up on standard hardware (CPUs/GPUs) without requiring specialized sparse kernels, but it is typically less aggressive for a given accuracy drop compared to unstructured methods.

These criteria use gradient information from the training loss to estimate a parameter's importance, rather than just its final static value. The core idea is that a weight's contribution to learning is more informative than its magnitude alone.

• Movement Pruning: Scores weights based on how much their value changes (moves) during fine-tuning. Weights that move little are pruned. This is particularly effective for task-specific pruning during fine-tuning.
• First-Order Methods: Use the product of a weight's value and its gradient (or similar measures) as an importance score. These methods often outperform simple magnitude pruning when applied during training or fine-tuning, as they identify weights the model is not relying on for the current task.

Instead of pruning once, iterative schedules prune a small fraction of weights over multiple cycles, allowing the network to recover accuracy. This is crucial for achieving high sparsity levels (>80%). Common schedules:

• Iterative Magnitude Pruning (IMP): Train → Prune smallest weights → Retrain. Repeat.
• Gradual Magnitude Pruning (GMP): A smooth variant where sparsity is increased from 0% to a target (e.g., 90%) over many training steps according to a cubic or polynomial schedule.
• Rewinding: After pruning, reset weights to values from an earlier training checkpoint (e.g., epoch 5) before retraining, which often yields better performance than retraining from the final pre-pruned state.

This approach designs the sparsity pattern to align with the target hardware's execution engine for maximum efficiency, moving beyond simple unstructured sparsity.

• N:M Sparsity: A form of fine-grained structured sparsity. In every block of M consecutive weights (e.g., 4), at most N are non-zero (e.g., 2:4). This pattern can be executed efficiently on NVIDIA's Sparse Tensor Cores (Ampere+ GPUs).
• Block Sparsity: Weights are pruned in contiguous blocks (e.g., 4x4), improving memory access patterns and enabling vectorized operations.
• Channel/Filter Alignment: For structured pruning, aligning pruned filters across layers can prevent mismatches in tensor dimensions and simplify the runtime graph.

A influential theory that reframes the goal of pruning. It states: Within a dense, randomly-initialized neural network, there exists a sparse subnetwork (a 'winning ticket') that, when trained in isolation from the initialization, can match the test accuracy of the original dense network. Key implications for pruning criteria:

• Pruning at Initialization: The search for criteria to identify winning tickets before training.
• Importance of Rewinding: Supports resetting to early epoch weights, not random initialization.
• It suggests that successful pruning criteria are those that can identify this trainable sparse architecture early in or before the training process.

This is the most common heuristic pruning technique. It operates on the principle that weights with small absolute values contribute less to the model's output. The standard workflow involves:

• Ranking all parameters (or parameters within a layer) by their absolute value.
• Applying a threshold (e.g., remove the bottom 20% of weights) or a target sparsity level.
• Zeroing out the selected weights, creating a sparse model. While simple and computationally cheap, it is a local, static criterion that doesn't consider the interaction between weights. Frameworks like TensorFlow Model Optimization Toolkit and PyTorch provide built-in utilities for magnitude-based pruning.

IMP is an algorithm designed to mitigate the accuracy loss from one-shot pruning. Instead of pruning once, it alternates in cycles:

1. Train a model to convergence or a high-performance checkpoint.
2. Prune a small fraction (e.g., 20%) of the smallest-magnitude weights.
3. Retrain the remaining sparse network to recover accuracy. This cycle repeats until the target sparsity is reached. IMP is foundational to the Lottery Ticket Hypothesis, which posits that the successful sparse subnetwork found by IMP (the 'winning ticket') exists within the initial dense network. The rewinding technique, where weights are reset to an early training checkpoint (not initial randomization) after pruning, is often used within IMP.

A gradient-based technique developed for task-specific fine-tuning of large pre-trained models (e.g., BERT). Unlike magnitude pruning, which uses a static snapshot of weights, movement pruning scores parameters based on how their values change during fine-tuning.

• Saliency Score: Calculated as the product of the weight's value and its gradient. Weights that move towards zero are pruned.
• Dynamic Mask: The pruning mask is learned jointly with the model weights via a regularized objective.
• Advantage: It can prune weights that start with large magnitudes but become unimportant for the downstream task. This method is particularly effective in NLP fine-tuning scenarios where the pre-trained model's knowledge needs to be selectively adapted.

These methods remove entire structural components to maintain hardware-efficient, dense computation. Key approaches include:

• Channel/Filter Pruning: Removes entire output channels from convolutional layers. Importance can be judged by L1 norm of filters, activation rank, or impact on the next layer's activation.
• Neuron Pruning: Removes entire neurons from fully connected layers.
• N:M Structured Sparsity: Enforces a pattern where in every block of M consecutive weights, at most N are non-zero. This pattern can be executed efficiently on NVIDIA Ampere/Ada/Hopper GPU sparse tensor cores. Frameworks like TorchSparse and hardware vendor libraries (e.g., NVIDIA cuSPARSELt) provide kernels to accelerate inference with structured sparsity patterns.

These paradigms integrate sparsity into the training loop from the start.

• Pruning-Aware Training: The model is trained normally but with a regularization term (e.g., L0 or L1) that encourages weights to become small and prunable, or with the pruning schedule applied during training (e.g., Gradual Magnitude Pruning).
• Sparse Training (RigL, SET): The network is initialized as a sparse topology (e.g., Erdős–Rényi random graph) and the connectivity pattern is dynamically updated during training. Weights that become unimportant are dropped and new connections are grown based on gradient information. This avoids the memory and compute cost of ever training a dense model.

Structured pruning removes entire structural components—such as neurons, channels, or filters—to maintain hardware-friendly, dense matrix operations. Unlike unstructured pruning, it results in a smaller but still regularly shaped network.

• Key Benefit: Directly reduces the dimensions of weight matrices, leading to immediate speed-ups on standard hardware (CPUs/GPUs) without requiring specialized sparse kernels.
• Common Targets: Pruning entire output channels from convolutional layers or rows/columns from fully connected weight matrices.
• Trade-off: While more hardware-efficient, it is often less fine-grained than unstructured pruning and may remove more parameters to achieve a similar accuracy drop.

Unstructured pruning removes individual weights anywhere in a network, resulting in an irregular, sparse connectivity pattern. This offers maximal granularity for parameter removal.

• Key Characteristic: Creates a sparse weight tensor where most elements are zero, but the tensor's original dimensions remain unchanged.
• Hardware Challenge: The irregular pattern does not leverage standard dense linear algebra libraries efficiently. It requires specialized software support (e.g., sparse BLAS libraries) or hardware with sparse tensor cores (like NVIDIA's Ampere architecture with 2:4 sparsity) for acceleration.
• Use Case: Often achieves higher final compression rates for a given accuracy target when the appropriate inference infrastructure is available.

Magnitude-based pruning is the most common heuristic technique, which removes network parameters with the smallest absolute values, operating on the assumption that low-weight connections contribute less to the model's output.

• Algorithm Simplicity: It is straightforward to implement: after training, a global or local threshold is applied, and weights below it are set to zero.
• Foundation for Iterative Methods: Serves as the core operation in advanced schedules like Iterative Magnitude Pruning (IMP) and Gradual Magnitude Pruning (GMP), where pruning is interleaved with retraining to recover accuracy.
• Limitation: It is a static, post-hoc measure that doesn't account for the parameter's sensitivity or interaction with other weights, which can sometimes lead to suboptimal selections.

The Lottery Ticket Hypothesis is a influential theory stating that within a dense, randomly-initialized neural network, there exists a sparse subnetwork (a 'winning ticket') that, when trained in isolation from the original initialization, can match the performance of the original full network.

• Core Finding: Challenges the notion that over-parameterization is solely for optimization; it suggests dense networks are effective because they contain these trainable sparse subnetworks.
• Practical Algorithm: Iterative Magnitude Pruning (IMP) is the primary method used to find these winning tickets. It involves iterative cycles of training, pruning the smallest-magnitude weights, and rewinding the remaining weights to their values from an early training epoch before retraining.
• Impact: Inspired research into efficient sparse training from the start, rather than pruning after expensive dense training.

A sparsity pattern defines the specific locations of zero-valued elements within a weight tensor. The pattern dictates the potential for inference acceleration and is a key consideration in hardware-aware pruning.

• Unstructured Pattern: Zeros are randomly distributed. Maximizes parameter removal flexibility but requires specialized runtime support.
• Structured Pattern (e.g., N:M Sparsity): A hardware-friendly constraint where, in every block of M consecutive weights, at most N are non-zero. For example, 2:4 sparsity is natively supported on NVIDIA A100/A6000 GPUs, allowing sparse weights to be stored compressed and computed at nearly double the dense throughput.
• Block-wise Pattern: Zeros are arranged in contiguous blocks (e.g., 4x4 blocks), offering a balance between flexibility and hardware efficiency for certain accelerators.

Pruning-aware training is a model development paradigm where the eventual pruning process is anticipated and integrated into the initial training loop, as opposed to applying pruning as a separate post-training step.

• Objective: To encourage the network to learn representations that are robust to the removal of parameters, minimizing accuracy loss when pruning is finally applied.
• Common Techniques:
  • Adding sparsity-inducing regularizers like L1 regularization to push weights toward zero.
  • Using a gradual pruning schedule that slowly increases sparsity during training, allowing the network to adapt continuously.
  • Employing techniques like Movement Pruning, which uses gradient signals during fine-tuning to identify unimportant weights.
• Benefit: Typically yields a more accurate final sparse model compared to aggressive post-training pruning.