Pruning Criterion

The most common and intuitive criterion, which removes weights with the smallest absolute values. The underlying assumption is that small-magnitude weights contribute less to the network's output.

• L1 Norm (Absolute Value): Directly uses |w| as the importance score.
• L2 Norm (Squared Value): Uses w², which more aggressively penalizes larger weights.
• Global vs. Layer-wise: Weights can be ranked globally across the entire network or independently within each layer, affecting the final sparsity distribution.

Example: In Iterative Magnitude Pruning (IMP), the bottom 20% of weights by absolute value are removed after each training cycle.

This criterion uses gradient information from the training process to estimate a weight's importance. It identifies parameters whose removal would have the least impact on the loss function.

• SNIP (Single-shot Network Pruning): Scores connections at initialization using the gradient of the loss with respect to the weight: |g ⊙ w|.
• GraSP (Gradient Signal Preservation): Prunes to preserve the gradient flow through the network.
• Movement Pruning: Removes weights based on how much their value changes during training (the product of weight and gradient over time), not just their final magnitude.

These methods often outperform simple magnitude pruning, especially in pruning-at-initialization scenarios.

A theoretically rigorous approach that uses the second derivative (the Hessian matrix) of the loss with respect to the weights. It estimates the expected increase in loss from removing a parameter.

• Optimal Brain Damage (OBD): An early method that uses a diagonal approximation of the Hessian to compute saliency scores: saliency = (w²) / (2 * H). High saliency weights are kept.
• Optimal Brain Surgeon (OBS): A more complex method that uses the full inverse Hessian and can account for weight interactions.

While highly accurate for identifying important weights, the computational cost of calculating the Hessian for large models is often prohibitive, limiting its practical use.

The most common pruning criterion, which removes weights with the smallest absolute values. It operates on the principle that small-magnitude weights contribute less to the model's output.

• Mechanism: Applies a global or layer-wise threshold.
• Hardware Impact: Often results in unstructured sparsity, requiring specialized libraries for speedup.
• Example: In Iterative Magnitude Pruning (IMP), the bottom 20% of weights by magnitude are removed each cycle.

Criteria that use gradient information to estimate a parameter's importance to the training loss. They aim to prune weights that, if removed, would cause the least increase in loss.

• Mechanism: Scores like gradient*magnitude (e.g., Movement Pruning) or the gradient's effect on the loss (e.g., SNIP).
• Advantage: Can identify important weights that have small magnitude but large gradient flow.
• Use Case: Often used in pruning-at-initialization methods before training begins.

Criteria that prune based on the statistics of neuron or channel activations during a forward pass over a calibration dataset. Channels with low average activation are deemed less important.

• Mechanism: Measures the average percentage of zeros (APoZ) in activations or their L2 norm.
• Result: Naturally leads to structured pruning, like channel pruning in CNNs.
• Benefit: Produces dense, hardware-friendly sub-networks without irregular sparsity patterns.

A theoretically rigorous criterion that estimates the expected increase in loss from removing a weight using second-order derivative (Hessian) information. It approximates the loss function's curvature.

• Mechanism: Uses the diagonal of the Hessian or a Fisher Information matrix to compute optimal brain damage or optimal brain surgeon scores.
• Characteristic: Highly accurate but computationally expensive to calculate for large models.
• Application: Foundational for early pruning research and sensitivity analysis.

Criteria designed to remove entire structural units (filters, channels, attention heads) rather than individual weights. The score for a group is typically an aggregate of its constituent weights.

• Common Metrics: L2 norm of a filter's weights, mean activation magnitude, or rank of feature maps.
• Outcome: Enforces N:M sparsity or block sparsity patterns that are efficient on modern GPUs.
• Examples: Channel pruning in ResNet, attention head pruning in transformers.

Not a criterion itself, but a framework that influences how criteria are applied. It identifies sparse, trainable subnetworks ('winning tickets') within a dense network.

• Process: Uses a criterion (like magnitude) to prune, then rewinds weights to their early-training values before re-training.
• Insight: Suggests the initialization and early training dynamics are crucial for finding effective sparse networks.
• Impact: Led to advanced pruning schedules and pruning-aware training methodologies.