Pruning Criterion

This is the most common and intuitive criterion. It operates on the principle that weights with smaller absolute values contribute less to the model's output. Pruning is performed by ranking all weights (globally or locally) and removing those below a threshold.

• Key Assumption: Small weights are less important.
• Typical Use: Serves as a strong baseline; foundational to algorithms like Iterative Magnitude Pruning (IMP) and Gradual Magnitude Pruning (GMP).
• Advantage: Computationally cheap, requires no gradient information.
• Limitation: A small weight can be critical for fine-grained adjustments, and magnitude alone ignores the parameter's interaction within the network.

This criterion uses gradient information from a task-specific loss function to score parameters. Movement Pruning is a prominent example, where the importance score is the product of the weight's value and its gradient. Parameters are pruned if this score indicates they are moving towards zero during fine-tuning.

• Key Assumption: Weights that are actively decreasing in magnitude during training are less important for the task.
• Typical Use: Highly effective during task-specific fine-tuning, as it directly ties importance to the learning signal.
• Advantage: More task-aware than static magnitude; can preserve small weights that have large, important gradients.
• Limitation: Requires a training/fine-tuning phase and maintaining gradient computations, increasing overhead.

This advanced criterion estimates the impact of removing a parameter on the overall loss function using second-order derivatives (the Hessian matrix). Optimal Brain Damage (OBD) and its successor, Optimal Brain Surgeon, are classic algorithms in this category.

• Key Assumption: The optimal set of parameters minimizes the increase in training loss.
• Typical Use: Research-focused pruning for finding theoretically optimal sparse subnetworks.
• Advantage: Provides a rigorous, loss-aware importance metric.
• Limitation: Computationally prohibitive for large models due to the cost of calculating or approximating the Hessian (O(N²) complexity).

This criterion focuses on the model's activations rather than its weights. Importance is scored by the average magnitude or variance of a neuron's or channel's output activation over a dataset. Channels with consistently low activation magnitudes are pruned.

• Key Assumption: Neurons or channels that fire weakly across inputs are redundant.
• Typical Use: A common method for structured pruning like channel pruning or filter pruning in CNNs.
• Advantage: Directly targets reducing feature map computations; the pruned model has immediately smaller activation tensors.
• Limitation: Requires a calibration dataset to compute activation statistics, and the criterion is sensitive to the data distribution used for calibration.

This criterion is applied during training by adding sparsity-inducing regularizers to the loss function. The most common is L1 regularization (Lasso), which encourages weights to shrink to exactly zero. Group Lasso can be used to push entire structural groups (e.g., channels) to zero for structured pruning.

• Key Assumption: Sparsity can be learned as part of the optimization objective.
• Typical Use: Pruning-aware training or sparse training paradigms where the model is trained to be sparse from the start.
• Advantage: Unifies training and compression into a single phase; the emerging sparsity is naturally aligned with the training process.
• Limitation: Introduces hyperparameter tuning for the regularization strength and can make the optimization landscape more challenging.

This is not a single mathematical criterion but a design philosophy where the scoring function is co-designed with target hardware constraints. The goal is to produce sparsity patterns that maximize actual speedups and energy savings.

• Key Examples:
  • N:M Sparsity: Enforcing that in every block of M weights, at most N are non-zero. This pattern maps efficiently to NVIDIA's Sparse Tensor Cores.
  • Block-wise Sparsity: Pruning in contiguous blocks to improve memory access patterns and enable SIMD operations.
  • Pattern-Based Pruning: Restricting non-zero weights to specific, hardware-friendly shapes within a matrix.
• Advantage: Delivers real-world inference latency and power improvements, not just theoretical FLOP reduction.
• Limitation: The search for optimal hardware-specific patterns can be complex and requires detailed hardware performance models.

The most common pruning criterion, which removes network parameters with the smallest absolute values. It operates on the heuristic that weights near zero contribute minimally to the model's output.

• Key Assumption: Low-magnitude weights are less important.
• Implementation: Often applied with a global or local threshold.
• Example: Pruning all weights where |w| < 0.01.

These terms define the granularity of removal, which is a critical design choice linked to the pruning criterion.

• Unstructured Pruning: The criterion selects individual weights anywhere in the tensor, creating an irregular sparsity pattern. Maximizes parameter reduction but requires specialized libraries for speedup.
• Structured Pruning: The criterion selects entire structural components (e.g., filters, channels). This maintains dense matrix operations for efficient inference on standard hardware but offers less fine-grained compression.

The policy that dictates when and how many parameters are removed based on the criterion. It defines the progression from a dense to a sparse network.

• One-Shot: Apply the criterion once at a high sparsity target, often post-training. High risk of accuracy loss.
• Iterative: (e.g., Iterative Magnitude Pruning) Repeatedly: prune a small fraction (e.g., 20%) of lowest-magnitude weights → retrain. Allows the network to recover accuracy.
• Gradual: Smoothly increases sparsity from 0% to a target over many training steps, allowing continuous adaptation.

A hardware-friendly structured sparsity pattern that can be enforced by a pruning criterion. In every block of M consecutive weights, at most N are non-zero.

• Criterion Role: The scoring function must select the top N most important weights in each block.
• Hardware Advantage: This pattern can be executed efficiently on modern GPU sparse tensor cores (e.g., NVIDIA's 2:4 sparsity), delivering near-dense throughput.
• Example: 2:4 sparsity means for every block of 4 weights, 2 are zeroed. This achieves a 50% reduction in weights and memory with minimal inference overhead.

A gradient-based pruning criterion that scores weights not by their final magnitude, but by how much their value changes during fine-tuning.

• Core Idea: Weights that "move" (change) the most during task-specific training are the most important for that task.
• Process: Scores are computed as the product of the weight value and its gradient accumulated over training steps. Weights with low scores are pruned.
• Advantage: Particularly effective for pruning during fine-tuning, as it identifies weights relevant to the target domain.

An early second-order pruning criterion that uses an approximation of the Hessian matrix to estimate the saliency (importance) of each parameter.

• Saliency Formula: Estimates the increase in training error if a weight is removed. Weights with the lowest saliency are pruned.
• Computational Cost: The full Hessian is infeasible for large networks; OBD uses a diagonal approximation, assuming parameter interactions are negligible.
• Historical Significance: Introduced the concept of estimating the true cost of removing a parameter, moving beyond simple heuristics like magnitude.