Local Pruning

The defining characteristic of local pruning is that it operates independently on each layer. A pruning criterion (e.g., weight magnitude) is applied to rank parameters within a single layer, and the least important ones are removed based on a layer-specific target sparsity. This contrasts with global pruning, which ranks all parameters in the network together. For example, a convolutional layer might be pruned to 50% sparsity while a fully connected layer is pruned to 70%, based on their individual redundancy and sensitivity.

Local pruning requires a predefined sparsity distribution across the network. The engineer must specify the target sparsity (e.g., 30%, 50%, 80%) for each layer or block before the algorithm runs. This leads to a predictable, controlled compression profile but requires careful layer-wise sensitivity analysis. Common strategies include:

• Uniform sparsity: Applying the same target (e.g., 50%) to all layers.
• Sensitivity-based sparsity: Assigning higher sparsity to larger, more redundant layers (e.g., fully connected layers) and lower sparsity to critical, smaller layers (e.g., early convolutional filters).

Because it processes layers in isolation, local pruning is often simpler to implement and calibrate than global methods. The pruning threshold for a layer can be directly calculated from its weight distribution (e.g., the 30th percentile of absolute values). This avoids the need for complex, network-wide heuristics or iterative threshold searching. It is commonly used in post-training pruning workflows where a pre-trained model is pruned in a single pass, as the per-layer thresholds can be set using statistical rules.

A key limitation is that local pruning can lead to suboptimal global compression. By not considering cross-layer importance, it may aggressively prune a sensitive layer while leaving highly redundant parameters in another. This can result in greater accuracy loss for a given overall model sparsity compared to global pruning. The technique assumes that local importance is a good proxy for global importance, which may not hold true for all architectures, particularly those with residual connections or strong inter-layer dependencies.

Local pruning is frequently employed in scenarios requiring straightforward, predictable compression. Use cases include:

• Rapid prototyping of sparse models.
• Pruning as a regularization technique during training.
• Hardware-aware pruning where specific layers must meet exact memory or FLOP constraints.

It is a foundational method in libraries like TensorFlow Model Optimization Toolkit and PyTorch's torch.nn.utils.prune. For instance, torch.nn.utils.prune.l1_unstructured applies L1-norm-based local pruning to a specified module.

This table clarifies the primary distinction between local and global pruning strategies:

Global pruning often achieves higher accuracy at high sparsity but is more complex to tune.

Global pruning ranks and removes parameters across the entire network using a single, unified importance score. Unlike local pruning, which sets per-layer thresholds, global pruning applies one global threshold, often leading to more aggressive pruning of less critical layers. This can create a more optimal network-wide sparsity distribution but may require careful tuning to avoid damaging sensitive layers.

• Key Mechanism: A single saliency criterion (e.g., weight magnitude) is computed for all parameters. The bottom X% globally are pruned.
• Trade-off: Can achieve higher overall compression for a given accuracy drop but risks uneven sparsity that may not align with layer sensitivity.

A pruning criterion is the heuristic or metric used to score parameters for removal. The choice of criterion fundamentally defines what is considered 'unimportant.'

Common criteria include:

• Magnitude: Absolute weight value (|w|). The core assumption is small weights contribute less.
• Gradient-Based: Sensitivity of the loss function to a weight's removal.
• Activation-Based: Measures of neuron/output channel activity.
• Second-Order: Approximations of the Hessian (parameter importance curvature), as in Optimal Brain Damage (OBD).

Local pruning typically applies the chosen criterion independently within each layer to generate local rankings.

Pruning granularity specifies the structural unit of removal, defining the 'grain size' of sparsity. It is a key axis orthogonal to the local/global strategy.

• Fine-Grained (Unstructured): Individual weights are pruned anywhere. Maximizes parameter reduction but creates irregular memory access patterns.
• Structured: Coarser units like entire filters, channels, or neurons are removed. This maintains dense matrix operations for efficient hardware execution.
• Block/Semi-Structured: Intermediate patterns like N:M sparsity, where in every block of M weights, N are forced to zero.

Local pruning can be applied at any granularity, determining the sparsity pattern within each layer.

A pruning schedule is a policy defining when and how many parameters to prune over the course of training or fine-tuning. It manages the rate of sparsity introduction.

• One-Shot: Prune to target sparsity in a single step, often followed by retraining.
• Iterative: Prune a small percentage, retrain, and repeat over multiple cycles. This is the basis of Iterative Magnitude Pruning (IMP).
• Gradual: Smoothly increases sparsity from an initial rate (e.g., 0%) to a final target over many training steps, as in Gradual Magnitude Pruning (GMP).

A schedule is applied on top of a local or global strategy to control the transition to a sparse network.

Hardware-aware pruning co-designs the sparsity pattern with the target deployment hardware's execution engine and memory hierarchy. The goal is to translate theoretical parameter reduction into actual latency and energy savings.

• Targets: Exploits specific hardware support for sparse operations (e.g., NVIDIA's sparse tensor cores).
• Pattern Constraints: May enforce structured pruning or specific N:M sparsity patterns that the hardware can accelerate.
• Local Considerations: Even within a local pruning strategy, the chosen granularity (e.g., pruning 2:4 within blocks) is often dictated by hardware capabilities to ensure speedup.

Sparse model inference refers to the runtime systems, kernels, and libraries required to execute a pruned network efficiently. Simply having zero-valued weights does not guarantee faster inference; the software/hardware stack must skip computations on these zeros.

• Sparse Formats: Use data structures like CSR (Compressed Sparse Row) or CSC to store only non-zero values and their indices.
• Kernel Optimization: Specialized matrix multiplication kernels that avoid multiplications with zero.
• Challenge for Unstructured Pruning: Irregular sparsity often leads to overhead from index management, reducing the theoretical speedup. This motivates structured or hardware-aligned pruning patterns used in local strategies.