Global Pruning

The defining feature of global pruning is its single, network-wide criterion for parameter removal. Unlike local pruning, which applies a threshold or sparsity target independently to each layer, global pruning evaluates all parameters (e.g., weights, filters) across the model against one metric. Common criteria include:

• Absolute weight magnitude (smallest values removed first).
• Gradient-based saliency (e.g., from Movement Pruning).
• Second-order approximations (e.g., Optimal Brain Damage). This holistic view often leads to a more optimal distribution of sparsity, as it can aggressively prune less critical layers while preserving parameters in more sensitive ones.

Understanding global pruning requires contrasting it with its counterpart, local pruning. Their key operational differences are:

• Scope of Decision: Global uses a network-wide threshold; local uses per-layer thresholds.
• Resulting Sparsity: Global pruning can produce uneven sparsity across layers, potentially zeroing out an entire unimportant layer. Local pruning enforces uniform or predefined sparsity per layer.
• Use Case: Global is often favored for maximizing overall compression. Local is used when specific hardware or latency constraints require controlled, predictable reductions per layer. The choice impacts the final sparsity pattern and the techniques needed for efficient sparse model inference.

Global pruning directly engages the fundamental sparsity-accuracy tradeoff. By applying one threshold to the entire model, a small increase in the pruning rate can abruptly remove critical parameters from sensitive layers, causing a sharp accuracy drop. Therefore, global pruning is typically combined with:

• Iterative Pruning Schedules: Like Gradual Magnitude Pruning (GMP), which slowly increases global sparsity over training steps, allowing the network to adapt.
• Fine-Tuning/Rewinding: Retraining the pruned network to recover accuracy.
• Careful Criterion Selection: Using more informed metrics than simple magnitude to better identify globally unimportant parameters.

The irregular, layer-agnostic sparsity created by global pruning presents specific challenges and opportunities for deployment:

• Inference Libraries: It typically generates unstructured sparsity, requiring specialized kernels or libraries (e.g., those leveraging Sparse Tensor Cores) to realize speedups over dense operations.
• Memory Footprint: The pruning mask (defining zero locations) must be stored, adding overhead. Formats like Compressed Sparse Row (CSR) are used.
• Hardware-Aware Variants: Pure global pruning is often hardware-agnostic. Hardware-aware pruning techniques may hybridize global criteria with structured constraints (like N:M sparsity) to align with specific accelerator capabilities.

Global pruning is central to research inspired by the Lottery Ticket Hypothesis. This theory posits that dense networks contain sparse, trainable subnetworks ('winning tickets'). The standard algorithm to find them, Iterative Magnitude Pruning (IMP), is a global pruning method:

1. Train a dense network.
2. Prune a percentage of the globally smallest-magnitude weights.
3. Rewind the remaining weights to their values from an early training iteration.
4. Retrain the sparse subnetwork. This iterative global search demonstrates that globally identified sparse structures can achieve accuracy matching the original dense network.

A standard pipeline for applying global magnitude-based pruning involves these steps:

1. Train or Load a pre-trained dense model.
2. Score Parameters: Compute the chosen importance metric (e.g., absolute weight value) for all pruneable parameters.
3. Rank Globally: Sort all scored parameters from least to most important.
4. Apply Threshold: Remove (set to zero) a target percentage (pruning rate) of the lowest-ranked parameters, creating a global pruning mask.
5. Fine-Tune: Retrain the pruned network (often with the mask applied) to recover lost accuracy. This cycle (score-rank-prune-fine-tune) may be repeated iteratively.

Local pruning is a contrasting strategy to global pruning where parameters are ranked and removed independently within each layer or group of a network. Instead of a single, network-wide importance threshold, local pruning uses per-layer sparsity targets or thresholds.

• Key Difference: Prioritizes uniformity of sparsity across layers, which can prevent aggressive over-pruning in critical layers that might occur with a global budget.
• Common Use: Often used in structured pruning scenarios (e.g., channel pruning) where maintaining balanced layer widths is important for hardware efficiency.
• Trade-off: May preserve less globally important parameters in dense layers while removing more important ones in already sparse layers, potentially leading to suboptimal global compression.

A pruning criterion is the specific metric or heuristic used to score parameters and determine which ones are less important and can be removed. The choice of criterion fundamentally drives the pruning outcome.

• Magnitude-Based Criterion: Removes weights with the smallest absolute values (|weight|). This is the most common heuristic for global pruning due to its simplicity and low computational cost.
• Gradient-Based Criteria: Such as movement pruning, which removes weights based on how much their value changes during fine-tuning, under the assumption that weights that don't 'move' are less critical.
• Second-Order Criteria: Like Optimal Brain Damage (OBD), which use approximations of the loss function's Hessian to estimate a parameter's saliency. More accurate but computationally expensive.
• The global pruning strategy can be combined with any of these criteria to perform the network-wide ranking.

The sparsity pattern defines the specific locations of the zero-valued elements within the pruned weight tensors. Global pruning typically generates an unstructured sparsity pattern.

• Unstructured Sparsity: Zeros are distributed randomly throughout the weight matrices. This offers the highest theoretical compression but requires specialized software libraries (e.g., sparse linear algebra kernels) or hardware support (e.g., sparse tensor cores) for speedup.
• Structured Sparsity (N:M): A hardware-friendly pattern where, in every block of M consecutive weights, at most N are non-zero. Achieving this with pure global pruning is complex, often requiring post-processing.
• Block Sparsity: Zeros occur in contiguous blocks. This is a middle ground, offering more regular memory access than unstructured sparsity.
• The pattern is a critical determinant of actual inference speedup on real hardware, often more so than the sheer percentage of zeros.

A pruning schedule is the policy that dictates when and how many parameters to prune during an iterative pruning-and-training process. It defines the progression from a dense model to a sparse one.

• One-Shot Pruning: Applies the target pruning rate (e.g., 50%) in a single step after training. High risk of accuracy loss.
• Iterative Pruning: Prunes a small fraction of weights (e.g., 20%), then retrains (fine-tunes) the network to recover accuracy, repeating this cycle until the target sparsity is reached. This is the standard for global magnitude pruning.
• Gradual Magnitude Pruning (GMP): A specific iterative schedule that starts from 0% sparsity and gradually increases it according to a function (e.g., cubic) over many training steps, allowing smooth adaptation.
• The schedule is essential for managing the sparsity-accuracy tradeoff and is a key hyperparameter in any pruning workflow.

Pruning for inference is the overarching engineering goal that motivates techniques like global pruning: to create a model that is optimized specifically for the deployment (inference) phase.

• Primary Objectives: Reduce model size (memory footprint), computational FLOPs, and ultimately inference latency and power consumption.
• Hardware-Aware Pruning: An advanced approach that tailors the pruning granularity and sparsity pattern to the target deployment hardware (e.g., mobile NPU, GPU sparse tensor cores) to maximize real-world speedups.
• Toolchain Integration: The pruned model must be compatible with the deployment inference engine (e.g., TensorFlow Lite, ONNX Runtime, PyTorch Mobile). This often involves converting the pruning mask into a format the engine can leverage, such as a sparse model format.
• The ultimate validation of a pruning technique is its measured performance improvement in a production inference environment.

Sparse training is an alternative paradigm to the traditional 'train-then-prune' approach. Here, the network is initialized and trained with a fixed sparse connectivity pattern from the start.

• Contrast with Pruning: Avoids the costly dense pre-training phase entirely. The sparse architecture is defined a priori, often based on the Lottery Ticket Hypothesis.
• Connection to Global Pruning: The initial sparse mask can be discovered by applying a global pruning criterion to a randomly initialized network, identifying the 'winning ticket' subnetwork before any training occurs.
• Advantages: Can lead to significant reductions in total training compute and memory. It forces the network to learn efficient representations from the beginning.
• Challenges: Finding a good initial sparse topology is non-trivial, and training dynamics can be less stable than with dense networks.