Structured Pruning

Unlike unstructured pruning, which creates irregular sparsity, structured pruning removes entire structural units. This results in a smaller, dense model that can be executed efficiently on standard hardware (CPUs, GPUs) without requiring specialized sparse matrix multiplication libraries. The output model has a conventional, dense architecture, enabling immediate deployment with existing inference engines like ONNX Runtime or TensorRT.

The pruning granularity defines the smallest removable unit. Key granularities include:

• Filter/2D Kernel Pruning: Removes entire convolutional filters, reducing the number of output feature maps.
• Channel Pruning: Removes input channels to a convolutional layer, reducing the width of the network and the computational cost for subsequent layers.
• Attention Head Pruning: In transformers, removes entire multi-head attention units.
• Layer/Block Pruning: The coarsest granularity, removing entire layers or residual blocks.

The algorithm for selecting which structures to remove is defined by the pruning criterion. Common criteria include:

• Magnitude-based: Removes filters with the smallest L1/L2 norm.
• Activation-based: Removes structures that contribute least to the average activation.
• Gradient-based: Uses gradient flow to estimate importance.

The pruning schedule dictates the process: one-shot pruning removes a target percentage of weights at once, while iterative pruning cycles between pruning small amounts and fine-tuning to recover accuracy.

Pruning induces an accuracy drop. To recover performance, sparse fine-tuning retrains the pruned model with the sparsity pattern fixed. Advanced techniques like rewinding reset weights to an earlier training checkpoint before fine-tuning. The amount of recovery data and fine-tuning epochs required is a key trade-off between final accuracy and the cost of the pruning pipeline.

This is the fundamental trade-off in pruning strategy.

• Structured Pruning: Removes groups (filters, channels). Pros: Hardware-efficient, no specialized kernels needed. Cons: May remove more potentially useful parameters, leading to a higher accuracy drop for a given parameter count reduction.
• Unstructured Pruning: Removes individual weights. Pros: Finer-grained, can achieve higher sparsity with less accuracy loss. Cons: Results in irregular sparsity that requires specialized software/hardware (e.g., NVIDIA's sparse tensor cores) for speedups.

A hybrid approach that balances fine-grained control with hardware efficiency. N:M sparsity enforces a pattern where in every block of M consecutive weights (e.g., within a single vector), at most N are non-zero. This semi-structured pattern is directly supported by the sparse tensor cores in NVIDIA's Ampere and Hopper architectures, allowing for significant inference speedups (e.g., 2x) on compatible hardware without changing the model's dense architecture.

Layer pruning removes entire blocks or layers from a deep neural network, such as a residual block in a ResNet or an entire transformer encoder/decoder layer.

• Coarse-Grained: This is the most aggressive form of structured pruning, directly reducing the network's depth.
• Feasibility: Works well in architectures with redundancy, like very deep ResNets (where some blocks contribute minimally) or over-parameterized transformers.
• Challenge: Removing a layer breaks the network's topological structure. Solutions include:
  • Identity Mapping: Replacing a to-be-removed residual block with a simple identity shortcut.
  • Layer Collapsing: Merging the parameters of consecutive linear layers.
• Use Case: Extremely effective for creating shallower, faster models for deployment on latency-critical edge devices.

Unstructured pruning removes individual weights based on an importance criterion, creating a sparse model with an irregular pattern of zeros. Unlike structured pruning, it does not maintain hardware-friendly structures.

• Key Characteristic: Results in a highly sparse, irregular weight matrix.
• Hardware Requirement: Requires specialized software libraries (e.g., cuSPARSE) or hardware support (e.g., sparse tensor cores) for efficient computation, as standard dense linear algebra is inefficient.
• Typical Criterion: Weight magnitude (L1 norm) is a common heuristic for identifying 'unimportant' weights to zero out.

A pruning criterion is the metric or heuristic used to determine which weights or structural components are least important and can be removed from a network.

• Common Criteria:
  • Magnitude-based: L1 or L2 norm of weights. Simple and effective.
  • Gradient-based: Sensitivity of the loss function to a weight's removal (e.g., used in Movement Pruning).
  • Activation-based: Statistics like the average percentage of zeros in a filter's output.
• Role in Structured Pruning: The criterion is applied at a group level (e.g., sum of magnitudes for an entire filter) to rank and remove entire structural units.

A sparsity pattern defines the specific locations of zero-valued weights within a pruned neural network. This pattern dictates the model's memory layout and computational requirements during inference.

• Structured vs. Unstructured:
  • Unstructured: Irregular, random-like pattern. High theoretical sparsity but poor hardware utilization.
  • Structured: Regular, block-wise patterns (e.g., entire filters/channels are zero). Enables dense computation on pruned model.
• N:M Sparsity: A specific, hardware-friendly structured pattern where for every block of M consecutive weights (e.g., 4), at most N (e.g., 2) are non-zero. This is supported natively on NVIDIA Ampere GPUs and later.

A pruning schedule defines the rate, frequency, and granularity at which weights are removed from a network during training or fine-tuning.

• Common Schedules:
  • One-shot: Prune the model to target sparsity in a single step, often followed by fine-tuning.
  • Iterative: Cyclically prune a small percentage of weights (e.g., 20%) and then fine-tune. Repeats until target sparsity is reached. This is the basis of Iterative Magnitude Pruning (IMP).
  • Gradual: Smoothly increases sparsity according to a function (e.g., cubic) over the course of training.
• Objective: To allow the network to adapt gradually to parameter removal, minimizing the pruning-induced accuracy drop.

Sparse fine-tuning is the process of retraining a pruned neural network on a task-specific dataset to recover the accuracy lost during the pruning process. The sparsity pattern is typically held fixed.

• Purpose: Mitigates the pruning-induced accuracy drop. The remaining weights are adjusted to compensate for the removed parameters.
• Connection to Rewinding: A related technique, rewinding, resets the network's weights to an earlier checkpoint in the original training (e.g., epoch 5) before beginning sparse fine-tuning, often leading to better recovery than fine-tuning from the final trained weights.
• Critical Step: Essential for most pruning methods to produce a usable, accurate model for deployment.

Pruning for inference optimizes a neural network specifically for the deployment phase, focusing on reducing latency, memory bandwidth, and energy consumption on target hardware.

• Core Motivation: The computational graph used for inference is static. Pruning creates a permanently smaller, faster model.
• Hardware-Aware Design: Effective pruning must consider the target hardware's capabilities. Structured pruning (e.g., filter removal) is favored because it results in a dense, smaller model that runs efficiently on standard GPUs/CPUs without specialized kernels.
• End-to-End Goal: Part of a compression pipeline often combined with quantization to maximize inference speed and minimize model size for edge deployment or cost-reduced cloud serving.