Structured Pruning

Structured pruning enforces a regular sparsity pattern by removing entire structural units, such as filters, channels, or neurons. This results in smaller, dense weight matrices that maintain contiguous memory access and can be executed efficiently on standard hardware (CPUs, GPUs, NPUs) using optimized BLAS libraries and matrix multiplication kernels, without requiring specialized sparse compute units.

The technique operates at a coarse granularity, targeting groups of parameters rather than individual weights. Common granularities include:

• Filter Pruning: Removing entire 3D convolutional filters.
• Channel Pruning: Removing input or output channels from layers.
• Neuron Pruning: Removing entire neurons from fully connected layers. This group-level removal directly shrinks the dimensions of the weight tensors and the resulting activation maps.

Because entire structural elements are removed, structured pruning provides predictable and significant reductions in both model size (number of parameters) and computational cost (FLOPs). Removing a filter from a convolutional layer, for example, reduces the layer's output channels, which subsequently reduces the input channels for the next layer, creating a compounding effect on overall compute.

Pruning is inherently destructive. After removing structural components, the model's accuracy typically drops. Fine-tuning (or retraining) the pruned network on the original training data for a limited number of epochs is essential to recover lost performance. The network learns to compensate for the removed capacity with its remaining parameters.

This is the key differentiator. Unstructured pruning removes individual weights anywhere, creating irregular, fine-grained sparsity. While it can achieve higher sparsity rates, it requires specialized software/hardware for speedup. Structured pruning sacrifices some potential compression for guaranteed speedups on commodity hardware due to its regular, coarse-grained pattern.

Determining which filters or channels to prune requires a scoring criterion. Common heuristics include:

• Magnitude-based: Prune filters with the smallest L1/L2 norm.
• Activation-based: Prune channels with the lowest average activation.
• Gradient-based: Use gradient information (e.g., Movement Pruning) to estimate sensitivity.
• Reconstruction error: Minimize the error in the next layer's output after pruning.

Filter pruning removes entire convolutional filters from a layer. Each filter corresponds to an output channel, so pruning a filter reduces the number of output feature maps. This creates a narrower, computationally cheaper layer and reduces the input channels for all subsequent layers that depend on its output.

• Key Impact: Directly reduces FLOPs and parameters in 2D and 3D convolutions.
• Criterion: Filters are often ranked by L1/L2 norm of their weights, average activation magnitude, or contribution to the final loss.
• Hardware Benefit: Results in dense, smaller weight matrices ideal for standard GEMM (General Matrix Multiply) libraries on CPUs, GPUs, and NPUs.

Channel pruning removes input channels from a convolutional layer. Since an input channel is a feature map from the previous layer, this technique requires aligning the pruning of a layer's input with the previous layer's output. It effectively reduces the width of the network's feature representation at specific points.

• Dependency: Pruning an input channel to Layer N requires removing the corresponding output filter from Layer N-1.
• Criterion: Channels are scored based on the scaling factors in Batch Normalization layers, activation sparsity, or using reconstruction error minimization.
• Example: In a ResNet block, pruning channels from the first convolution allows for pruning corresponding channels in the shortcut connection and subsequent convolutions.

Neuron pruning removes entire neurons (units) from fully connected (dense) layers. This is the structured analog to unstructured weight pruning for linear layers. Pruning a neuron eliminates its incoming and outgoing connections, reducing the layer's width.

• Granularity: Coarser than individual weight pruning but finer than removing an entire layer.
• Criterion: Often based on the L2 norm of a neuron's incoming weight vector or its average activation magnitude across a dataset.
• Result: Produces a smaller, dense weight matrix for the layer, leading to faster matrix-vector products during inference.

N:M sparsity enforces a specific, hardware-friendly pattern where in every block of M consecutive weights, at most N are non-zero. This is a fine-grained but structured technique that maintains a dense tensor format while containing predictable zeros.

• Pattern: Common patterns are 2:4 or 1:4 sparsity (e.g., 2 non-zeros in every block of 4).
• Hardware Support: Explicitly supported by NVIDIA's Ampere/Ada/Hopper GPU Sparse Tensor Cores and other NPUs, which can skip zero multiplications and achieve near-dense throughput.
• Application: Applied to the weight matrices of fully connected layers or the kernels of convolutional layers. Requires specialized training or post-training algorithms to find high-accuracy N:M patterns.

Layer pruning removes entire layers from a deep neural network. This is the coarsest form of structured pruning and is most applicable to architectures with significant redundancy, such as very deep CNNs or Transformer models with many similar blocks.

• Impact: Dramatically reduces model depth and latency, but risks removing critical non-linear transformations.
• Criterion: Layers are ranked by their contribution to the output (e.g., via layer-wise relevance propagation) or by evaluating the performance drop when a layer is removed.
• Use Case: Pruning repetitive blocks in ResNets or BERT-style encoders where adjacent layers may learn similar features.

Block pruning removes predefined contiguous blocks of parameters from weight matrices or convolutional kernels. This generalizes filter/channel pruning to arbitrary, multi-dimensional blocks.

• Pattern: Prunes blocks of size [k_h, k_w, C_in, C_out] in conv layers or [Fan_in, Fan_out] blocks in linear layers.
• Hardware Alignment: Block shapes can be chosen to align with hardware vector processing units (VPUs) or memory burst sizes for efficient load/store operations.
• Trade-off: Offers a flexible balance between the high compression of unstructured pruning and the runtime efficiency of filter pruning.

Unstructured pruning removes individual weights anywhere in a network, creating an irregular, non-zero sparsity pattern. This offers the highest theoretical compression but requires specialized sparse libraries or hardware (e.g., sparse tensor cores) for efficient inference, as standard dense matrix multiplication cannot leverage the sparsity.

• Key Contrast: Unlike structured pruning, it does not maintain regular matrix shapes.
• Primary Challenge: The irregular memory access patterns often limit actual speedup on general-purpose hardware.

This is a foundational pruning criterion, not a structural type. It operates on the heuristic that parameters with the smallest absolute magnitude contribute least to the model's output. It can be applied for both unstructured and structured pruning.

• Process: Weights are ranked globally or locally by their absolute value, and the smallest are set to zero.
• Common Schedule: Often used with Gradual Magnitude Pruning (GMP), which slowly increases sparsity during training to allow the network to adapt.

These are the most common granularities for structured pruning in convolutional neural networks (CNNs). They remove entire structural components to maintain dense, hardware-friendly operations.

• Channel Pruning: Removes output channels from a convolutional layer, reducing the width of the resulting feature map and the input channels to the next layer.
• Filter Pruning: Removes entire convolutional filters (kernels), which is functionally equivalent to channel pruning on the output side. This directly reduces FLOPs and kernel parameters.

A semi-structured sparsity pattern that bridges unstructured and fully structured pruning. In every block of M consecutive weights (e.g., along a row or channel), at most N are non-zero (e.g., 2:4 sparsity).

• Hardware Advantage: This pattern is directly supported by sparse tensor cores in modern NVIDIA GPUs (Ampere+), enabling predictable speedups without custom kernels.
• Balance: It offers more fine-grained pruning than channel removal but provides guaranteed, efficient execution patterns.

A training paradigm where the model is optimized with the knowledge it will be pruned. This contrasts with post-training pruning, where a model is pruned after standard training, often leading to greater accuracy loss.

• Methods: Includes techniques like adding sparsity-inducing regularization (e.g., L1 penalty) or directly applying a pruning schedule during training.
• Goal: To encourage the network to learn representations that are robust to the removal of parameters, yielding a better sparsity-accuracy tradeoff.

An overarching design philosophy where the pruning strategy is co-designed with the target deployment hardware's characteristics. The goal is to maximize real-world latency, throughput, or energy efficiency gains, not just theoretical parameter count reduction.

• Considers: Memory bandwidth, cache sizes, supported matrix shapes (e.g., 4x4 tiles), and availability of sparse compute units.
• Example: Choosing filter pruning for a mobile CPU because it directly reduces dense GEMM operations, versus targeting N:M sparsity for an NVIDIA GPU with sparse tensor cores.