Pruning Granularity

Removes individual weights (parameters) anywhere in the network, creating an irregular, sparse pattern of zeros. This offers the highest theoretical compression, as it targets the most granular unit.

• Key Characteristic: Maximizes parameter count reduction but results in a sparse model.
• Hardware Challenge: The irregular memory access pattern does not align with standard dense matrix multiplication units (GEMM) on GPUs, often requiring specialized sparse kernels or hardware (e.g., NVIDIA's Sparse Tensor Cores) to realize speedups.
• Example: Pruning 70% of the smallest-magnitude weights in a fully-connected layer, leaving a random-looking pattern of zeros.

A semi-structured approach that enforces a regular sparsity pattern within small, fixed blocks of weights (e.g., within a 1x4 or 1x8 vector). This balances flexibility with hardware efficiency.

• Key Characteristic: Defines a predictable pattern, like keeping 2 non-zero values in every block of 4 weights (2:4 sparsity).
• Hardware Advantage: The regular pattern allows for efficient sparse matrix multiplication on modern architectures. NVIDIA's Ampere and later GPUs have dedicated hardware support for 2:4 sparsity, enabling near-dense performance at 50% weight reduction.
• Example: Enforcing a 2:4 sparsity pattern across all weight matrices, which can be efficiently executed via the torch.sparse library with CUDA backend support.

Removes entire, structurally coherent groups of weights, such as filters, channels, or attention heads. This results in a smaller, dense model.

• Key Characteristic: Directly reduces the model's width or depth, shrinking the actual architecture.
• Hardware Advantage: The resulting model is dense and can be run with standard, highly optimized libraries (e.g., cuDNN, TensorRT) without any special software, guaranteeing speedups on all hardware.
• Examples:
  • Channel Pruning: Removing 32 of the 64 output channels from a convolutional layer.
  • Attention Head Pruning: Removing 4 of the 12 heads in a transformer's multi-head attention block.

The coarsest form of pruning, which removes entire layers or residual blocks from a deep network. This directly reduces model depth and latency.

• Key Characteristic: Highly aggressive, targeting redundant or less critical macro-structures within the network.
• Application: Often guided by pruning sensitivity analysis to identify which layers contribute least to the final output. Common in compressing very deep models like ResNet-152 or large transformer stacks.
• Challenge: Risk of severe pruning-induced accuracy drop if critical blocks are removed, as it alters the fundamental information flow. Requires careful fine-tuning or rewinding to recover performance.

The choice of granularity dictates the compression-efficiency Pareto frontier.

• Fine-Grained Pruning: Achieves the highest sparsity (e.g., >90% zeros) but requires specialized software/hardware for acceleration. Speedup is not guaranteed on standard GPUs.
• Structured Pruning: Achieves lower sparsity (e.g., 30-50% parameter reduction) but yields reliable, immediate speedups on all hardware because it produces a smaller dense model.
• Engineering Implication: For deployment on general-purpose GPUs or edge devices, structured pruning is often preferred. For research pushing compression limits or targeting specialized inference chips (ASICs), fine-grained methods are explored.

The pruning criterion and schedule are intrinsically linked to the chosen granularity.

• Criterion:
  • Fine-grained: Uses weight magnitude (L1 norm) or gradient-based scores (Movement Pruning).
  • Structured: Uses filter norm, channel activation statistics, or Taylor expansion to score entire structures.
• Schedule:
  • One-shot pruning removes a target percentage of weights/structures in a single step, often used for coarse-grained pruning.
  • Iterative pruning (e.g., Iterative Magnitude Pruning) cycles between pruning and fine-tuning, which is crucial for recovering accuracy after fine-grained removal.
• Advanced Technique: Pruning at initialization (e.g., SNIP) scores parameters before training to determine a sparse subnetwork, which is inherently tied to fine-grained or pattern-based granularity.

Unstructured pruning removes individual weights based on an importance criterion like magnitude, creating a sparse model with an irregular pattern of zeros. This fine-grained granularity offers maximum flexibility to preserve critical connections, often achieving higher sparsity rates for a given accuracy budget. However, the irregular sparsity does not map efficiently to standard hardware, requiring specialized libraries (e.g., Sparse GPU Kernels) or hardware (e.g., NVIDIA's Sparse Tensor Cores) to realize speedups.

• Example: Pruning 90% of the smallest weights in a fully-connected layer, leaving a scattered pattern of non-zero values.

Structured pruning removes entire, structurally coherent groups of weights—such as filters, channels, or attention heads. This coarse-grained granularity results in a smaller, dense model that maintains hardware-friendly execution patterns, enabling immediate speedups on standard GPUs and CPUs without specialized runtimes. The trade-off is reduced flexibility, often requiring more aggressive retraining to recover accuracy after removing these larger structural units.

• Common Targets: Pruning entire convolutional filters, neurons in a dense layer, or heads in a multi-head attention mechanism.

N:M Sparsity is a semi-structured sparsity pattern that balances fine- and coarse-grained benefits. In this pattern, for every block of M consecutive weights (e.g., within a single vector), at most N are non-zero. This block-level granularity provides enough structure for efficient execution on modern hardware (like NVIDIA's Ampere architecture with 2:4 sparsity support) while offering more flexibility than removing entire rows or columns. It represents a hardware-aware design point in the granularity spectrum.

The sparsity pattern is the specific, deterministic map of which weights in a neural network are zero-valued after pruning. It is the direct output of applying a pruning algorithm at a chosen granularity. This pattern dictates:

• Memory Layout: How weights are stored (dense with zeros vs. compressed formats like CSR).
• Computational Kernels: Whether standard dense matrix multiplication or specialized sparse kernels must be used.
• Transferability: Whether the pattern is tied to a specific hardware configuration for optimal performance.

Channel pruning is a prevalent form of structured pruning applied to convolutional neural networks (CNNs). It removes entire feature map channels (also called filters) from a convolutional layer. This granularity is particularly effective because:

• It reduces tensor dimensions: Removing a channel in layer L reduces the number of input channels for layer L+1, creating a cascading reduction in FLOPs and parameters.
• It enables immediate speedup: The resulting model is a standard, smaller CNN executable on any hardware.
• Criterion is key: Channels are typically scored by importance using metrics like L1-norm of filter weights, average percentage of zeros in activations, or gradient-based measures.

Attention head pruning is a structured pruning technique specific to Transformer models. It removes entire multi-head attention units, a natural coarse-grained granularity given the modular architecture of Transformers. Pruning heads reduces the computational complexity of the quadratic self-attention operation. Research indicates that attention heads are often redundant, and a significant portion can be removed with minimal impact on language modeling performance, making this a powerful method for optimizing models like BERT or GPT for inference.