Pruning Granularity

Removes individual weights anywhere in the network, creating an irregular sparsity pattern. This offers the highest potential compression but requires specialized sparse linear algebra libraries (e.g., cuSPARSELt) or hardware support for efficient inference, as standard dense matrix multiplication cannot leverage the sparsity.

• Example: Pruning 70% of weights in a ResNet-50, resulting in a pattern where zeros are scattered randomly.
• Primary Use: Maximum model size reduction in research or for deployment on hardware with dedicated sparse compute units.

Removes entire structural components like neurons, channels, or filters. This maintains dense matrix operations, ensuring compatibility with standard hardware and deep learning frameworks without specialized kernels.

• Neuron Pruning: Removes an entire unit from a fully connected layer.
• Channel/Filter Pruning: Removes an output channel from a convolutional layer, reducing the width of the resulting feature map and the compute in the next layer.
• Primary Use: Production deployment where ease of deployment and guaranteed speedup on commodity hardware (CPUs, GPUs) are critical.

Operates at an intermediate granularity, enforcing sparsity patterns within defined blocks of the weight matrix. N:M sparsity is a prominent example, where in every block of M consecutive weights, at most N are non-zero.

• Example: 2:4 sparsity, a common pattern for NVIDIA Ampere GPU Tensor Cores, where 2 out of every 4 weights are non-zero.
• This approach balances high compression ratios with hardware efficiency, as the regular pattern can be encoded and executed efficiently by specialized sparse tensor cores.

The coarsest form of pruning, which removes entire layers or residual blocks from a deep network. This is a highly aggressive technique that significantly alters the model's architecture.

• Often guided by layer sensitivity analysis to identify the least critical sections.
• Can lead to substantial reductions in FLOPs and latency but risks severe accuracy degradation if not done carefully.
• Primary Use: Drastic latency reduction for real-time applications on edge devices, or for creating specialized sub-architectures from a larger parent model.

The choice of granularity is fundamentally a hardware co-design decision. Fine-grained sparsity offers high theoretical compression but may yield no speedup—or even slowdown—on hardware optimized for dense math. Coarse-grained sparsity provides predictable, direct speedups by reducing the dimensions of dense operations.

• Dense Hardware (CPUs/GPUs): Benefit from structured pruning.
• Sparse Accelerators (e.g., Cerebras, certain NPUs): Can exploit fine-grained sparsity.
• Modern Sparse Tensor Cores (e.g., NVIDIA): Require specific semi-structured patterns like N:M sparsity.

Selection depends on the deployment target and performance goals. Key considerations:

• Target Hardware: Does the deployment chip have sparse compute support?
• Toolchain Support: Do your inference frameworks (TensorFlow Lite, ONNX Runtime) support the sparsity pattern?
• Accuracy Budget: How much performance degradation is acceptable?
• Compression Goal: Is the aim to reduce model size, inference latency, or both? A practical workflow often starts with structured pruning for broad compatibility, then explores finer granularities if supported by the hardware stack.