Sparsity Pattern

Unstructured sparsity removes individual weights at arbitrary locations throughout the network. This fine-grained pattern achieves the highest theoretical compression but creates irregular memory access patterns.

• Key Characteristic: Irregular, random-like distribution of zeros.
• Hardware Challenge: Standard dense matrix multiplication units (like GEMM kernels) cannot leverage this sparsity for speedup without specialized sparse libraries (e.g., Sparse GPU Kernels).
• Primary Benefit: Maximizes parameter reduction for a given accuracy loss.
• Common Use: Research settings, archival storage of models, or when using custom inference engines with sparse compute support.

Structured sparsity, specifically N:M sparsity, enforces a rule where in every block of M consecutive weights (e.g., along a row), at most N are non-zero. This balances regularity with flexibility.

• Key Characteristic: Regular, predictable pattern at a fine block level (e.g., 2:4 sparsity).
• Hardware Advantage: Directly supported by NVIDIA's Sparse Tensor Cores (Ampere architecture and later), which can skip computations on zeros and deliver up to 2x throughput for matrix operations.
• Design Trade-off: Less aggressive compression than unstructured sparsity for the same N:M ratio, but with guaranteed hardware acceleration.
• Common Use: Production deployment on modern GPUs where inference latency is critical.

Block sparsity prunes contiguous blocks of weights (e.g., 4x4 or 8x8 sub-matrices) instead of individual elements. It is a coarser-grained form of structured sparsity.

• Key Characteristic: Zeros occur in aligned, contiguous blocks within the weight matrix.
• Hardware Advantage: Enables the use of blocked sparse matrix formats (like Blocked Compressed Sparse Row - BSR), which improve cache locality and reduce metadata overhead compared to fully unstructured formats.
• Efficiency: More efficient than unstructured sparsity on CPUs and some accelerators, but may not be as finely tuned as N:M for latest GPUs.
• Common Use: Optimizing inference on a wider range of hardware, including mobile CPUs and older accelerators.

Channel pruning (a form of structured pruning) removes entire output channels from convolutional layers. Filter pruning removes entire convolutional filters. This changes the network's architecture.

• Key Characteristic: Results in a physically smaller, dense model. The sparsity is at the architectural level, not within the weight tensors.
• Hardware Advantage: The resulting model is a standard, smaller dense network. It can run with high efficiency on any hardware using optimized dense kernels without any special sparse software.
• Impact: Directly reduces feature map dimensions and FLOPs count. Requires layer-by-layer sensitivity analysis to decide which channels/filters to remove.
• Common Use: Creating deployable models for edge devices with limited support for sparse operations.

Pattern-based pruning defines and enforces a specific, repeating kernel-level pattern within convolutional weight tensors. For example, a checkerboard pattern or a centered cross pattern for a 3x3 kernel.

• Key Characteristic: Applies a pre-defined, hardware-friendly mask to the weights of convolutional filters.
• Hardware Advantage: The fixed pattern allows for the design of custom, efficient convolution circuits in ASICs/FPGAs that skip the multiplications for the always-zero positions.
• Design Process: Involves co-designing the sparsity pattern with the target hardware's compute architecture.
• Common Use: Highly specialized deployments for custom silicon (e.g., vision processors in smartphones, automotive chips).

This dimension defines how the pattern is determined, orthogonal to its structure.

• Random Patterns: The locations of zeros are randomly selected at initialization. Used in Sparse Training (e.g., RigL algorithm), where a network is trained from scratch with a fixed sparse connectivity.
• Learned Patterns: The pattern emerges from a pruning criterion (e.g., magnitude, gradient movement) applied during or after training. The algorithm discovers which weights are less important.

Key Insight: Unstructured patterns are typically learned. Structured patterns (N:M, Block) can be either enforced randomly or learned via constrained optimization. Learned patterns generally preserve accuracy better than random ones for a given sparsity level.

Unstructured pruning removes individual weights anywhere in a network, creating an irregular, non-predictable sparsity pattern. This fine-grained approach can achieve high theoretical compression ratios but results in a sparse matrix that lacks a regular structure. Efficient execution requires specialized libraries with sparse matrix multiplication kernels or hardware support for gather-scatter operations, as standard dense linear algebra libraries see no benefit. It is often used with magnitude-based pruning criteria.

Structured pruning removes entire structural components like filters, channels, or blocks. A prominent hardware-friendly pattern is N:M sparsity, where in every block of M consecutive weights (e.g., 4), at most N (e.g., 2) are non-zero. This creates a predictable, regular pattern that can be exploited by sparse tensor cores in modern GPUs (like NVIDIA's Ampere architecture) for 2x speedups in matrix operations. The pattern is enforced during training or via a post-training algorithm, balancing compression and efficient execution.

A pruning mask is a binary matrix (of 1s and 0s) that has the exact same shape as a model's weight tensor. It defines the sparsity pattern by indicating which parameters are active (1) and which are pruned (0).

• Application: The mask is applied via element-wise multiplication with the weight values during the forward pass: W_effective = W * Mask.
• Storage: The mask must be stored alongside the non-zero weights. For structured patterns, it can be compressed efficiently.
• Dynamic Masking: In pruning-aware training, the mask can be updated periodically based on a pruning criterion like weight magnitude.

This approach co-designs the sparsity pattern with the target deployment hardware's execution engine. The goal is to maximize actual (not just theoretical) speedup and energy efficiency. Key considerations:

• Memory Access Patterns: Aligning sparsity to minimize cache misses and enable contiguous memory reads.
• Supported Operations: Targeting patterns like N:M that match the processor's sparse compute units (e.g., NVIDIA's 2:4 pattern).
• Kernel Support: Ensuring the inference runtime (e.g., TensorRT, TFLite) has optimized kernels for the chosen pattern. This moves beyond algorithmic sparsity to deployable efficiency.

This term encompasses the runtime systems and optimizations required to execute a model with a defined sparsity pattern efficiently. It addresses the gap between having sparse weights and achieving faster inference.

• Kernel Libraries: Using libraries like cuSPARSELt (NVIDIA) or specialized TFLite delegates that skip multiplications with zeros.
• Compressed Storage Formats: Storing weights in formats like CSR (Compressed Sparse Row) or CSC (Compressed Sparse Column) to save memory and bandwidth.
• Activation Sparsity: Exploiting the fact that ReLU activations also produce zeros, leading to dynamic sparsity in activations that can be leveraged during computation.

Granularity defines the smallest unit or block that is pruned as a single entity, directly influencing the sparsity pattern's regularity and hardware efficiency.

• Fine-Grained: Individual weights (unstructured). Highest flexibility, lowest hardware efficiency.
• Vector-Grained: Rows or columns of a matrix.
• Kernel-Grained: Entire convolutional kernels or filters (structured).
• Channel-Grained: Entire input or output channels (structured).
• Layer-Grained: Removing whole layers. Coarser granularities like kernel or channel pruning produce structured patterns that are easier to accelerate but may offer less compression for the same accuracy drop.