Structured Sparsity

This coarse-grained pattern removes entire channels (for convolutional layers) or neurons (for fully-connected layers) from a network. Pruning a channel equates to removing the corresponding 3D filter and all its associated weights.

• Impact: Dramatically reduces the number of feature maps and the computational cost of subsequent layers.
• Hardware Benefit: Results in genuinely smaller, dense matrices that are efficiently processed by standard dense linear algebra libraries (BLAS).
• Common Technique: Often uses L1-norm of channel weights to determine importance for removal.

Weights are pruned in contiguous blocks (e.g., 4x4, 8x8, 16x1). This pattern trades some flexibility in pruning granularity for greater hardware efficiency.

• Hardware Alignment: Blocks align with the vector processing units (VPUs) or small matrix engines in CPUs and some AI accelerators, enabling single-instruction processing of the entire block.
• Storage Efficiency: Block indices require less metadata overhead compared to fine-grained patterns.
• Use Case: Effective in transformer models for pruning attention heads or MLP blocks in a structured manner.

The coarsest form of structured sparsity, where entire layers of a neural network or attention heads in a transformer model are removed. This directly alters the model's architecture.

• Transformer Example: In a Multi-Head Attention layer, less important heads can be pruned, reducing the dimension of the concatenated head outputs.
• Significance: Leads to direct reductions in model depth or width, impacting both memory and inference latency.
• Analysis Required: Requires careful analysis of layer/head sensitivity, often via saliency metrics, to avoid catastrophic performance drops.

These are sparsity patterns explicitly designed for the self-attention mechanism in transformers to overcome its quadratic complexity. They are fixed, non-trainable patterns applied to the attention mask.

• Sliding Window Attention: Each token only attends to a local window of W preceding tokens (common in autoregressive language modeling).
• Big Bird Pattern: Combines random attention, window attention, and global attention to create a sparse graph that approximates full attention.
• Hardware Implication: These patterns directly sparsify the attention matrix, allowing the use of optimized sparse-dense matrix multiplication kernels during the attention score calculation.

While not a weight pattern, activation sparsity is a critical complementary concept. It exploits the fact that many activation functions (e.g., ReLU) produce zero outputs. When combined with structured weight sparsity, it can lead to dynamic sparsity where entire computation paths are skipped.

• Mechanism: If a weight is zero, its multiplication is skipped. If the preceding activation is also zero, the entire multiply-accumulate operation is redundant.
• Hardware Support: Advanced AI accelerators like Google's TPUs and certain NPUs contain circuitry to detect and skip these zero-based operations dynamically.
• Synergy with Weight Sparsity: The combination can yield super-linear reductions in actual compute cycles beyond what weight sparsity alone predicts.

This technique exploits the idea that weight matrices in neural networks often have a low intrinsic rank. It compresses the model by factorizing large weight matrices into products of smaller ones.

• Singular Value Decomposition (SVD): A classical method to approximate a matrix as a product of three matrices, allowing truncation of small singular values.
• Tensor Decomposition: Extends factorization to higher-dimensional weight tensors.
• Efficiency Gain: Reduces parameters and FLOPs by replacing a large matrix multiplication with two smaller ones. Unlike sparsity, it results in dense, smaller matrices that are universally accelerated.

Knowledge distillation is a form of model compression that transfers knowledge from a large, accurate teacher model to a smaller, more efficient student model. The student is trained not just on ground-truth labels, but to mimic the teacher's softened output probabilities (logits) and sometimes intermediate representations.

• Logits Distillation: The primary method, using a temperature parameter to create a softer probability distribution.
• Feature Distillation: Matching intermediate layer activations between teacher and student. This technique produces a dense, compact model and is orthogonal to sparsity—a distilled student model can subsequently be pruned or quantized.

Sparse training bypasses the traditional 'train large, then prune' pipeline by training a network with a fixed, sparse connectivity pattern from the start.

• Lottery Ticket Hypothesis: Suggests dense networks contain sparse, trainable subnetworks ('winning tickets') that can match original performance.
• Static Sparse Masks: A fixed sparsity pattern (often unstructured) is applied before training begins.
• Dynamic Sparse Training: The sparse connectivity pattern evolves during training. This approach saves the massive compute cost of pre-training a dense model and is a frontier research area for efficient model development.

Mixture of Experts is a conditional computation architecture that achieves sparsity in activation, not weights. For each input, a gating network selects only a small subset of many expert sub-networks to be active.

• Sparse Activation: Only 2-4 experts out of potentially hundreds or thousands are activated per token.
• Massive Parameter Count with Fixed FLOPs: Enables models with trillions of parameters (e.g., GPT-4, Mixtral) while keeping computational cost per token manageable.
• Routing Challenge: Designing efficient, load-balanced gating functions is a key engineering challenge. MoE represents a high-level, systematic form of structured sparsity in the model's execution pathway.