Sparse Training

Sparse training is defined by its fixed sparsity mask, which remains constant throughout training. This contrasts with:

• Dynamic sparsity: Where connections are pruned and regrown iteratively (e.g., RigL).
• Post-training pruning: Where a dense model is trained first, then compressed. The static approach provides predictable memory and compute footprints, enabling precise hardware optimization from day one.

The initial sparse topology is critical. Common methods include:

• Random Erdos-Rényi (ER): Each connection exists with a fixed probability p, creating a uniform random graph.
• Gradient-based Saliency: Initialize based on the magnitude of gradients from a few preliminary dense training steps.
• Structured Patterns: Enforcing sparsity in hardware-friendly blocks (e.g., pruning entire channels or 2:4 fine-grained patterns). Poor initialization can trap the network in a bad local minimum, preventing effective learning.

A binary sparsity mask M (0 or 1) is applied element-wise to the weight tensor W before every computation: W_sparse = M ⊙ W During the forward pass, only unmasked weights participate in activations. During the backward pass, gradients are computed only for unmasked weights. The mask is non-differentiable and does not receive gradients itself; it acts as a persistent gate. This requires specialized sparse linear algebra kernels for efficiency.

Sparse training is empirically supported by the Lottery Ticket Hypothesis. It posits that dense, randomly-initialized networks contain sparse winning tickets—subnetworks that, when trained in isolation, can match the original network's accuracy. Sparse training aims to initialize one of these tickets from the start, avoiding the cost of finding it within a dense model. This validates the feasibility of training sparse networks from scratch.

To realize speedups, sparse training requires:

• Sparse Tensor Cores: Modern GPUs (e.g., NVIDIA Ampere+ with sparse tensor cores) accelerate 2:4 fine-grained structured sparsity, delivering up to 2x theoretical throughput.
• Sparse Linear Algebra Libraries: Frameworks like cuSPARSELt provide kernels for sparse matrix-dense matrix multiplication (SpMM).
• Framework Support: Deep learning frameworks (PyTorch, TensorFlow) offer APIs for applying and managing sparse masks, though full training pipeline optimization remains complex.

The primary challenge is the accuracy-efficiency trade-off.

• Efficiency Gains: Fixed sparsity (e.g., 90%) reduces FLOPs, memory footprint, and energy consumption proportionally.
• Accuracy Cost: There is typically an accuracy drop compared to a dense model of equivalent parameter count, as the sparse network has reduced representational capacity. The goal of advanced algorithms is to minimize this gap. For example, a 90% sparse ResNet-50 might achieve within 1-2% accuracy of its dense counterpart.

Pruning is a post-training compression technique that removes less important weights, neurons, or entire layers from a dense neural network. Unlike sparse training, it follows a 'train, prune, fine-tune' cycle.

• Goal: Reduce model size and accelerate inference.
• Methods: Includes magnitude-based pruning (removing small weights) and structured pruning (removing entire channels).
• Relation to Sparse Training: Sparse training aims to create a sparse network during training, avoiding the expensive pruning cycle altogether.

Quantization reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This directly decreases memory footprint and can accelerate computation on supporting hardware.

• Types: Post-training quantization (PTQ) and quantization-aware training (QAT).
• Impact: Can reduce model size by 4x with minimal accuracy loss.
• Synergy: Often combined with pruning and sparse training in a full compression pipeline to achieve extreme efficiency.

Structured sparsity is a pruning paradigm where weights are removed in contiguous, hardware-friendly blocks (e.g., entire 2x2 blocks, or 2 non-zero values per 4-element vector). This contrasts with unstructured sparsity, where individual zeros are scattered.

• Hardware Advantage: Enables practical speedups on GPUs and NPUs with dedicated sparse tensor cores.
• Trade-off: May be less flexible than unstructured sparsity, potentially removing more important weights to meet the structural pattern.
• Application: A target pattern for advanced sparse training algorithms.

Mixture of Experts (MoE) is a conditional computation architecture. A routing network dynamically selects a small subset of specialized 'expert' sub-networks (e.g., 2 out of 128) to process each input.

• Sparsity Analogy: Introduces sparsity in activation, not just weights. Most of the massive model parameters are inactive for any given input.
• Benefit: Enables vastly larger model capacity (trillions of parameters) without a proportional increase in computation per token.
• Example: Used in models like Mixtral 8x7B and GPT-4.

Deep compression is a holistic, three-stage pipeline for extreme neural network compression, famously outlined by Song Han et al. It is a sequential application of techniques, not a single algorithm.

• Pipeline Stages: 1) Pruning to create sparsity. 2) Quantization to reduce weight precision. 3) Huffman coding (entropy coding) to compress the quantized weights.
• Result: Can reduce model size by 35x to 49x. Sparse training can be seen as an alternative to the first stage of this pipeline.

A Sparse Transformer is a variant of the Transformer architecture that uses a fixed or learned sparse attention pattern to reduce the quadratic O(n²) complexity of self-attention.

• Mechanism: Instead of attending to all previous tokens, each token attends to a fixed subset (e.g., a local window + strided pattern).
• Benefit: Enables processing of extremely long sequences (tens of thousands of tokens) that are infeasible for dense transformers.
• Connection: Represents sparsity applied at the algorithmic level (attention pattern) rather than the parameter level (weights).