Pruning

Removes individual weights based on their magnitude, setting near-zero values to exactly zero. This produces a sparse weight matrix where most connections are eliminated.

• Typically removes 50-90% of parameters with minimal accuracy loss
• Results in irregular sparsity patterns that require specialized hardware or software for acceleration
• Often requires fine-tuning after pruning to recover accuracy
• Common criterion: prune weights below a percentile threshold of absolute magnitude

Removes entire structural components such as channels, filters, attention heads, or entire layers rather than individual weights.

• Produces a smaller, dense model compatible with standard hardware
• Directly reduces FLOPs and memory footprint without sparse computation overhead
• Example: removing 30% of channels in a convolutional layer yields immediate inference speedup
• More aggressive than unstructured pruning; requires careful layer-wise sensitivity analysis

The most common heuristic: weights with the smallest absolute values are assumed to contribute least to the output and are removed first.

• Simple to implement and computationally cheap
• Works well for unstructured pruning but can be suboptimal for structured pruning
• Iterative magnitude pruning (IMP) repeatedly prunes and retrains, often matching or exceeding the original model's performance
• Limitation: small weights can still be important if they gate critical pathways

Global pruning ranks all weights across the entire network and removes the lowest-magnitude connections regardless of layer, automatically determining per-layer sparsity ratios.

Layer-wise pruning applies a fixed sparsity percentage to each layer independently.

• Global pruning often yields better accuracy for a given compression ratio
• Layer-wise pruning provides predictable per-layer dimensions for hardware optimization
• Some layers (e.g., first and last) are typically pruned less aggressively due to sensitivity

Post-training pruning compresses a fully trained model in one shot, then applies light fine-tuning to recover accuracy. Fast but may leave recoverable accuracy on the table.

Pruning during training (gradual magnitude pruning) progressively increases sparsity throughout the training process, allowing the network to adapt.

• Gradual pruning often achieves higher final accuracy at extreme sparsity levels (>90%)
• Requires more compute but produces more robust sparse subnetworks
• The Lottery Ticket Hypothesis suggests that dense networks contain sparse subnetworks that can be trained from scratch to match full accuracy

An alternative to magnitude-based pruning that scores weights based on their tendency to move away from zero during fine-tuning, rather than their absolute magnitude.

• Particularly effective for transfer learning scenarios with pre-trained models
• Weights that grow in magnitude during fine-tuning are retained; those shrinking toward zero are pruned
• Outperforms magnitude pruning when pruning large language models for specific downstream tasks
• Implemented in libraries like Hugging Face Transformers for sparse BERT and GPT models

Reduces the numerical precision of weights and activations from FP16/FP32 to INT8 or INT4. Unlike pruning, which removes connections, quantization compresses the bit-width of remaining parameters. Post-Training Quantization (PTQ) requires no retraining, while Quantization-Aware Training (QAT) preserves accuracy by simulating quantization during fine-tuning.

• INT8 quantization: 4x memory reduction vs FP32
• INT4 quantization: 8x memory reduction vs FP32
• Often combined with pruning for compound compression

Trains a compact student model to mimic the output distribution of a larger teacher model. Unlike pruning, which surgically removes weights from an existing architecture, distillation creates an entirely new, smaller model. The student learns from both hard labels and the teacher's soft logits, capturing nuanced class relationships.

• Knowledge transfer via softened probability distributions
• Often yields better accuracy than pruning alone
• Common in BERT-to-TinyBERT compression pipelines

Unstructured pruning zeroes out individual weights based on magnitude, producing sparse matrices that require specialized hardware or software for speedups. Structured pruning removes entire neurons, channels, or attention heads, yielding dense sub-networks that run efficiently on standard hardware without custom sparse kernels.

• Unstructured: Higher compression ratio, irregular sparsity
• Structured: Direct latency reduction, hardware-friendly
• Channel pruning: Removes entire convolutional filters

Proposes that randomly initialized dense networks contain sparse subnetworks that can be trained in isolation to match the full model's accuracy. These 'winning tickets' are identified through iterative magnitude pruning and weight resetting. The hypothesis explains why pruning works and guides the search for optimal sparse architectures.

• Iterative pruning identifies trainable sub-networks
• Winning tickets often generalize better than full models
• Challenges the necessity of overparameterization

Modern accelerators increasingly support native sparse computation. NVIDIA Ampere architecture introduced 2:4 structured sparsity, where exactly 2 of every 4 contiguous weights are zero, enabling up to 2x throughput gains. This hardware trend makes unstructured pruning practically viable for production inference.

• 2:4 sparsity: 50% weights pruned, 2x theoretical speedup
• Sparse Tensor Cores accelerate pruned matrix multiplications
• Requires fine-grained pruning patterns during training

The simplest and most widely used pruning criterion: remove weights with the smallest absolute magnitude below a threshold. Can be applied globally across the network or locally per layer. Often combined with gradual pruning schedules that progressively increase sparsity during training to allow the network to adapt.

• One-shot pruning: Remove weights in a single step
• Gradual pruning: Polynomial or cosine decay schedules
• Typically followed by fine-tuning to recover accuracy