Pruning-Aware Training

This core technique adds penalty terms to the training loss function to encourage sparsity. Unlike standard L1 or L2 regularization, sparsity-inducing methods are designed to push many weights to exactly zero, creating a naturally pruned network.

• L1 Regularization (Lasso): Adds a penalty proportional to the absolute value of weights, which can drive less important parameters to zero.
• Group Lasso: Applies L1 penalties to groups of parameters (e.g., an entire filter or channel), encouraging structured sparsity for hardware efficiency.
• Sparse Variational Dropout: A Bayesian approach that learns dropout rates for individual weights; high dropout rates effectively prune the weight.

A family of algorithms that maintain a fixed level of sparsity throughout training by dynamically evolving the network's connectivity. The topology is not static; unimportant connections are pruned and new ones are regrown.

• Sparse Evolutionary Training (SET): Initializes a sparse Erdős–Rényi network and periodically removes the smallest-magnitude weights, regrowing new connections to random neurons.
• RigL (Rigged Lottery): A more sophisticated DST method that regrows connections based on the magnitude of gradients, not randomly, leading to faster convergence and better final accuracy.
• DST maintains efficiency as the model is always sparse, reducing memory and compute costs during training itself.

Based on the Lottery Ticket Hypothesis, this approach identifies a sparse, trainable subnetwork (the 'winning ticket') within a dense network. Pruning-aware training via rewinding involves:

1. Train a dense network for a few iterations.
2. Prune a percentage of smallest-magnitude weights.
3. Rewind the remaining weights to their values from an early training iteration (iteration 0 or a later 'rewind point').
4. Continue training the sparse subnetwork from this rewound state. This process often yields sparse networks that match or exceed the accuracy of the original dense model when trained to completion.

A scheduled, iterative technique that integrates pruning directly into the training loop. Instead of pruning once at the end, sparsity is increased gradually according to a predefined schedule.

• Process: Starting from a dense model, after a warmup period, a small percentage of the smallest-magnitude weights are pruned every N training steps or epochs.
• Schedule: The sparsity level follows a function (e.g., cubic, exponential, or cosine) from an initial sparsity (often 0%) to a final target sparsity (e.g., 90%).
• Benefit: The network continuously adapts to its sparser structure, preventing severe, sudden accuracy drops and leading to higher final accuracy for a given sparsity level compared to one-shot pruning.

Focuses on removing entire structural components (filters, channels, attention heads) from the start, ensuring the final model has hardware-friendly sparsity patterns.

• Channel Pruning via Regularization: Applies group sparsity penalties to all weights associated with a specific output channel or filter, encouraging entire features to be pruned.
• Neuron-Level Pruning: Targets entire neurons in fully connected layers by penalizing the L2 norm of incoming weights.
• Hardware-Aware Loss: The training objective can include a latency or FLOPs term, incentivizing the model to learn which structures are most costly and dispensable, co-optimizing for accuracy and inference speed.

For full model compression, pruning-aware training is often combined with Quantization-Aware Training (QAT). This co-design approach produces models optimized for both sparsity and low-precision integer execution.

• Simultaneous Optimization: The model is trained with simulated quantization (fake quantization) and sparsity-inducing penalties simultaneously.
• Phased Approach: A common schedule is: 1) Dense pre-training, 2) Gradual pruning + fine-tuning, 3) QAT fine-tuning of the sparse model.
• Hardware Targeting: This combined methodology is essential for deploying models on edge devices with NPUs that support sparse, low-precision compute kernels, maximizing throughput and energy efficiency.

A pruning schedule is a predefined strategy dictating the timing, rate, and criteria for removing parameters during training. It is the concrete execution plan that implements pruning-aware training principles.

• Key Components: Defines the sparsity ramp function (e.g., cubic, linear), the pruning frequency, and the pruning criterion (e.g., weight magnitude, gradient).
• Example: A schedule might start pruning at epoch 10, increase sparsity from 0% to 80% over 50 epochs using a cubic function, and prune every 100 training steps based on L1 norm.

Iterative Magnitude Pruning (IMP) is a foundational algorithm that repeatedly removes the smallest-magnitude weights and fine-tunes the network. It is a common practical implementation of pruning-aware training schedules.

• Process: Cycles of prune -> train/re-tune are performed until a target sparsity is reached.
• Connection to LTH: This method empirically validates the Lottery Ticket Hypothesis, as it often finds high-performing sparse subnetworks ('winning tickets') within the dense model.

Structured pruning removes entire groups of parameters, such as filters, channels, or blocks, to maintain hardware-efficient sparsity patterns. Pruning-aware training often targets structured sparsity for direct inference speedups.

• Hardware Alignment: Produces sparsity that maps efficiently to vectorized instructions and GPU/TPU/NPU kernels, unlike unstructured pruning.
• Common Targets: Pruning convolutional filters, attention heads in transformers, or full neurons in dense layers.

The Lottery Ticket Hypothesis (LTH) is the theoretical underpinning for many pruning-aware methods. It posits that dense networks contain sparse, trainable subnetworks that can match original performance.

• Core Claim: A 'winning ticket' is a subnetwork that, when trained from its original initialization, achieves comparable accuracy to the full network.
• Impact on Training: Motivates pruning-at-initialization and early-bird ticket strategies, where pruning constraints are applied from the very start of training.

Quantization-Aware Training (QAT) is a parallel methodology to pruning-aware training. It simulates lower numerical precision (e.g., INT8) during training so the model learns to compensate for quantization error.

• Shared Philosophy: Like pruning-aware training, QAT incorporates the deployment constraint (reduced precision) into the training loop.
• Combined Use: Often used in a multi-stage compression pipeline after pruning-aware training to produce a model that is both sparse and quantized.

Automated Model Compression (AMC) is a framework that uses reinforcement learning or search algorithms to automatically determine the optimal pruning policy per layer. It automates the design of pruning-aware schedules.

• Process: An RL agent or controller network learns to predict the sparsity ratio for each layer based on a reward signal (e.g., accuracy vs. FLOPs).
• Advantage: Discovers non-uniform, layer-wise sparsity distributions that are more optimal than hand-designed schedules.