Pruning Schedule

The pruning rate defines the percentage or absolute number of parameters removed at each scheduled pruning step. It is the most direct control over the speed of sparsification.

• Aggressive Schedules use a high initial rate for rapid compression but risk significant pruning-induced accuracy drop.
• Conservative Schedules use a low, gradual rate, allowing more time for the network to adapt and recover accuracy during sparse fine-tuning.
• The rate can be constant, decay exponentially, or follow a polynomial schedule, often informed by pruning sensitivity analysis per layer.

Pruning frequency determines when parameters are removed, defined by the interval between pruning steps during training or fine-tuning.

• One-shot pruning applies removal once, typically after training is complete (post-training pruning).
• Iterative pruning, such as in Iterative Magnitude Pruning (IMP), alternates cycles of pruning and retraining. Frequency here is defined by epochs or training steps.
• Higher frequency (smaller intervals) with tiny pruning rates mimics pruning-aware training, where the network continuously adapts to sparsity.

Pruning granularity specifies the smallest unit of removal, fundamentally shaping the resulting sparsity pattern and hardware efficiency.

• Fine-grained (Unstructured): Removes individual weights. Maximizes parameter reduction but creates irregular sparsity, requiring specialized libraries for efficient sparse matrix multiplication.
• Coarse-grained (Structured): Removes groups like entire filters (channel pruning) or attention heads. Results in a smaller, dense model that runs efficiently on standard hardware but offers less theoretical compression.
• Semi-structured (e.g., N:M Sparsity): A hardware-friendly compromise, enforcing patterns like 2:4 sparsity within small blocks.

The pruning criterion is the heuristic used to score and select parameters for removal at each step of the schedule.

• Magnitude-based (e.g., L1 Norm): Removes weights with the smallest absolute values. Simple and foundational to algorithms like IMP.
• Gradient-based (e.g., Movement Pruning): Removes weights that change the least during training, using gradient information to identify inactivity.
• Saliency-based (e.g., SNIP): Estimates a connection's effect on the loss function, often used for pruning at initialization. The chosen criterion directly interacts with the schedule's rate and frequency.

The recovery mechanism is the process for regaining accuracy after pruning damage. It is an integral, scheduled part of the overall sparsification process.

• Sparse Fine-Tuning: The standard method, where the pruned network continues training (often with a reduced learning rate) with the sparsity mask fixed.
• Rewinding: A specific technique where weights are reset to values from an earlier training checkpoint (the 'rewind point') before fine-tuning, often improving recovery in IMP.
• The duration and hyperparameters (learning rate, optimizer) of the recovery phase are critical schedule components.

The termination condition defines the stopping point for the pruning schedule, finalizing the model's sparsity level.

• Target Sparsity: The most common condition, a pre-defined global percentage of zero weights (e.g., 90% sparse).
• Accuracy Budget: Pruning stops when validation accuracy drops below a specified threshold relative to the dense baseline.
• Hardware Constraint: Pruning continues until the model fits a specific memory footprint or meets a latency target on deployment hardware, crucial for pruning for inference. The condition ensures the process aligns with end goals of compression or acceleration.

The fundamental choice is between removing weights in a single pass or gradually over time.

• One-Shot Pruning: All target weights are removed at once, after training or at initialization (e.g., SNIP). It's simple and fast but often leads to a larger pruning-induced accuracy drop.
• Iterative Pruning: Small percentages of weights are removed over multiple cycles, each followed by a sparse fine-tuning phase (e.g., Iterative Magnitude Pruning). This allows the network to adapt gradually, typically preserving more accuracy at high sparsity levels but requiring significantly more compute and time.

This defines how the target sparsity level increases over the pruning process. A common approach is a cubic or exponential ramp.

• Early Slow, Late Fast: Starting with a gentle ramp allows the network to adapt to initial sparsity. A steeper ramp later aggressively prunes once the model is more resilient.
• Linear Ramp: A simple, constant increase in sparsity per step. While easy to implement, it may not be optimal for accuracy recovery.
• The schedule must be co-designed with the fine-tuning budget (e.g., number of epochs per iteration) to ensure the model has sufficient time to recover.

The schedule must align with the chosen pruning granularity.

• Unstructured Pruning: Schedules often prune a global percentage of all weights, ignoring layer differences. More advanced schedules perform pruning sensitivity analysis to apply layer-specific rates.
• Structured Pruning (e.g., channel pruning, attention head pruning): The schedule must account for the structural impact. Removing an entire filter may require a slower ramp or a rewinding step to reset adjacent layers. Schedules for N:M sparsity are often one-shot, applied post-training to meet hardware constraints.

A schedule can be independent of, or deeply integrated with, the core training loop.

• Post-Training Pruning: The schedule is a separate phase after standard training is complete. Simpler but offers limited recovery potential.
• Pruning-Aware Training: The schedule is woven into training from the start. Techniques like dynamic network surgery or movement pruning continuously evaluate and prune weights during gradient updates. This can yield more robust sparse networks but adds complexity.
• Rewinding: A key technique in iterative schedules where, after a pruning step, weights are reset to an earlier training checkpoint ('rewound') before fine-tuning continues, often improving the recovery of the lottery ticket subnetworks.

The ultimate goal is efficient inference. The schedule must produce a sparsity pattern executable on target hardware.

• Hardware-Informed Final Sparsity: Targeting 50% unstructured sparsity is ineffective if the deployment hardware lacks sparse acceleration support. The schedule should aim for a hardware-friendly pattern (e.g., structured pruning or 2:4 sparsity).
• Schedule-Accelerator Co-Design: For systems using GPUs with structured sparse tensor cores (e.g., NVIDIA Ampere), the pruning schedule should enforce the required N:M pattern throughout the iterative process to ensure the final model is immediately deployable.

A schedule requires clear metrics to determine progress and completion.

• Primary Signal: Validation Accuracy/Loss: After each prune-and-fine-tune cycle, performance is measured. A significant drop may trigger a reduction in the pruning rate for the next cycle or a rollback.
• Target Sparsity Level: The schedule often concludes when a pre-defined global or layer-wise sparsity is reached.
• Performance Plateau: Pruning may stop if fine-tuning no longer recovers accuracy, indicating the network cannot sustain further parameter reduction without architectural change.
• Automated tools track the trade-off curve between sparsity and accuracy to identify the optimal stopping point.

Pruning granularity defines the smallest structural unit removed by the pruning algorithm, directly dictating the schedule's step size and retraining complexity.

• Fine-grained (Unstructured): Removes individual weights. Schedules are often aggressive but require specialized runtimes for sparse computation.
• Coarse-grained (Structured): Removes entire filters, channels, or attention heads. Schedules must account for the larger impact of each removal, often requiring careful layer-wise sensitivity analysis.
• The choice of granularity determines whether the schedule produces a hardware-friendly dense model or a highly sparse model requiring specialized kernels.

The pruning criterion is the metric used to score parameters for removal, forming the decision logic executed at each step of the schedule.

• Magnitude-based (e.g., L1/L2 norm): Simple and common. A schedule might prune the bottom 10% of weights by absolute value each iteration.
• Gradient-based (e.g., Movement Pruning): Removes weights whose values change the least during training, requiring the schedule to integrate with the training loop.
• Saliency-based (e.g., SNIP): Estimates a connection's effect on the loss before training begins, enabling one-shot pruning schedules.
• The criterion influences whether the schedule is static (pre-defined) or dynamic (adaptive based on training progress).

Iterative Magnitude Pruning (IMP) is a canonical algorithm that defines a specific, cyclical pruning schedule. It establishes the foundational pattern of prune-then-retrain.

• The schedule follows a strict loop: Train → Prune a fixed percentage of smallest-magnitude weights → Rewind weights to an earlier checkpoint → Retrain. This cycle repeats until the target sparsity is reached.
• IMP demonstrates that a gradual, iterative schedule with rewinding is more effective at preserving accuracy than one-shot pruning.
• Modern schedules often build upon or modify the IMP template, adjusting the pruning rate, frequency, and retraining duration.

The sparsity pattern is the specific, final map of zero-valued weights produced by a pruning schedule. The schedule's design directly controls this pattern's properties.

• Unstructured patterns are irregular and require the schedule to manage a growing number of isolated zeros.
• Structured patterns (e.g., N:M Sparsity) are regular and hardware-aware. A schedule for 2:4 sparsity must enforce that exactly 2 of every 4 contiguous weights are pruned.
• The pattern determines the model's memory layout and the efficiency of Sparse Matrix Multiplication kernels during inference. A poor schedule can create a pattern that is difficult to execute efficiently.

Pruning-induced accuracy drop is the immediate performance degradation after a pruning step. The primary goal of a well-designed schedule is to manage and recover from this drop.

• Aggressive schedules (large pruning steps) cause a larger initial drop but converge to the target sparsity faster.
• Conservative schedules (small, frequent steps) minimize the immediate drop but lengthen the overall process.
• The schedule allocates fine-tuning or rewinding phases specifically to recover accuracy. The duration and learning rate of these phases are critical hyperparameters of the schedule itself.

Pruning-aware training is a paradigm where the pruning schedule is not a separate post-processing step but is integrated into the initial model training.

• Techniques like Dynamic Network Surgery or regularization (e.g., L0) allow the schedule to be continuous and adaptive from the start of training.
• The schedule is defined by gradual, automated sparsification as the network learns, rather than a predefined cycle of pruning events.
• This approach often eliminates the need for a separate, complex multi-phase schedule, as the model is trained to be sparse from the outset.