Pruning Schedule

The sparsity target is the final, desired percentage of zero-valued parameters in the model after pruning is complete. This is the primary objective of the schedule.

• Global vs. Layer-wise: A target can be applied uniformly across the entire network (global sparsity) or allocated differently per layer based on layer-wise sensitivity.
• Trade-off: Higher sparsity targets yield smaller models and faster inference but risk greater accuracy degradation. The target is often determined by profiling the compression-accuracy Pareto frontier.

The pruning criterion is the metric used to identify which parameters are less important and can be removed. The choice of criterion directly impacts which subnetworks are discovered.

• Magnitude-Based: The most common method, removing weights with the smallest absolute values (e.g., Iterative Magnitude Pruning).
• Gradient-Based: Removes parameters that have the smallest effect on the loss function.
• Activation-Based: Prunes neurons or filters that produce minimal activation across a dataset.
• Structured Criteria: Uses criteria to remove entire groups (filters, channels) for hardware efficiency, as in Structured Pruning.

Pruning frequency defines when during training the network is sparsified. It dictates the interval between pruning steps, measured in training iterations or epochs.

• One-Shot Pruning: Prune once, typically after training is complete or at a specific epoch, then fine-tune.
• Iterative Pruning: Prune repeatedly at regular intervals (e.g., every N epochs), allowing the network to recover between steps. This is the core of Gradual Pruning schedules.
• Continuous Pruning: An extreme form where pruning is applied at very high frequency, often intertwined with regrowth as seen in Sparse Evolutionary Training (SET).

The pruning rate function (or sparsity ramp) determines how much to prune at each step. It maps the training progress to a current sparsity level.

• Linear Schedule: Sparsity increases by a fixed amount at each pruning step.
• Cosine Schedule: Sparsity increases slowly at the beginning and end of the schedule, and more rapidly in the middle, following a cosine annealing curve. This is a common Cosine Pruning Schedule.
• Exponential Schedule: Sparsity increases rapidly early on and then tapers.
• Adaptive Schedule: The rate is adjusted dynamically based on feedback-driven scheduling signals like validation loss.

After parameters are removed, a fine-tuning or recovery phase is essential to allow the remaining weights to adapt and regain lost accuracy. The schedule must define the duration and hyperparameters for this phase.

• Duration: Can be a fixed number of epochs or continue until validation accuracy plateaus.
• Learning Rate: Typically uses a reduced learning rate compared to the initial training phase.
• Connection to Hypothesis: This phase is critical for validating the Lottery Ticket Hypothesis, as it allows the identified sparse subnetwork ('winning ticket') to be retrained.

The sparsity distribution policy dictates how the global sparsity target is allocated across different layers, blocks, or parameter types within the network. A uniform distribution is often suboptimal.

• Sensitivity-Guided: Allocates less sparsity to layers identified as highly sensitive via layer-wise sensitivity analysis.
• Parameter-Count Based: Allocates more sparsity to layers with more parameters to achieve greater overall reduction.
• Automated: Determined by search algorithms like Automated Model Compression (AMC) or Hardware-Aware Neural Architecture Search (HW-NAS) that optimize the distribution for a specific objective (e.g., latency, accuracy).

A scheduling strategy that incrementally increases the sparsity of a model over many training steps or epochs, allowing the network to adapt smoothly and preserve accuracy. This contrasts with one-shot pruning.

• Key Mechanism: Sparsity is increased from an initial value (often 0%) to a final target (e.g., 90%) following a function like cubic, linear, or cosine annealing.
• Benefit: Provides the network time to recover and re-distribute representational capacity, typically resulting in higher final accuracy compared to aggressive one-shot removal.
• Example: A schedule might prune 2% more weights every 100 training steps until the target 80% sparsity is reached.

A comprehensive set of rules and algorithms that govern which compression techniques to apply, when to apply them, and with what intensity across a model's lifecycle. It is the overarching plan of which a pruning schedule is a part.

• Components: Includes decisions on technique sequencing (e.g., prune then quantize), layer-wise sparsity targets, timing triggers (epoch count, loss plateau), and recovery mechanisms.
• Scope: Can be uniform or highly customized based on layer-wise sensitivity analysis.
• Objective: To systematically achieve a target model size, latency, or energy profile while constrained by an acceptable accuracy loss.

The set of optimal model configurations where no further compression can be achieved without sacrificing accuracy, and vice-versa. It is the empirical boundary that guides scheduling decisions and policy design.

• Visualization: Typically a 2D plot with model size (or latency) on one axis and accuracy (e.g., Top-1) on the other. The frontier is the curve of best-performing points.
• Use Case: Engineers evaluate multiple schedules (e.g., different final sparsities) to map this frontier and select an operating point that meets deployment constraints.
• Relation to Scheduling: A pruning schedule aims to produce a model that lies on this frontier for its target sparsity level.

A strategy for removing entire groups of parameters, such as filters, channels, or neurons, according to a planned timeline to maintain hardware-efficient sparsity patterns. This contrasts with unstructured pruning.

• Targets: Prunes based on group importance metrics (e.g., L2 norm of a filter, channel activation).
• Hardware Advantage: Results in coarse-grained sparsity that can leverage dedicated kernel support (e.g., for dense matrix multiplication of smaller matrices) without requiring specialized sparse hardware.
• Schedule Consideration: Often requires different sensitivity analysis and recovery fine-tuning than unstructured schedules, as removing entire structures has a more significant impact on feature maps.

An adaptive approach where compression decisions (e.g., pruning rate, quantization bit-width) are continuously adjusted based on live metrics like validation loss or gradient norms. It moves beyond static, predefined schedules.

• Mechanism: Monitors a feedback signal (e.g., increase in validation loss) during compression. If the signal exceeds a threshold, the schedule pauses or reduces the aggressiveness of compression.
• Benefit: Can prevent catastrophic accuracy drops by reacting to the model's state during training/fine-tuning.
• Example: A schedule might only prune weights in a layer if the average gradient magnitude for that layer remains above a certain level, indicating the parameters are still actively learning.