Pruning-Induced Accuracy Drop

Pruning directly removes the parameters that encode the network's learned mapping from input to output. Critical, non-redundant weights that represent high-level features or long-range dependencies are sometimes eliminated. This is especially damaging in later layers where representations are more task-specific. The network loses its capacity to compute certain intermediate functions, leading to a direct drop in task performance on the validation set before any recovery via fine-tuning.

Neural networks are trained via backpropagation, which relies on continuous paths of non-zero weights to propagate error signals. Unstructured pruning can create dead neurons—units whose outputs are zeroed out because all incoming weights are pruned. This severs the gradient flow through those paths, making it impossible to fine-tune upstream layers that contributed to the neuron. Even with sparse fine-tuning, the optimization landscape becomes more fractured and difficult to navigate.

The Lottery Ticket Hypothesis suggests that dense networks contain sparse, trainable subnetworks ('winning tickets'). Pruning-induced accuracy drop occurs when the pruning criterion fails to identify a true winning ticket. Removing weights indiscriminately (e.g., by magnitude alone) can break the critical, synergistic connections that formed the ticket. The remaining subnetwork may be untrainable to the original accuracy, necessitating the rewinding of weights to an earlier training iteration to find a viable optimization path.

Aggressive or unstructured pruning does not remove capacity uniformly. It can create severe architectural imbalances where one layer becomes excessively sparse relative to its neighbors. For example, pruning 80% of a convolutional layer's filters creates a bottleneck, forcing the next layer to work with severely reduced feature maps. This mismatch damages the information flow that the original architecture was designed to preserve. Structured pruning methods like channel pruning are designed to mitigate this by removing coherent structural units.

The distribution of activations (layer outputs) in a trained network is stable. Pruning changes the linear transformations performed by each layer, which can cause a covariate shift in the input distribution for subsequent layers. This shift violates the statistical assumptions baked into the parameters of later layers and batch normalization statistics. The network's performance degrades because its internal data processing pipeline is no longer aligned, a phenomenon studied in the context of post-training quantization as well.

Dense networks often have a robustness margin where small parameter changes don't affect output. Pruning removes this redundancy, making the sparse network more sensitive. Weights that previously had minor, compensatory roles can become critical single points of failure. This amplifies sensitivity to adversarial examples and noisy inputs. The pruned model may perform well on clean data but fail on the slightly varied data present in the validation set, contributing to the observed accuracy drop.

The process of retraining a pruned neural network on a task-specific dataset to recover the accuracy lost during pruning. The sparsity pattern is typically held fixed, and only the remaining non-zero weights are updated. This is the primary method for mitigating pruning-induced accuracy drop.

• Goal: Regain performance without regrowing pruned connections.
• Practice: Often involves a lower learning rate and fewer epochs than the original training.

The metric or heuristic used to determine which weights or structures are least important and can be removed. The choice of criterion directly influences the severity of the subsequent accuracy drop.

• Common Criteria:
  • Magnitude (L1/L2 Norm): Removes weights with the smallest absolute values.
  • Gradient-based (e.g., Movement Pruning): Removes weights whose values change the least during training.
  • Activation Statistics: Removes filters or channels that cause minimal activation.

An analysis that measures how the removal of specific weights, filters, or layers affects a model's output or loss. It is used to design layer-specific pruning strategies to minimize accuracy drop.

• Purpose: Identify which parts of a network are most vulnerable to pruning.
• Outcome: Informs non-uniform pruning schedules, where sensitive layers are pruned less aggressively than robust ones.

A technique used in Iterative Magnitude Pruning (IMP) where, after a pruning step, the network's weights are reset to values from an earlier training checkpoint (e.g., early in training) before fine-tuning continues.

• Mechanism: The 'rewound' weights are believed to retain the capacity for learning, which helps recovery during sparse fine-tuning.
• Benefit: Often leads to better final accuracy compared to fine-tuning from the final trained weights, reducing the overall accuracy drop.

A training paradigm that incorporates sparsity-inducing regularization or progressive pruning directly into the model training loop. The goal is to produce a network that is inherently robust to parameter removal, thus reducing the final accuracy drop.

• Examples:
  • Adding L0 or L1 regularization to encourage weights toward zero.
  • Gradually increasing sparsity during training (Pruning Schedule).
• Contrast: Differs from standard post-training pruning, where the model is fully trained before compression.