Pruning Sensitivity

Pruning sensitivity is not uniform across a network. It forms a gradient where certain layers are far more critical than others. Typically, early convolutional layers and final classification layers exhibit high sensitivity, as they capture fundamental features and make final decisions. In contrast, middle layers often show greater redundancy and can tolerate higher sparsity. Sensitivity is measured by the increase in loss or drop in accuracy when a specific layer or filter is pruned, often visualized as a sensitivity profile chart.

The measured sensitivity varies drastically with the pruning granularity. The impact of removing a single weight (unstructured pruning) is often negligible, but the cumulative effect can be large. Removing an entire filter (structured pruning) has an immediate, measurable impact on the feature maps passed to subsequent layers. Therefore, sensitivity analysis must be performed at the same granularity intended for the final pruned model. A filter may be sensitive to removal, but the individual weights within it may not be equally important.

A model's pruning sensitivity profile is intrinsically tied to its training task and dataset. A filter critical for ImageNet object classification may be irrelevant for a medical imaging dataset. Therefore, sensitivity analysis should be performed on a representative validation set from the target domain. Pruning based on sensitivity measured on a generic dataset can remove features essential for the deployment task, leading to significant, unrecoverable accuracy loss.

Sensitivity is influenced by the underlying network architecture. In ResNet models with skip connections, pruning sensitive paths can disrupt the gradient flow essential for training stability. In Transformer models, sensitivity analysis often reveals that FFN (Feed-Forward Network) layers are more prune-tolerant than attention heads, especially in later layers. The analysis must account for architectural components like batch normalization, where pruning channels affects the running statistics.

Sensitivity is not a static property. It evolves during training as the network learns and features become more refined. A weight deemed unimportant early in training may become critical later. This is the principle behind iterative pruning algorithms, which repeatedly measure sensitivity and prune after phases of retraining. One-shot pruning methods, like SNIP, attempt to predict final sensitivity at initialization, but this is an approximation of the dynamic process.

Sensitivity is formally quantified using second-order information. The Hessian matrix (or approximations like the Fisher Information Matrix) measures the curvature of the loss landscape. Parameters associated with high eigenvalues in the Hessian are highly sensitive—their removal causes a large increase in loss. First-order methods, like Movement Pruning, use gradient signals over time to estimate sensitivity. In practice, cheaper proxies like weight magnitude (L1/L2 norm) or activation-based scores are often used as sensitivity estimators.

A pruning criterion is the specific metric or heuristic used to score the importance of weights, filters, or layers for removal. Sensitivity analysis often evaluates the impact of applying different criteria. Common criteria include:

• Magnitude-based (L1/L2 Norm): Removes weights with the smallest absolute values.
• Gradient-based: Scores parameters by their influence on the training loss gradient.
• Activation-based: Uses statistics like average percentage of zeros in a neuron's output. The choice of criterion directly determines which parameters are flagged as 'sensitive' and heavily influences the final sparsity pattern and accuracy trade-off.

Pruning granularity defines the smallest structural unit that can be removed. Sensitivity must be analyzed at the corresponding level, as the impact of removing a single weight is vastly different from removing an entire layer.

• Fine-grained (Unstructured): Individual weights. High flexibility but creates irregular sparsity.
• Structured: Groups like filters, channels, or attention heads. Less flexible but produces hardware-friendly, dense sub-networks.
• Layer-wise: Entire layers. Coarsest granularity; requires analyzing the sensitivity of the layer's function to the overall task. Granularity choice dictates the analysis method and the hardware optimizations possible post-pruning.

Sparse fine-tuning is the critical recovery phase applied after pruning based on sensitivity analysis. The identified sparse architecture (with zeros fixed) is retrained on task data to regain lost accuracy.

• Key Technique: The sparsity pattern is typically frozen; only the remaining non-zero weights are updated.
• Connection to Sensitivity: Layers or weights identified as highly sensitive often require more fine-tuning epochs or a lower learning rate to recover effectively. Sensitivity maps can guide the fine-tuning schedule, allocating more recovery budget to critical regions of the network.

This fundamental dichotomy in pruning approaches necessitates different sensitivity analysis techniques.

• Unstructured Pruning: Removes individual weights. Sensitivity is often measured as the expected change in loss for removing a specific parameter. Tolerates higher pruning rates in non-sensitive areas but requires specialized libraries/hardware (e.g., sparse kernels) for speedup.
• Structured Pruning: Removes entire structural units (e.g., a filter). Sensitivity is measured as the impact on the output feature maps or the overall task loss. Yields immediately executable dense models on standard hardware but may have a higher accuracy drop per parameter removed. The choice dictates whether sensitivity is analyzed per-weight or per-structure.

The Lottery Ticket Hypothesis provides a theoretical framework that intersects with pruning sensitivity. It posits that dense networks contain sparse, trainable subnetworks ('winning tickets') that can match original performance.

• Sensitivity as a Ticket Finder: Pruning sensitivity analysis can be viewed as a method to identify these high-performance subnetworks. Parameters deemed 'insensitive' to removal are likely not part of the critical winning ticket.
• Iterative Process: The hypothesis is often validated through Iterative Magnitude Pruning (IMP), which repeatedly prunes low-magnitude weights and rewinds weights to early training values—a process that inherently measures and responds to parameter sensitivity over time.

Hardware-aware pruning ensures the pruned model is efficient on target deployment hardware (e.g., GPUs, NPUs, mobile CPUs). Sensitivity analysis must incorporate hardware constraints.

• N:M Sparsity: A hardware-friendly structured pattern (e.g., 2:4) where 2 out of every 4 consecutive weights are non-zero. Sensitivity analysis must operate within these block constraints.
• Latency/Cycle Estimation: True sensitivity isn't just about accuracy loss, but also inference speedup. A layer may be accuracy-sensitive but pruning it might yield massive latency gains on specific hardware, altering the cost-benefit analysis. Effective pruning uses sensitivity metrics that blend accuracy impact with hardware performance models.