Dynamic Network Surgery

Unlike one-shot pruning, Dynamic Network Surgery operates in a continuous loop during training. It prunes connections with low importance scores and can splice (re-introduce) previously pruned connections if their importance rises in later training iterations. This allows the network architecture to evolve dynamically, searching for an optimal sparse structure.

• Core Mechanism: A binary mask is applied to the weights, which is updated based on a saliency criterion.
• Key Benefit: Mitigates the risk of permanently removing weights that may become important later in training.

The decision to prune or splice is governed by a real-time importance metric. The original paper introduced a Taylor expansion-based criterion, approximating the change in the loss function if a specific weight were removed.

• Saliency Formula: Importance is often calculated as the product of the weight's magnitude and the absolute value of its gradient (|w * ∇L|).
• Dynamic Threshold: Connections with saliency below a moving threshold are pruned; pruned connections whose saliency rises above a (higher) threshold can be regrown.

The technique typically produces unstructured sparsity, meaning individual weights are zeroed out without regard to architectural patterns like entire neurons or filters. This allows for high theoretical compression rates but requires software libraries or hardware that support sparse tensor operations for efficient inference.

• Sparsity Pattern: Irregular and data-dependent, determined by the training process.
• Deployment Consideration: The resulting model is a sparse checkpoint; achieving actual speedups requires inference on hardware with sparse compute support (e.g., NVIDIA A100+ with sparse tensor cores).

Dynamic Network Surgery differs fundamentally from classic Iterative Magnitude Pruning (IMP). IMP uses a rigid schedule: train → prune smallest-magnitude weights → retrain from final weights. Dynamic Surgery is continuous and reversible.

• IMP: Pruning is a discrete, one-way event during scheduled intervals.
• Dynamic Surgery: Pruning and splicing are continuous, parallel processes guided by real-time gradients.
• Objective: Dynamic Surgery seeks to maintain a performant network throughout training, not just recover performance after pruning.

The method is conceptually aligned with the Lottery Ticket Hypothesis, which suggests dense networks contain sparse, trainable subnetworks. Dynamic Network Surgery can be seen as an active search for these "winning tickets" during the primary training run, rather than a post-hoc discovery process.

• Exploration vs. Exploitation: It explores the sparse architecture space by regrowing connections, potentially finding better subnetworks than magnitude-based pruning alone.

The central goal is to achieve high sparsity levels with minimal pruning-induced accuracy drop. The ability to restore connections acts as a safety mechanism, allowing for more aggressive pruning early in training. The final model often achieves a better accuracy-sparsity trade-off compared to static, one-shot pruning methods, as the network has actively adapted to its constrained parameter budget.

A foundational pruning algorithm that serves as a key precursor to dynamic methods. IMP operates in cycles:

• Prune: Remove a small percentage of weights with the smallest absolute magnitude.
• Retrain: Fine-tune the remaining network to recover accuracy.
• Repeat: Iterate this cycle until the target sparsity is achieved.

Unlike dynamic surgery, IMP's pruning decisions are final; removed weights are not restored. It established the core paradigm of iterative pruning and retraining that more advanced techniques build upon.

A gradient-based importance criterion closely related to the logic of dynamic surgery. Instead of pruning based on a weight's final magnitude, movement pruning scores connections by how much their value changes (or 'moves') during training.

• Weights that move significantly (large positive or negative gradient) are considered important for learning.
• Weights that remain static are candidates for removal. This method aligns with the dynamic surgery principle of evaluating importance in real-time based on training dynamics, rather than a static snapshot.

The specific metric or heuristic used to decide which parameters to remove. Dynamic Network Surgery requires a real-time, computable criterion. Common criteria include:

• Magnitude (L1 Norm): Absolute value of the weight.
• Gradient-based: Sensitivity of the loss to the weight's removal.
• Activation Statistics: How much a neuron/filter contributes to downstream outputs.

The choice of criterion directly impacts the quality of the resulting sparse network and the efficiency of the pruning process.

The retraining phase applied to a pruned network to recover accuracy. In dynamic surgery, fine-tuning occurs iteratively after each 'surgery' step.

Key aspects:

• The sparsity pattern (locations of zero weights) is typically held fixed during fine-tuning.
• Only the remaining non-zero weights are updated.
• The goal is to redistribute the representational capacity of the network to compensate for the removed connections. This process is critical for mitigating the pruning-induced accuracy drop.

The class of pruning that removes individual weights anywhere in the network, creating an irregular, non-structured pattern of zeros. Dynamic Network Surgery is inherently an unstructured pruning technique.

Characteristics:

• Achieves very high theoretical sparsity (e.g., 90%+ zeros).
• The resulting model is a sparse neural network.
• Requires specialized software libraries (e.g., those supporting sparse matrix multiplication) or hardware to realize computational speedups, as the irregular memory access patterns don't map efficiently to standard dense hardware.

A hardware-friendly structured sparsity pattern that contrasts with the unstructured output of dynamic surgery. In N:M sparsity, for every block of M consecutive weights (e.g., within a single vector), at most N are allowed to be non-zero.

Example: 2:4 sparsity means in every group of 4 weights, 2 are non-zero and 2 are zero.

• This pattern is efficiently supported by modern NVIDIA Ampere/Ada/Hopper GPUs via the Sparse Tensor Core.
• It represents a middle ground between the flexibility of unstructured pruning (like dynamic surgery) and the runtime efficiency of coarse-grained structured pruning.