Model Pruning

The most straightforward pruning criterion where weights with the smallest absolute values are removed. The core assumption is that near-zero weights contribute negligibly to the network's output.

• Unstructured pruning zeros out individual weights, creating sparse matrices
• Structured pruning removes entire neurons, channels, or filters for hardware-friendly compression
• Typically achieves 50-90% sparsity with minimal accuracy loss when combined with fine-tuning

Example: In a ResNet-50 trained on ImageNet, magnitude pruning can remove 80% of weights while maintaining accuracy within 1% of the original.

Rather than removing weights in a single pass, iterative pruning alternates between pruning small fractions of weights and retraining the network to recover accuracy.

• Gradual pruning schedules prevent catastrophic accuracy collapse
• Each retraining cycle allows remaining weights to compensate for removed connections
• Lottery Ticket Hypothesis suggests that dense networks contain sparse subnetworks that can be trained in isolation

This approach is critical for federated learning where local models may be pruned before transmission to reduce communication overhead.

The choice between structured and unstructured pruning represents a fundamental trade-off between compression flexibility and hardware efficiency.

Unstructured Pruning:

• Removes individual weights regardless of position
• Achieves higher theoretical compression ratios
• Requires specialized sparse matrix hardware for acceleration

Structured Pruning:

• Removes entire channels, filters, or attention heads
• Directly reduces model dimensions and FLOPs
• Immediately accelerates inference on standard hardware without custom kernels

For federated edge deployment, structured pruning is often preferred as it produces models that run efficiently on commodity NPUs and microcontrollers.

Beyond simple magnitude, gradient-based methods evaluate weight importance by measuring how removal impacts the loss function.

• Taylor expansion pruning approximates the change in loss from removing each weight using first-order or second-order gradient information
• SNIP (Single-shot Network Pruning) identifies important connections at initialization before any training begins
• Movement pruning tracks how weights change during fine-tuning, retaining those that move away from zero

These methods are particularly valuable in federated settings where retraining opportunities are limited and pruning decisions must be made with high confidence.

Model pruning serves dual purposes in federated learning: reducing communication costs and enabling edge deployment.

• Clients can prune local models before transmitting updates, dramatically reducing bandwidth requirements
• The global model can be progressively pruned across rounds to converge on a sparse final architecture
• FedPrune and similar algorithms incorporate pruning directly into the federated optimization objective

Key considerations include handling non-IID data distributions where different clients may require different sparse subnetworks, and ensuring pruned models maintain performance across heterogeneous local datasets.

Proposed by Frankle and Carbin in 2019, this hypothesis states that dense, randomly-initialized networks contain sparse subnetworks that can be trained in isolation to match the full network's accuracy.

• These winning tickets are identified through iterative magnitude pruning
• The key insight: the initialization values of the winning ticket matter, not just the architecture
• In federated contexts, identifying winning tickets early could allow clients to train only sparse models from the start

This has profound implications for resource-constrained federated learning, suggesting that sparse training may be possible without the full dense pretraining phase.

Trains a compact student model to mimic the output distribution of a larger, more accurate teacher model. The student learns from both hard labels and the teacher's soft probabilities, capturing inter-class relationships that pruning alone cannot preserve.

• Teacher-student architecture transfer
• Often combined with pruning for compound compression
• Effective when pruning causes irreversible accuracy degradation

Unstructured pruning zeroes out individual weights, creating sparse matrices that require specialized hardware or software for speedup. Structured pruning removes entire neurons, channels, or layers, yielding dense sub-networks compatible with standard accelerators.

• Unstructured: higher compression ratio, irregular sparsity
• Structured: immediate latency reduction on commodity hardware
• Channel pruning is the most common structured approach for CNNs

Posits that dense, randomly-initialized networks contain sparse sub-networks that can be trained in isolation to match the original's accuracy. These winning tickets are identified through iterative magnitude pruning and weight resetting.

• Discovered by Frankle & Carbin (2019)
• Challenges the necessity of over-parameterization
• Implies pruning can be a training strategy, not just compression

Automates the discovery of efficient model architectures optimized for specific hardware constraints. NAS can directly search for pruned architectures or incorporate pruning as a search dimension alongside layer count and kernel size.

• Hardware-aware NAS targets latency and memory budgets
• Often discovers architectures more efficient than hand-pruned models
• Used in production for mobile and edge deployment targets

Modern accelerators like NVIDIA Ampere and Apple Neural Engine include native support for structured sparsity patterns, enabling direct speedup from pruned models. Without hardware support, unstructured sparsity yields limited inference gains.

• 2:4 structured sparsity supported in cuSPARSELt
• Sparse Tensor Cores accelerate pruned matrix multiplications
• Critical bridge between algorithmic pruning and real-world latency reduction