Weight Pruning

The most common heuristic where weights with the smallest absolute values are removed, based on the assumption that they contribute least to the network's output.

• Unstructured pruning zeros out individual weights, creating irregular sparsity patterns
• Structured pruning removes entire channels, filters, or neurons, preserving dense matrix operations
• Typically applied iteratively with fine-tuning between pruning steps to recover accuracy
• A sparsity ratio (e.g., 80-95%) defines the target percentage of weights to eliminate

A training-aware approach that progressively increases sparsity from an initial value to a target over many training steps, allowing the network to adapt.

• Uses a polynomial decay or cubic sparsity schedule to slowly zero out weights
• Starts from a dense or pre-trained checkpoint and prunes while continuing to train
• Often combined with weight rewinding, resetting surviving weights to their earlier values
• Produces sparse subnetworks that match or exceed the accuracy of one-shot pruning methods

The arrangement of zero-valued weights directly impacts achievable speedup on FPGA and custom hardware.

• Unstructured sparsity requires specialized sparse matrix libraries and irregular memory access patterns
• Block sparsity constrains zeros to fixed-size blocks (e.g., 4x4), enabling vectorized computation
• N:M sparsity enforces exactly N non-zero values per M consecutive weights, as in NVIDIA's 2:4 pattern
• FPGA implementations benefit from structured sparsity that maps efficiently to DSP slices and on-chip memory

A cyclical process that alternates between removing weights and fine-tuning the remaining connections to recover modulation classification accuracy.

• One-shot pruning removes weights in a single step, often causing significant accuracy degradation
• Iterative pruning removes a small fraction (e.g., 10-20%) per cycle, followed by retraining
• Each iteration allows the network to redistribute representational capacity among surviving weights
• The Lottery Ticket Hypothesis suggests that winning sparse subnetworks exist within randomly initialized dense networks

Beyond simple magnitude, more sophisticated saliency metrics use gradient information to identify which weights are truly expendable.

• SNIP (Single-shot Network Pruning) uses a connection sensitivity score based on the product of weight magnitude and gradient
• GraSP (Gradient Signal Preservation) prunes weights that least affect the gradient flow through the network
• Taylor expansion methods approximate the change in loss function caused by removing each weight
• These criteria better preserve signal classification accuracy at extreme sparsity levels (>95%)

Deploying pruned modulation classifiers on FPGA requires specialized dataflow architectures and memory access patterns.

• Sparse matrix-vector multiplication kernels skip zero-valued weights to reduce MAC operations
• Compressed Sparse Row (CSR) or Compressed Sparse Column (CSC) formats store only non-zero values and their indices
• Load imbalance between processing elements is a key challenge due to irregular sparsity distributions
• Fine-grained structured sparsity (e.g., vector-level pruning) balances compression ratio with hardware utilization