Neural Network Pruning

The most common pruning criterion removes weights with the smallest absolute magnitudes, based on the assumption that near-zero weights contribute negligibly to the network's output. After training a dense predistorter, a threshold is applied to zero out connections below a specified percentile.

• Unstructured pruning removes individual weights, creating irregular sparsity patterns
• Structured pruning removes entire neurons, filters, or channels for hardware-friendly regularity
• Typical pruning ratios range from 50% to 95% for PA linearization networks
• Fine-tuning after pruning is essential to recover any lost linearization performance

Example: A fully connected RVTDNN with 10,000 weights may have 80% of its connections pruned, leaving only 2,000 active parameters while maintaining ACLR within 0.5 dB of the dense model.

Rather than pruning in a single aggressive step, iterative pruning alternates between removing a small fraction of weights and retraining the remaining connections. This gradual approach allows the network to adapt and redistribute representational capacity.

• Gradual sparsification prevents catastrophic accuracy collapse
• Each iteration typically removes 5-20% of remaining weights
• Learning rate schedules are often reduced during retraining phases
• The process continues until the target sparsity or performance threshold is reached

Example: A predistorter trained for 100 epochs may undergo 10 pruning-retraining cycles, each removing 10% of weights, resulting in a final sparsity of 65% with minimal EVM degradation.

Pruned networks require specialized hardware and software support to translate theoretical parameter reduction into actual latency and power savings. FPGAs and custom ASICs can exploit sparsity through dedicated architectures.

• Sparse matrix multiplication kernels skip zero-valued operands
• Compressed sparse row (CSR) or column (CSC) formats store only non-zero weights
• Irregular sparsity patterns challenge SIMD and vectorized execution units
• Structured pruning produces dense sub-blocks that map efficiently to systolic arrays

Example: An FPGA implementing a pruned DPD network with 90% sparsity may achieve a 3-5x reduction in DSP slice utilization and a proportional decrease in dynamic power consumption compared to the dense implementation.

Pruning can be integrated directly into the training process by adding regularization terms that encourage sparsity. This eliminates the need for a separate post-training pruning step and often produces better-performing sparse networks.

• L1 regularization adds the sum of absolute weight values to the loss function, driving small weights to exactly zero
• Group Lasso regularizes entire groups of parameters together, promoting structured sparsity
• Variational dropout learns per-weight dropout rates, automatically pruning weights with high dropout probabilities
• The regularization strength hyperparameter controls the trade-off between sparsity and accuracy

Example: Training a predistorter with L1 regularization (λ=0.001) naturally produces a network where 70% of weights are zero at convergence, requiring no additional pruning step.

The lottery ticket hypothesis posits that within a randomly initialized dense network, there exists a sparse subnetwork that can be trained in isolation to match the full network's performance. This has implications for designing inherently efficient predistorter architectures.

• Winning tickets are identified through iterative magnitude pruning and weight resetting
• The initial random weights of the winning ticket are crucial—random reinitialization degrades performance
• This suggests that network architecture search can focus on finding sparse connectivity patterns
• Winning tickets typically achieve 80-95% sparsity while matching dense network accuracy

Example: A winning ticket extracted from a 3-layer RVTDNN for Doherty PA linearization achieves equivalent ACLR improvement with only 12% of the original parameters, enabling deployment on ultra-low-cost FPGAs.

Beyond simple magnitude thresholds, sensitivity-based methods estimate the impact of removing each weight on the network's loss function. This provides a more principled approach to identifying truly redundant parameters.

• Hessian-based pruning uses second-order derivatives to estimate weight importance
• Optimal Brain Damage and Optimal Brain Surgeon algorithms compute saliency scores
• Taylor expansion approximates the change in loss when a weight is removed
• Sensitivity analysis is computationally expensive but produces higher-quality sparse networks

Example: A sensitivity-pruned predistorter may retain a small-magnitude weight that is critical for modeling a specific PA memory effect, while removing a larger weight that contributes redundantly—achieving better linearization than magnitude-based pruning at the same sparsity level.