Residual Learning

The defining structural element of residual learning. A skip connection adds the input of a layer (or stack of layers) directly to its output, forming H(x) = F(x) + x. This identity mapping ensures that the network can always learn to pass information forward unchanged if beneficial. For digital predistortion, skip connections prevent the vanishing gradient problem in deep networks with 10+ layers, enabling stable training of complex PA inverse models that capture subtle memory effects without degradation in earlier layers.

Residual networks transform the optimization problem from learning an unreferenced target mapping to learning a perturbation around identity. This smooths the loss surface, making it significantly easier for gradient descent to navigate. Key benefits for DPD:

• Faster convergence during coefficient extraction
• Reduced sensitivity to weight initialization choices
• Ability to train networks with 20-50 layers for wideband signals
• More robust convergence when using the Indirect Learning Architecture (ILA)

By structuring the network as F(x) + x, residual learning embeds a strong inductive bias: the optimal predistorter is often close to a linear pass-through. This is physically meaningful for power amplifiers operating with modest back-off, where the required predistortion is a small correction to the linear response. The network only needs to learn the nonlinear deviation from identity, which is inherently a sparser and simpler function than the full predistorter characteristic.

Skip connections create multiple paths for gradient flow during backpropagation. The error signal can propagate directly through the identity shortcut without attenuation, while also flowing through the weighted layers. This dual-path gradient propagation:

• Provides deep supervision to early layers
• Prevents the degradation problem where deeper networks perform worse than shallower ones
• Enables effective training of predistorters for mmWave and wideband signals requiring deep architectures to capture long memory spans

Multiple residual block designs exist for different DPD requirements:

• Basic Block: Two convolutional or fully-connected layers with a skip connection, suitable for moderate-depth predistorters
• Bottleneck Block: Uses 1x1 convolutions to reduce then restore dimensionality, reducing parameters for very deep networks
• Pre-activation Block: Places batch normalization and activation before the weight layer, improving gradient flow further
• Dense Residual Block: Combines skip connections with dense connectivity, useful for capturing complex memory polynomial interactions

Residual learning extends naturally to Complex-Valued Neural Networks (CVNNs) for direct I/Q signal processing. The skip connection preserves both magnitude and phase of the complex baseband signal, while the residual branch learns the complex nonlinear correction. This is critical for predistortion because:

• Phase information is preserved through the identity path
• The residual branch can focus on modeling AM/AM and AM/PM distortion
• Complex batch normalization and complex weight initialization must be adapted for residual CVNN structures

The fundamental architectural element of residual learning. A skip connection provides an identity shortcut that bypasses one or more weighted layers, summing the layer's input directly with its output. This creates a residual block that explicitly computes F(x) + x rather than learning a direct mapping.

• Gradient Highway: Prevents vanishing gradients by allowing error signals to propagate directly through the network during backpropagation
• Identity Mapping: Ensures that deeper layers can, at minimum, learn the identity function, preventing degradation
• Pre-activation Design: Placing batch normalization and activation before the weight layer improves regularization and gradient flow

The canonical architecture that introduced residual learning to deep neural networks. A ResNet stacks multiple residual blocks, each containing two or three convolutional or fully-connected layers with a skip connection. For digital predistortion, ResNet-inspired designs enable training networks with 10+ layers to model complex PA nonlinearities.

• Bottleneck Blocks: Use 1x1 convolutions to reduce and then restore dimensionality, making very deep networks computationally feasible
• ResNet-50/101/152: Proven depth configurations that demonstrate residual learning scales effectively to hundreds of layers
• Application to DPD: Enables deep predistorter networks that capture high-order nonlinearities without training instability

The optimization failure mode that residual learning directly solves. In very deep networks, gradients become exponentially smaller as they are backpropagated through successive layers, effectively preventing weight updates in early layers. Residual connections provide an alternative gradient path that preserves signal magnitude.

• Exponential Decay: Gradient magnitude shrinks multiplicatively with each layer in standard feedforward networks
• Residual Solution: The additive identity path ensures a minimum gradient of 1.0 flows backward
• Impact on DPD: Allows training of predistorter networks deep enough to model PA memory effects spanning hundreds of samples

A counterintuitive phenomenon where adding more layers to a neural network increases training error, not just test error. This is not overfitting—the deeper network performs worse on the training data itself. Residual learning resolves this by ensuring deeper models can always fall back to shallower representations.

• Not Overfitting: Training accuracy degrades, distinguishing it from the bias-variance tradeoff
• Solution Space Nesting: Residual blocks ensure the deeper model's solution space contains the shallower model's solutions
• DPD Relevance: Critical for building predistorters deep enough to capture long-term thermal memory effects without sacrificing short-term accuracy

A precursor architecture to residual networks that introduced gated skip connections. Highway networks use learned gating mechanisms—transform gates and carry gates—to control the flow of information along skip paths, blending the original input with the transformed output rather than simply adding them.

• Gating Mechanism: T(x) controls how much of the transformed output passes through; C(x) = 1 - T(x) controls the skip connection
• Relationship to Residuals: Residual networks are a special case where both gates are always 1 (no gating)
• Computational Overhead: Gating adds parameters, making pure residual connections more efficient for DPD hardware implementation

A refined residual block design where batch normalization and ReLU activation are applied before the weight layers, rather than after. This ordering creates a direct, clean identity path from input to output through the entire network, further improving gradient flow and regularization.

• Ordering: BN → ReLU → Conv/FC → BN → ReLU → Conv/FC, with the skip connection bypassing both
• Full Identity Path: Information can propagate directly from any layer to any subsequent layer without modification
• DPD Training Stability: Pre-activation design enables training of predistorter networks exceeding 100 layers for extreme wideband applications