Batch Normalization

Batch Normalization directly addresses internal covariate shift—the continuous change in the distribution of layer inputs during training. As preceding layers update their weights, the input distribution to subsequent layers shifts, forcing them to continuously adapt. By normalizing each mini-batch to have a mean of zero and a variance of one, Batch Normalization stabilizes these distributions, allowing each layer to learn independently. This is particularly critical in deep predistorter networks where gradient propagation through many nonlinear layers can become unstable.

After normalization, Batch Normalization introduces two trainable parameters per feature:

• Gamma (γ): A learnable scale factor that restores the layer's representational capacity
• Beta (β): A learnable shift factor that allows the layer to learn the optimal mean

This ensures the network can undo the normalization if it proves suboptimal for the task. For PA linearization, this flexibility allows the network to preserve critical amplitude-dependent phase relationships in the I/Q signal that pure normalization might distort.

By preventing the explosion or vanishing of activations, Batch Normalization enables the use of significantly higher learning rates without divergence. This accelerates convergence by 5-10x in typical predistorter training scenarios. The smoother optimization landscape reduces sensitivity to weight initialization choices, making the training process more robust. For wireless R&D teams iterating on PA models, this translates to faster experimentation cycles and reduced GPU compute costs.

Batch Normalization acts as a mild regularizer by injecting noise through the mini-batch statistics. Since the mean and variance are computed on small, random subsets of the training data, each normalization step introduces slight stochasticity. This prevents the network from overfitting to specific signal characteristics in the training dataset. In PA linearization, this improves model generalization across varying signal bandwidths, modulation schemes, and power levels not seen during training.

Batch Normalization operates differently during training and inference:

• Training: Uses per-mini-batch mean and variance for stochastic normalization
• Inference: Uses running averages of mean and variance accumulated during training via exponential moving average

This distinction is critical for real-time DPD deployment on FPGAs. The inference mode uses fixed statistics, enabling deterministic, low-latency execution without batch dependency. The running statistics must be frozen and exported alongside the trained weights for hardware implementation.

In neural network predistorters, Batch Normalization is typically inserted:

• After linear transformations (fully-connected or convolutional layers)
• Before nonlinear activation functions (ReLU, tanh, or custom activation)

This ordering normalizes the pre-activation values, preventing saturation in activation functions and maintaining healthy gradient flow. For complex-valued neural networks, specialized complex Batch Normalization must handle the correlation between I and Q components, using a 2×2 covariance matrix instead of scalar variance.

The phenomenon Batch Normalization directly addresses. It refers to the change in the distribution of network activations due to parameter updates during training. As weights update, the input distribution to each layer shifts, forcing subsequent layers to continuously adapt. This slows convergence and requires careful weight initialization and lower learning rates. BN mitigates this by normalizing layer inputs to a stable mean and variance.

After normalizing activations to zero mean and unit variance, Batch Normalization introduces two trainable parameters: gamma (γ) for scaling and beta (β) for shifting. This restores the network's representational power. Without these, the layer would be constrained to a fixed distribution. The network can learn to set γ to the standard deviation and β to the mean, effectively recovering the original activation if that is optimal.

Batch Normalization operates differently during training and inference. During training, it uses the mini-batch statistics (mean and variance) for normalization. During inference, it uses a running average of these statistics accumulated over training. This distinction is critical for deployment: inference must be deterministic and independent of batch size, which is often 1 in real-time predistortion systems.

Batch Normalization acts as a mild regularizer. Because the normalization statistics are computed per mini-batch, the activations are subject to noise from the batch's specific composition. This noise prevents the network from overfitting to exact activation values, similar to Dropout Regularization. In practice, adding BN often allows reducing or eliminating Dropout, as the stochasticity introduced by batch statistics aids generalization.

In the context of Digital Predistortion, Batch Normalization enables the use of significantly higher learning rates when training deep RVTDNN or CVNN architectures. By stabilizing the distribution of I/Q signal components through the network, BN prevents exploding or vanishing gradients in deep predistorter models. This is essential for training very deep residual networks that capture subtle PA memory effects across wideband signals.

An alternative normalization technique often compared to Batch Normalization. Instead of normalizing across the batch dimension, Layer Normalization normalizes across the feature dimension for each individual sample. This makes it independent of batch size, which is advantageous for Online Learning scenarios in DPD where samples arrive sequentially. However, BN remains preferred for convolutional architectures processing I/Q tensors.