Hyperparameter Tuning

Defines the depth of the neural network predistorter. A single hidden layer may suffice for weakly nonlinear PAs, but deep architectures with 3-5 hidden layers are often required to model strong nonlinearities with long-term memory effects in GaN Doherty amplifiers. Excessive depth increases inference latency and risks vanishing gradients during training. Residual connections can mitigate this, enabling depths of 8+ layers for wideband mmWave applications.

Controls the model capacity or expressive power of each layer. Typical configurations range from 10 to 50 neurons per layer for RVTDNN predistorters. Too few neurons cause underfitting and poor ACLR improvement; too many lead to overfitting on training signals and increased computational cost. A common heuristic is to use a tapering structure—more neurons in early layers, fewer in later layers—to create an information bottleneck that learns compact PA representations.

The step size for gradient descent updates during training. Typical values range from 0.001 to 0.01 for Adam optimizers in DPD applications. A rate too high causes divergent training where the loss oscillates; too low results in impractically slow convergence. Learning rate scheduling—such as reducing the rate by a factor of 0.5 when validation loss plateaus—is standard practice for fine-tuning predistorter coefficients near convergence.

The number of I/Q sample pairs processed before updating network weights. Typical batch sizes range from 64 to 512 samples. Smaller batches introduce gradient noise that can help escape local minima but slow training. Larger batches provide more stable gradient estimates but require more GPU memory. For wideband signals with 100 MHz+ bandwidth, batch sizes must balance memory constraints against the need to capture sufficient temporal context for memory effect modeling.

The number of delayed signal taps used to capture PA memory effects. Memory depth M typically ranges from 2 to 10 taps, corresponding to memory spans of tens to hundreds of nanoseconds. Insufficient depth fails to model long-term thermal trapping effects in GaN HEMTs. Excessive depth introduces redundant inputs that increase model complexity without meaningful linearization gain. The optimal depth is often determined via autocorrelation analysis of the PA output error signal.

The highest polynomial degree K used in envelope-dependent basis functions feeding the neural network. Typical values range from 5 to 11 for capturing strong compression and AM/PM distortion. Odd-order terms (3rd, 5th, 7th) dominate spectral regrowth near the carrier; even-order terms contribute to harmonic distortion. The order must be high enough to suppress adjacent channel power below regulatory limits but kept minimal to avoid numerical instability in coefficient estimation.

The strategy for dynamically adjusting the step size during gradient descent optimization. A high initial learning rate enables fast convergence, while decay schedules (step, exponential, cosine annealing) refine the solution in later epochs. For PA linearization, an improperly tuned learning rate can cause the model to oscillate around the optimal predistorter coefficients or converge to a poor local minimum, directly degrading Adjacent Channel Leakage Ratio (ACLR).

The number of I/Q training samples processed before updating the neural network weights. Small batch sizes (e.g., 32-128) introduce beneficial gradient noise that helps escape saddle points but increase training time. Large batch sizes (e.g., 1024+) provide more accurate gradient estimates but risk sharp minima that generalize poorly to unseen signal bandwidths. The choice directly impacts GPU memory utilization during model extraction.

The architectural hyperparameters defining the network's capacity: number of hidden layers (depth) and neurons per layer (width). Deeper networks with more neurons can model highly complex PA nonlinearities and long memory effects but are prone to overfitting and increase inference latency. A typical RVTDNN predistorter might use 2-3 hidden layers with 20-50 neurons each, balancing linearization performance against real-time FPGA throughput constraints.

The nonlinear function applied to each neuron's output, enabling the network to learn complex PA distortion curves. ReLU offers fast computation but can suffer from 'dying neurons'. Tanh and sigmoid provide smooth saturation regions that naturally model AM/AM compression but are prone to vanishing gradients. For complex-valued networks, specialized activations like modReLU or cardioid functions preserve phase information critical for predistortion.

Hyperparameters controlling model complexity to prevent memorization of measurement noise. L1 regularization (Lasso) drives irrelevant weights to zero, performing automatic feature selection for the predistorter. L2 regularization (Ridge) penalizes large weight magnitudes, encouraging smoother nonlinear mappings. Dropout rate (e.g., 0.2-0.5) randomly deactivates neurons during training, forcing the network to learn redundant representations that generalize better across varying signal PAPR.

The number of tapped delay lines (TDLs) used to capture the PA's temporal memory effects. This hyperparameter defines how many past I/Q samples are fed into the neural network input layer. Insufficient memory depth fails to model long-term thermal trapping effects in GaN amplifiers, while excessive depth introduces irrelevant features that increase computational complexity. Typical values range from 3 to 10 taps, corresponding to memory spans of 30-100 ns for 100 MHz bandwidth signals.