Overfitting

The hallmark diagnostic of overfitting is a divergence between training and validation metrics. The predistorter achieves exceptionally low Normalized Mean Squared Error on the training dataset—often below -50 dB—but exhibits significantly worse performance on unseen test signals. This gap indicates the model has memorized specific training samples rather than learning the smooth, continuous PA transfer function. Engineers should monitor both metrics during coefficient estimation and halt training when validation error begins to rise while training error continues to decrease.

Overfitting is directly correlated with model capacity exceeding the information content of the training data. Key indicators include:

• Memory polynomial order too high: Nonlinearity orders beyond 7-9 and memory depths exceeding 3-4 taps often fit noise
• Neural network over-parameterization: Hidden layers with hundreds of neurons when the PA's true nonlinearity requires far fewer parameters
• Volterra series kernel explosion: Including high-order cross-terms that capture spurious correlations rather than physical distortion mechanisms The parsimony principle applies: the simplest model that adequately captures the PA behavior generalizes best.

An overfit predistorter becomes brittle to changes in signal statistics that differ from the training waveform. Symptoms include:

• Peak-to-average power ratio dependency: Excellent linearization for the specific PAPR used during training, but degraded ACLR for signals with different crest factors
• Bandwidth configuration fragility: Performance collapses when signal bandwidth changes from the training condition
• Modulation scheme over-specialization: Predistorter tuned to 256-QAM fails on 64-QAM or OFDM waveforms with different subcarrier configurations This occurs because the model has learned spurious correlations between specific signal features and distortion products that don't generalize across modulation formats.

Overfit DPD models exhibit excessively large or unstable coefficient magnitudes that amplify measurement noise rather than canceling distortion. Diagnostic signs include:

• Coefficient norm explosion: The L2 norm of the predistorter coefficient vector grows large as the model fits noise components
• High coefficient variance across training runs: Small changes in training data produce dramatically different coefficient sets
• Numerical ill-conditioning: The condition number of the regression matrix becomes extremely high, indicating the solution is sensitive to input perturbations Tikhonov regularization or ridge regression directly addresses this by penalizing large coefficient magnitudes, constraining the solution space to physically plausible values.

An overfit model captures the specific thermal state and aging condition of the PA during training rather than the invariant nonlinear behavior. This manifests as:

• Temperature drift failure: NMSE degrades significantly as the PA junction temperature changes from the training condition
• Aging compensation breakdown: Performance deteriorates over weeks or months as transistor characteristics shift
• Device-to-device variation sensitivity: A predistorter trained on one PA unit fails when deployed on another unit of the same model due to manufacturing tolerances Robust DPD requires training data that spans the expected operating envelope of temperature, frequency, and power levels, combined with online adaptation to track slow-varying changes.

A distinctive signature of overfitting in frequency-domain analysis is the appearance of unnatural spectral nulls or holes in the linearized output spectrum. These artifacts occur because:

• The model has learned to cancel specific frequency components present in the training signal's distortion profile
• When a new signal with different spectral content is applied, the predistorter over-compensates at certain frequencies while under-compensating at others
• The resulting ACLR may appear acceptable in aggregate but exhibits non-uniform spectral regrowth that violates emission mask requirements at specific offsets Cross-validation across multiple signal realizations during training helps identify and prevent these frequency-selective overfitting artifacts.

A stabilization technique that adds a penalty term to the least squares cost function to prevent overfitting. By constraining the magnitude of the DPD coefficients, it suppresses the model's sensitivity to noise in the observation path. This is mathematically equivalent to ridge regression and is essential when the condition number of the data matrix is high, indicating an ill-posed inverse problem.

A model selection framework that partitions the captured PA input-output data into distinct training and validation sets. The predistorter is trained on one subset and evaluated on the unseen hold-out set. A diverging NMSE between the training and validation errors is the primary diagnostic for overfitting, indicating the model is memorizing specific training artifacts rather than the true inverse nonlinearity.

The process of determining the optimal nonlinearity order and memory depth of a predistorter. Excessively high polynomial orders or deep memory taps allow the model to fit high-frequency noise, leading to poor generalization. Information criteria like AIC and MDL balance model fit against complexity, penalizing the addition of superfluous parameters that do not significantly improve the error.

The true measure of a DPD model's performance, defined as the expected error on all possible signals, not just the training waveform. Overfitting is characterized by a low training error but a high generalization error. In practice, this manifests as excellent ACPR correction on the lab signal but degraded performance when the modulation scheme or channel configuration changes in the field.

A regularization technique used in iterative SGD or RLS training where the coefficient update process is halted before convergence to the minimum of the training error. By monitoring the validation error during training, the algorithm stops when the validation error begins to increase, even if the training error is still decreasing. This prevents the model from entering the overfitting regime.

The most fundamental defense against overfitting. Training a predistorter on a single-tone or narrowband signal will not capture the full dynamic nonlinearity of the PA. A robust model requires a training dataset that spans the expected operational envelope:

• Varying signal bandwidths
• Different PAPR distributions
• Multiple carrier configurations
• A range of average input power levels