Overfitting

The most definitive signature of overfitting: training error (e.g., NMSE on the captured dataset) continues to decrease monotonically while validation error on a held-out signal begins to rise. This divergence indicates the model is fitting noise specific to the training capture rather than the underlying amplifier nonlinearity. Monitor both curves during iterative estimation and halt training when validation error reaches its minimum. For ILA architectures, validate on a separate signal realization with different peak-to-average power ratio statistics.

Overfit models exhibit excessively large coefficient magnitudes, particularly in higher-order nonlinear terms. In memory polynomial models, coefficients for high-order kernels (e.g., 7th, 9th, 11th order) grow unbounded as the estimator attempts to fit noise fluctuations. This violates the principle of parsimony—a well-behaved power amplifier model should have smoothly decaying coefficient magnitudes with increasing nonlinearity order. Monitor the L2 norm of the coefficient vector during training; a sharp increase signals overfitting.

The operational test: apply the extracted predistorter to a new signal not present in the training set. An overfit DPD will show excellent Adjacent Channel Leakage Ratio (ACLR) on the training signal but degraded ACLR on any other waveform. This occurs because the predistorter has learned the specific spectral pattern of the training signal's noise rather than the amplifier's true distortion characteristic. Always validate with at least two distinct signal types (e.g., LTE and NR test models).

Overfitting is often preceded by numerical instability in the least squares solution. The condition number of the input correlation matrix R = X^H X grows large (typically > 10^8), indicating that the basis functions are nearly linearly dependent for the given training data. This ill-conditioning amplifies noise sensitivity. Mitigation requires Tikhonov regularization (ridge regression) by adding a diagonal loading term λI, where λ is selected via cross-validation or the L-curve criterion.

An overfit predistorter exhibits out-of-band noise amplification when driven with a clean signal. Analyze the predistorter output spectrum without the power amplifier in the loop: overfit models inject high-frequency noise components that were present in the training residuals. This is particularly visible in wideband DPD systems where the linearization bandwidth approaches the Nyquist rate. A properly regularized model should produce a smooth predistorter output spectrum without spurious noise peaks.

A robust model should produce similar coefficient sets when trained on slightly different captures of the same amplifier. An overfit model shows high variance in extracted coefficients across repeated measurements under identical conditions. Quantify this by computing the coefficient standard deviation across multiple training runs with different noise realizations. Large variance in high-order terms indicates the estimator is fitting measurement noise rather than deterministic amplifier behavior.