Overfitting

The most definitive sign of overfitting is a decreasing training error (e.g., Normalized Mean Squared Error) concurrent with an increasing validation error on a held-out dataset. During iterative coefficient estimation, monitor both curves. When the validation NMSE begins to rise while the training NMSE continues to fall, the model has crossed the point of generalization and begun fitting noise. This is the bias-variance tradeoff visualized in real-time.

In an overfitted Volterra series, the estimated coefficients for high-order nonlinearity and deep memory taps become pathologically large in magnitude, often with alternating signs. This occurs because the model uses these terms to cancel out noise in the training data rather than representing physical distortion mechanisms. Inspect the coefficient vector for:

• Magnitude spikes at high nonlinear orders (e.g., 7th, 9th order)
• Sign oscillation between adjacent memory taps
• Coefficients exceeding physically plausible bounds for the amplifier's gain compression curve

A model that achieves excellent ACPR on the training signal but fails to suppress spectral regrowth when excited with a different modulation scheme or bandwidth is overfitted. The predistorter has learned the specific spectral characteristics of the training waveform rather than the amplifier's nonlinear transfer function. Validate against a diverse test suite including signals with different PAPR, bandwidth, and modulation formats (QPSK, 64-QAM, 256-QAM) not seen during training.

An ill-conditioned data matrix with a condition number exceeding 40-60 dB indicates that the Volterra basis functions are nearly collinear for the given training data. This numerical instability amplifies estimation variance, causing the coefficient extraction to fit noise. Regularization techniques like LASSO regression or ridge regression can mitigate this, but a persistently high condition number after regularization suggests the model order or memory depth is excessive for the available data diversity.

An overfitted model exhibits high variance: small changes in the training dataset (e.g., slightly different noise realizations or a minor time shift) produce dramatically different coefficient sets. If re-estimating the Volterra kernel on a bootstrap sample of the same signal yields a completely different predistorter response, the model lacks stability. A well-generalized model should produce consistent kernels across minor data variations.

Overfitting is structurally guaranteed when the number of Volterra coefficients approaches or exceeds the number of independent information-bearing samples in the training data. For a memory polynomial with nonlinear order K and memory depth M, the parameter count grows as O(K*M). If training on a short capture with limited amplitude diversity, even a modest K and M can overfit. Use the Akaike Information Criterion (AIC) to penalize unnecessary parameters and select a parsimonious model.

The fundamental modeling dilemma where a Volterra model with too few parameters underfits the data (high bias), while one with too many parameters overfits (high variance). The goal is to find the optimal model complexity that minimizes total error on unseen data, balancing the systematic error from an overly simple model against the sensitivity to noise from an overly complex one.

A model validation technique that partitions measurement data into distinct training and testing sets to ensure the identified Volterra model generalizes well to unseen signals. Common strategies include:

• K-Fold: Data is split into K subsets, training on K-1 and testing on the remainder, rotating through all folds.
• Hold-Out: A single, fixed split into a training set and a test set. This process directly detects overfitting by revealing a significant drop in performance on the test set compared to the training set.

A statistical metric that balances model fit against model complexity, used to select the optimal nonlinear order and memory depth for a Volterra series. The AIC penalizes the number of parameters, providing a quantitative score that helps avoid overfitting. A lower AIC value indicates a better tradeoff, explicitly discouraging the inclusion of unnecessary Volterra kernel terms that merely fit noise.

A modeling approach where the number of active Volterra coefficients is minimized using L1-norm regularization (LASSO) . This technique forces many kernel coefficients to exactly zero, automatically performing model pruning. By retaining only the most statistically significant terms, sparse Volterra models inherently resist overfitting, reducing variance without a substantial increase in bias.

A measure of the sensitivity of the Volterra coefficient solution to errors in the measurement data. A high condition number indicates an ill-conditioned, numerically unstable estimation problem, often caused by highly correlated regressors in an over-parameterized model. This instability is a direct numerical signature of overfitting, where small noise fluctuations in the training data cause massive swings in the estimated kernel coefficients.

The systematic process of decreasing the number of parameters in a Volterra model while preserving its ability to accurately predict the power amplifier's nonlinear behavior. Techniques like Dynamic Deviation Reduction separate static nonlinearity from low-order dynamics, drastically cutting parameters. This directly combats overfitting by simplifying the model structure based on physical insight rather than just statistical pruning.