Overfitting

The hallmark of overfitting is a significant divergence between training and validation performance metrics. The model achieves an exceptionally low Normalized Mean Square Error (NMSE) on the training waveform—often below -45 dB—but exhibits a degradation of 5–10 dB when evaluated on a held-out validation dataset or a different modulation scheme. This gap indicates the model has fit the specific noise realization rather than the smooth, underlying amplifier transfer function. Cross-validation with disjoint datasets is the primary diagnostic tool for detecting this discrepancy.

Overfitted models exhibit coefficient vectors with excessively large magnitudes and high variance. In memory polynomial or Volterra series models, higher-order nonlinear terms and deep memory taps acquire inflated weights as the estimator struggles to fit random noise fluctuations. These large coefficients cancel each other out on the training data but produce wildly inaccurate predictions on new inputs. The condition number of the regression matrix skyrockets, and the model becomes numerically unstable. Regularization techniques like ridge regression directly penalize this coefficient bloat.

A model extracted using a specific training waveform—such as a 20 MHz LTE signal with a particular Peak-to-Average Power Ratio (PAPR)—may perform adequately on that exact stimulus but fail catastrophically when the amplifier is driven with a different modulation format, bandwidth, or power level. An overfitted model has learned the statistical idiosyncrasies of the training signal rather than the device's invariant nonlinear behavior. True generalization requires the model to maintain consistent NMSE and Adjacent Channel Leakage Ratio (ACLR) prediction accuracy across diverse test signals.

When an overfitted model is used to predict the amplifier's output spectrum, it often introduces spurious spectral components that do not exist in the physical device. The model may attempt to replicate a specific noise floor shape or a transient interference spike present only in the training capture. In the time domain, the predicted waveform may exhibit non-physical, high-frequency oscillations between sample points as the model interpolates noise rather than the smooth nonlinear characteristic. This is particularly visible in the error vector magnitude (EVM) across the band.

Overfitting is frequently caused by selecting a nonlinearity order or memory depth that far exceeds the physical requirements of the amplifier. A GaN Doherty amplifier may only require a 9th-order nonlinearity and 3 memory taps, but an overfitted model might use a 15th-order polynomial with 10 taps. Information-theoretic criteria such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) penalize this unnecessary complexity. A well-regularized model will show that adding more parameters yields diminishing or negative returns on validation performance.

When an overfitted forward model is inverted to create a digital predistorter (DPD) using an indirect learning architecture, the noise memorized during extraction is amplified in the predistortion function. This results in a DPD lookup table or polynomial that introduces out-of-band emissions rather than suppressing them. The post-distortion error spectrum may show elevated noise shoulders that were not present in the original amplifier output. This is a critical failure mode in production DPD systems where robust generalization is mandatory.

The fundamental tension in model selection between the error from overly simplistic assumptions (high bias) and the error from excessive sensitivity to fluctuations in the training data (high variance). Overfitting represents the extreme high-variance end of this spectrum, where the model captures noise instead of the true signal. The goal of model extraction is to find the optimal complexity that minimizes total error.

A technique that adds a penalty term to the cost function during coefficient extraction to prevent overfitting. By constraining the magnitude of model parameters, regularization discourages the model from fitting noise.

• L2 Regularization (Ridge Regression): Penalizes the sum of squared coefficients, shrinking all parameters smoothly.
• L1 Regularization (Lasso): Penalizes the sum of absolute coefficients, forcing some to exactly zero for automatic feature selection.

A model validation technique that partitions captured data into distinct training and validation sets. The model is extracted using only the training set, and its performance is then evaluated on the unseen validation set. A significant drop in performance on the validation set compared to the training set is the primary diagnostic for overfitting. This confirms the model has memorized the training data rather than learning the underlying amplifier physics.

The process of determining the optimal complexity—such as nonlinearity order and memory depth—of a behavioral model. This directly addresses the bias-variance tradeoff. A model with too few parameters underfits, while one with too many overfits. Information criteria like the Akaike Information Criterion (AIC) provide a quantitative metric that penalizes model complexity, guiding the selection of the most parsimonious model that generalizes well.

A numerical state where the correlation matrix of basis functions is nearly singular, causing coefficient estimates to be highly sensitive to measurement noise. In an ill-conditioned problem, tiny perturbations in the data lead to wildly different model parameters—a phenomenon often confused with, or exacerbating, overfitting. Principal Component Analysis (PCA) is a common technique to orthogonalize basis functions and mitigate this instability.

A carefully designed stimulus signal used to excite the power amplifier and capture its full nonlinear dynamic range during model extraction. A poorly designed waveform that does not sufficiently excite all operational modes can lead to a model that appears to fit the limited training data perfectly but fails catastrophically on real-world signals. A robust, spectrally rich waveform is a first line of defense against overfitting.