Overfitting

An overfit model achieves exceptionally low error on training data but exhibits high variance when exposed to new inputs. The model's predictions fluctuate wildly with small changes in the input signal. This is the classic bias-variance tradeoff where the model has essentially zero bias on known data but fails to generalize. In PA modeling, this manifests as a model that perfectly reconstructs the specific training waveform but introduces significant distortion when linearizing a different modulation scheme.

Overfitting is directly correlated with model capacity exceeding the information content of the training data. Key indicators include:

• Too many coefficients: Using a high-order memory polynomial (e.g., nonlinearity order K=15, memory depth M=10) when a simpler model (K=7, M=4) suffices
• Dense Volterra kernels: Retaining cross-terms that model noise rather than physical memory effects
• Deep neural networks: Employing architectures with more parameters than training samples, causing the network to memorize the dataset

The definitive diagnostic for overfitting is a significant divergence between training and validation error. During model extraction, the Normalized Mean Square Error (NMSE) on the training set continues to decrease while the validation NMSE plateaus or increases. This inflection point indicates the model has begun fitting noise. In DPD applications, this is measured using Adjacent Channel Error Power Ratio (ACEPR) on a hold-out signal not used during coefficient estimation.

An overfit model is unstable—small changes in the training dataset produce dramatically different model coefficients. This is particularly problematic in online training algorithms where thermal drift or supply voltage variations introduce new operating conditions. The extracted model lacks robustness and requires frequent re-extraction. In neural network PA models, this manifests as coefficient brittleness where weights have large magnitudes with alternating signs to cancel out noise-specific patterns.

Rather than smoothing through measurement noise, an overfit model treats random fluctuations as deterministic system dynamics. This produces:

• Spurious spectral components in the model output not present in the physical PA
• Non-physical gain curves with sharp inflections that violate the smooth, monotonic nature of real amplifier compression
• Temporal instabilities where the model predicts physically impossible transient responses In complex baseband representation, this appears as erratic AM-AM and AM-PM curves with high-frequency ripples.

Overfitting is mitigated through regularization techniques that constrain model complexity:

• L2 (Ridge) regularization: Adds a penalty proportional to the squared coefficient magnitude, shrinking weights toward zero
• L1 (LASSO) regularization: Promotes coefficient sparsity by driving unnecessary terms to exactly zero
• Pruning algorithms: Remove low-magnitude Volterra cross-terms or neural network connections post-training
• Early stopping: Halting iterative training when validation error begins to rise The optimal regularization strength is determined through k-fold cross-validation.

A metric quantifying the average power of the error signal normalized by the power of the reference signal, expressed in dB. NMSE is the primary fidelity metric for behavioral model validation:

• Values below -30 dB indicate excellent model fit
• Values above -20 dB suggest significant modeling error or overfitting A critical diagnostic is comparing training NMSE against test NMSE. A large gap—where training NMSE is excellent but test NMSE is poor—is the definitive signature of overfitting. The model has memorized training data noise rather than learning the underlying amplifier dynamics.

A model validation metric that measures the prediction error power specifically in the adjacent channels. While NMSE captures overall in-band fidelity, ACEPR assesses how accurately the model predicts spectral regrowth—the out-of-band distortion critical for regulatory compliance. An overfit model may achieve good NMSE on training data but fail to generalize spectral predictions to new signals. Monitoring ACEPR on independent test data reveals whether the model has learned the true nonlinear dynamics or simply memorized the training signal's specific spectral characteristics.

A property of a behavioral model where a significant number of coefficients are zero or near-zero, enabling complexity reduction through pruning without substantial loss of fidelity. Sparse models are inherently less prone to overfitting because they have fewer degrees of freedom. Techniques for promoting sparsity include:

• L1 regularization during extraction
• Orthogonal matching pursuit for greedy coefficient selection
• Principal component analysis of the regressor matrix For generalized memory polynomial models, many cross-term coefficients contribute negligibly to accuracy. Removing them simplifies the model and improves generalization to new signals.

The robustness of a coefficient estimation algorithm against errors caused by finite-precision arithmetic, quantified by the condition number of the data matrix. A high condition number indicates that small input perturbations produce large output variations—a hallmark of an ill-conditioned problem prone to overfitting. In least squares estimation for power amplifier models, correlated regressors from oversampled wideband signals create near-singular matrices. Tikhonov regularization and singular value decomposition with truncation are standard remedies that improve numerical stability and prevent the model from fitting noise.