Ill-Conditioning

Small perturbations in measurement data—such as thermal noise, quantization error, or minor loop delay misalignment—produce disproportionately large swings in estimated model coefficients. A 0.1% change in a single sample can alter coefficient values by orders of magnitude, rendering the extracted model non-reproducible across repeated captures. This sensitivity violates the fundamental engineering requirement that a model's parameters should be stable given stable underlying physics.

The condition number κ(A) of the regression matrix quantifies ill-conditioning severity. For well-conditioned problems, κ(A) is close to 1. In DPD model extraction, condition numbers exceeding 10⁶ to 10¹² are common when basis functions exhibit high correlation. The condition number directly bounds the amplification of relative errors:

• Rule of thumb: κ(A) ≈ 10ᵏ means up to k digits of precision are lost in the solution
• Double-precision arithmetic (≈16 decimal digits) becomes insufficient when κ(A) > 10¹⁶
• Single-precision FPGA implementations fail at much lower thresholds, around κ(A) > 10⁷

The covariance matrix of the basis function set approaches singularity. Its determinant approaches zero, and eigenvalue decomposition reveals a wide spread between the largest and smallest eigenvalues. This occurs because polynomial basis functions—especially high-order odd terms like x³, x⁵, x⁷—are nearly linearly dependent over the finite amplitude range of practical communication signals. The smallest eigenvalues correspond to directions in parameter space that are effectively unobservable from the measurement data.

Unregularized least-squares solutions produce coefficient vectors with unrealistically large magnitudes that alternate in sign. For example, adjacent memory tap coefficients might oscillate between +10⁴ and -10⁴, even though the underlying amplifier physics demands smoothly decaying memory. These large coefficients cancel each other almost exactly during forward prediction, but the cancellation is fragile—any numerical rounding or truncation destroys the delicate balance and produces garbage outputs. This is a hallmark of overfitting to noise rather than capturing true system dynamics.

A model extracted under ill-conditioned conditions may fit the training waveform with excellent Normalized Mean Square Error (NMSE)—often below -40 dB—yet fail catastrophically when applied to a different signal type. This occurs because the model has memorized the specific correlation structure of the training data's basis functions rather than learning the amplifier's true nonlinear transfer characteristic. The model is said to have high variance in the bias-variance tradeoff: it is exquisitely tuned to one dataset but useless for any other.

The numerical rank of the regression matrix—determined by counting singular values above a tolerance threshold—is significantly lower than the theoretical rank. While a model with 50 basis functions should theoretically span a 50-dimensional space, the effective rank might be only 15-20 due to near-linear dependencies. This rank deficiency means that many coefficient combinations produce essentially identical model outputs, creating an ill-posed inverse problem with infinitely many equally valid solutions in the absence of regularization.