Normalized Mean Squared Error (NMSE)

NMSE normalizes the mean squared error by the power of the reference signal, making it independent of absolute signal amplitude. This allows engineers to compare DPD performance across different power levels, modulation schemes, and amplifier types without recalibration.

• Formula: NMSE = 10 log₁₀( E[|y_ideal - y_measured|²] / E[|y_ideal|²] )
• Expressed in decibels (dB) — more negative values indicate better linearization
• Typical targets: -35 dB to -45 dB for commercial wireless systems
• Eliminates ambiguity from unnormalized MSE comparisons

NMSE serves as a proxy metric for both spectral and in-band signal quality. A well-trained DPD model that minimizes NMSE simultaneously improves Adjacent Channel Power Ratio (ACPR) and Error Vector Magnitude (EVM), though the relationships are nonlinear.

• ACPR improvement: Lower NMSE directly reduces spectral regrowth into adjacent channels
• EVM improvement: Residual in-band distortion decreases as NMSE improves
• NMSE provides a single scalar objective for multi-objective DPD optimization
• Enables unified cost function design for gradient-based coefficient estimation

NMSE is sensitive to ill-conditioning in the estimation problem. When the input signal has low dynamic range or the PA model is overparameterized, the denominator in the normalization can approach zero, causing numerical instability.

• Regularization techniques like Tikhonov regularization stabilize NMSE computation
• High condition number in the data covariance matrix amplifies estimation noise
• NMSE degradation often signals overfitting in neural network-based DPD models
• Cross-validation with held-out data prevents misleadingly optimistic NMSE values

Standard NMSE provides a broadband error metric that averages distortion across the entire signal bandwidth. For wideband 5G and mmWave signals, frequency-selective NMSE analysis reveals band-edge degradation that broadband metrics mask.

• Sub-band NMSE: Computes error power within specific frequency bins
• Identifies memory effect compensation quality at band edges
• Critical for wideband signal linearization where PA memory effects dominate
• Guides model structure selection between memory polynomial and neural network architectures

The difference between training NMSE and validation NMSE is a diagnostic indicator of model generalization quality. A large gap signals overfitting to training-specific noise or artifacts rather than learning the true PA nonlinearity.

• Small gap (< 2 dB): Good generalization, model captures underlying physics
• Large gap (> 5 dB): Overfitting, model memorized training data idiosyncrasies
• Monitor NMSE trajectory during online training to detect coefficient drift
• Essential for direct learning architecture (DLA) convergence monitoring

In closed-loop DPD systems, NMSE serves as the real-time cost function driving coefficient updates. Adaptive algorithms like Recursive Least Squares (RLS) and Least Mean Squares (LMS) minimize NMSE iteratively as signal statistics and PA characteristics drift.

• Convergence rate: Time required for NMSE to reach steady-state after a PA state change
• Misadjustment: Excess NMSE beyond the theoretical Wiener solution due to gradient noise
• NMSE monitoring detects thermal memory effects requiring coefficient recalibration
• Burst training uses NMSE thresholds to trigger block updates

A measure of in-band distortion quality. EVM quantifies the magnitude of the difference vector between the ideal reference constellation point and the actual linearized signal point.

• Expressed as a percentage or in dB
• Directly correlates with bit error rate (BER)
• Sensitive to both amplitude and phase errors
• 5G NR requires EVM as low as 1.5% for 256-QAM

The critical regulatory compliance metric for spectral regrowth. ACPR measures the ratio of power leaking into adjacent frequency channels relative to the main channel power.

• Typically specified at defined frequency offsets (e.g., ±5 MHz, ±10 MHz)
• 3GPP mandates ACPR of -45 dBc or better for base stations
• NMSE minimization directly improves ACPR performance
• Poor ACPR causes interference with neighboring carriers

The mathematical objective minimized during DPD coefficient training. While NMSE is standard, cost functions can be weighted to prioritize specific performance aspects.

• Standard NMSE weights all samples equally
• Frequency-weighted NMSE emphasizes out-of-band suppression
• Peak-weighted NMSE targets high-PAPR sample accuracy
• Multi-objective functions balance EVM, ACPR, and PA efficiency simultaneously

The normalization factor in NMSE divides the mean squared error by the input signal power, making the metric scale-invariant and comparable across different signal levels.

• Without normalization, MSE scales with signal amplitude
• Normalization enables fair comparison between different test signals
• Expressed in dB: NMSE(dB) = 10·log₁₀(||y_ideal - y_measured||² / ||y_ideal||²)
• Typical DPD targets: -35 dB to -45 dB NMSE

NMSE serves as the primary convergence indicator during iterative DPD training. Tracking NMSE over training epochs reveals algorithm stability and speed.

• Monotonic NMSE decrease indicates stable convergence
• NMSE plateaus signal the algorithm has reached steady-state
• Sudden NMSE spikes suggest coefficient drift or numerical instability
• Used to trigger early stopping to prevent overfitting to measurement noise

The excess error beyond the theoretical Wiener optimum in adaptive DPD systems. Misadjustment arises from gradient noise in stochastic coefficient updates.

• Defined as: M = (Excess MSE) / (Minimum MSE)
• Higher learning rates increase misadjustment but speed convergence
• LMS algorithms exhibit higher misadjustment than RLS
• Represents the price paid for real-time adaptivity vs. batch optimality