Normalized Mean Square Error

NMSE normalizes the mean squared error by the power of the reference signal, making it independent of signal amplitude. This allows direct comparison of model fidelity across different power levels, signal types, and amplifier classes without recalibration.

• Formula: NMSE = 10·log₁₀( Σ|y_meas - y_model|² / Σ|y_meas|² )
• Expressed in decibels (dB) for intuitive interpretation
• A value of -40 dB means the error power is 0.01% of the signal power

NMSE can be computed over different frequency regions to separately evaluate model performance for in-band signal fidelity and out-of-band spectral regrowth prediction.

• Time-domain NMSE: Captures total error across the full bandwidth, including both in-band distortion and adjacent channel leakage
• Frequency-domain NMSE: Evaluates error within specific spectral regions, critical for ACLR compliance verification
• A model may achieve excellent in-band NMSE while failing to predict out-of-band behavior accurately

NMSE is mathematically related to other key RF metrics, providing a unified framework for model validation.

• Error Vector Magnitude (EVM): For a properly normalized signal, EVM² ≈ 10^(NMSE/10) when NMSE is expressed in dB
• Adjacent Channel Power Ratio (ACLR): Poor out-of-band NMSE directly correlates with inaccurate ACLR prediction by the behavioral model
• NMSE below -35 dB is typically required for models used in digital predistortion applications targeting -50 dBc ACLR

NMSE computed on independent test data that was not used during model extraction is the definitive measure of generalization capability.

• Training NMSE: Measures fit to the data used for coefficient estimation; can be misleadingly optimistic
• Test NMSE: Evaluated on unseen signals with different statistics (e.g., different PAPR or bandwidth)
• A large gap between training and test NMSE indicates overfitting, where the model has memorized noise rather than learning the underlying amplifier dynamics
• Cross-validation with multiple signal realizations provides statistical confidence in the reported NMSE

NMSE computation requires careful alignment of measured and modeled signals to prevent timing misalignment from artificially inflating the error.

• Sub-sample delay estimation and correction is essential before NMSE calculation
• Coherence-based alignment using cross-correlation peak detection is standard practice
• For memory-effect models, the error signal must be computed after the model has reached steady-state, discarding initial transient samples
• Numerical precision: double-precision floating-point is recommended to avoid quantization errors in the squared summation

Acceptable NMSE thresholds vary by use case and signal characteristics.

• Behavioral modeling for system simulation: -30 to -35 dB typically sufficient
• Digital predistortion model extraction: -38 to -45 dB required for competitive linearization performance
• Wideband signals (100+ MHz): Achieving sub -40 dB NMSE is challenging due to increased memory effect complexity
• mmWave applications: NMSE targets may be relaxed to -30 dB due to measurement noise and hardware impairments at higher frequencies