Normalized Mean Squared Error (NMSE)

NMSE normalizes the squared Frobenius norm of the error matrix by the squared norm of the true channel. This scale invariance ensures the metric is independent of the absolute received signal power. A model achieving -20 dB NMSE on a strong signal and -20 dB on a weak signal demonstrates identical relative reconstruction fidelity, unlike unnormalized MSE which would fluctuate with path loss. This is critical for fair benchmarking across different UE distances and link budgets.

For a true channel matrix H and an estimate Ĥ, NMSE is defined as:

NMSE = ||H - Ĥ||²_F / ||H||²_F

where ||·||_F denotes the Frobenius norm. The result is typically expressed in decibels (dB) via 10 * log10(NMSE). A perfect estimate yields -∞ dB. In practice, values range from -10 dB (poor) to -30 dB (excellent) for neural estimators. An NMSE > 0 dB indicates the estimator performs worse than simply guessing the zero matrix.

NMSE is particularly informative in massive MIMO systems where the channel exhibits angular domain sparsity. Because the metric normalizes by the total channel energy, it heavily penalizes errors in reconstructing the few dominant multipath components that carry most of the signal power. A model might accurately estimate noise-like low-energy paths but still yield a poor NMSE if it misses the primary spatial cluster. This aligns the metric with the practical goal of accurate beamforming and precoding.

While NMSE is dominant, it is often reported alongside cosine similarity and normalized correlation coefficient.

• Cosine Similarity: Measures the angular alignment between the vectorized true and estimated channels. Sensitive to phase errors but ignores magnitude scaling.
• NMSE: Penalizes both magnitude and phase mismatches, providing a holistic error measure.
• BER/Spectral Efficiency: End-to-end system metrics. A 1 dB improvement in NMSE does not guarantee a linear gain in throughput, as the impact depends on the precoding algorithm used downstream.

In the CsiNet literature and subsequent neural CSI feedback research, NMSE is the primary loss function and evaluation criterion. The seminal CsiNet paper demonstrated a ~5-10 dB NMSE improvement over compressive sensing baselines like LASSO at equivalent compression ratios. Modern transformer-based architectures (e.g., Transformer CSI) push NMSE below -25 dB even at aggressive compression rates of 1/16 or lower, making them viable for practical 3GPP feedback channels.

NMSE must be interpreted with caution in specific scenarios:

• Low SNR Regimes: At very low signal-to-noise ratios, the normalization factor ||H||²_F is dominated by noise power, artificially suppressing the NMSE and giving a misleadingly optimistic score.
• Averaging Domain: NMSE averaged over many channel realizations can mask catastrophic failures on rare, difficult channel types (e.g., deep fades). Reporting the cumulative distribution function (CDF) of NMSE is a more robust practice.
• Complex vs. Real: Computing NMSE on real and imaginary parts separately versus on the complex magnitude can yield different rankings between models.

The fundamental complex-valued matrix that NMSE quantifies the reconstruction accuracy of. CSI captures the combined effects of scattering, fading, and power decay on a wireless link.

• Represented as an N_t × N_r matrix for MIMO systems
• NMSE measures the Frobenius norm of the error between estimated and true CSI
• Accurate CSI is prerequisite for precoding, beamforming, and link adaptation

A deep learning model trained to infer CSI from received pilot signals, often achieving significantly lower NMSE than classical Least Squares (LS) or Minimum Mean Square Error (MMSE) estimators.

• Architectures include convolutional neural networks, transformers, and autoencoders
• Learns to exploit spatial-frequency correlations that model-based methods miss
• NMSE is the primary training loss function for these networks

Architectures that natively operate on complex numbers rather than separating IQ components into real-valued channels, preserving the magnitude and phase relationships critical for low NMSE.

• Uses complex convolution, complex batch normalization, and complex activation functions
• Avoids the information loss inherent in real-valued decomposition
• Demonstrated to achieve lower NMSE than equivalent real-valued networks on CSI tasks

A model-driven deep learning technique that maps iterative optimization algorithms (like ISTA for sparse recovery) into neural network layers, combining classical signal processing structure with learnable parameters.

• Each layer corresponds to one iteration of the original algorithm
• Achieves faster convergence and lower NMSE than purely data-driven or purely model-based approaches
• Provides interpretable intermediate representations unlike black-box networks

The property exploited by compressed sensing and deep learning to achieve low NMSE with minimal pilot overhead. In massive MIMO, multipath components concentrate in a small number of discrete angles.

• Channel matrix becomes sparse in the DFT domain
• Enables compressive recovery from undersampled measurements
• NMSE degrades when the sparsity assumption breaks in rich scattering environments