Normalized Residual Test

The test operates on a null hypothesis (H₀) that the measurement is good data with zero mean Gaussian noise. It calculates the normalized residual (rᵢⁿ) by dividing the raw residual (zᵢ - hᵢ(x̂)) by its corresponding residual standard deviation (√Rᵢᵢ). If |rᵢⁿ| exceeds a critical threshold (typically 3.0 for 99.7% confidence), the null hypothesis is rejected and the measurement is flagged as bad data. This transforms raw engineering unit errors into dimensionless, comparable test statistics.

The key to normalization lies in the Residual Sensitivity Matrix (S = I - H·G⁻¹·Hᵀ·W), which maps measurement errors to residuals. The diagonal elements (Sᵢᵢ) represent the self-sensitivity of a measurement's residual to its own error. The residual covariance matrix is computed as Rᵣ = S·R, where R is the measurement error covariance. The normalized residual denominator is the square root of the i-th diagonal of Rᵣ. Measurements with Sᵢᵢ close to 1.0 are highly detectable; those near 0.0 are critical measurements whose errors are masked.

In practice, the test is applied iteratively using the Largest Normalized Residual (rₘₐₓ) strategy:

• Compute all normalized residuals after state estimation convergence
• Identify the measurement with the maximum |rᵢⁿ|
• If |rₘₐₓ| > threshold (e.g., 3.0), remove that measurement
• Re-run state estimation and repeat until all |rᵢⁿ| < threshold This sequential elimination prevents smearing, where a single large error distorts the state estimate and corrupts residuals of neighboring good measurements.

A fundamental limitation occurs with critical measurements—measurements whose removal renders the system unobservable. For these, the residual sensitivity Sᵢᵢ = 0, meaning their residual is always zero regardless of error magnitude. The normalized residual test is blind to errors in critical measurements. Similarly, critical k-tuples (sets of k measurements where removal of all k causes unobservability) can mask coordinated errors. This motivates the use of robust estimators like the Least Absolute Value (LAV) or Huber M-Estimator as alternatives.

The detection threshold balances Type I error (false positive: rejecting good data) against Type II error (false negative: missing bad data). Standard thresholds:

• 2.0: ~95.5% confidence, higher false alarm rate
• 2.5: ~98.8% confidence, common in transmission systems
• 3.0: ~99.7% confidence, standard for distribution DSSE
• 3.5: ~99.95% confidence, conservative for critical infrastructure The optimal threshold depends on the measurement redundancy ratio (m/n) and the acceptable risk of false alarms triggering unnecessary field investigations.

When multiple bad data points exist simultaneously, they can interact non-linearly through the residual sensitivity matrix. A large error in one measurement can inflate or suppress the normalized residual of another, a phenomenon called residual masking. The LNR strategy partially addresses this by removing only one measurement per iteration, but conforming bad data (errors that produce small normalized residuals despite being large) can evade detection entirely. Advanced techniques like hypothesis testing identification (HTI) or combinatorial optimization are required for robust multiple bad data identification.

A global hypothesis test that evaluates the sum of squared normalized residuals against a chi-square distribution threshold. While the Normalized Residual Test identifies individual bad data points, the Chi-Square test first determines whether any bad data exists in the measurement set. If the objective function exceeds the critical value for the given degrees of freedom, the null hypothesis of clean data is rejected, triggering individual residual screening.

The standard implementation of the Normalized Residual Test that flags the measurement with the maximum absolute normalized residual as bad data. After removal, the state estimation is re-run and the test repeats iteratively. This sequential elimination approach prevents smearing, where a single gross error corrupts residuals of neighboring clean measurements, causing false positives in a single-pass screening.

The mathematical foundation enabling normalized residual computation. Defined as Ω = R - H·G⁻¹·Hᵀ, where:

• R is the measurement error covariance matrix
• H is the Jacobian matrix
• G is the gain matrix

The diagonal elements of Ω provide the residual variances used to normalize raw residuals. Off-diagonal elements capture residual correlation, which is critical for distinguishing interacting bad data from isolated errors.

The Normalized Residual Test operates under a null hypothesis (H₀) that the measurement is valid with zero mean error, versus an alternative hypothesis (H₁) that it contains a gross error. The test statistic follows a standard normal distribution under H₀. A measurement is flagged when:

• |rN,i| > 3 for 99.7% confidence (3-sigma rule)
• |rN,i| > 2.576 for 99% confidence The threshold selection balances detection sensitivity against false alarm probability.

The matrix S = I - H·G⁻¹·Hᵀ·R⁻¹ maps measurement errors to residuals. Its diagonal elements, called leverage points, indicate how much of a measurement's error appears in its own residual. Measurements with low leverage have errors that spread to other residuals, making bad data harder to detect. The Normalized Residual Test accounts for this by dividing by the residual standard deviation rather than the measurement standard deviation alone.

A limitation of the standard Normalized Residual Test occurs when multiple gross errors exist simultaneously. Correlated residuals can cause:

• Masking: one error hides another by reducing its normalized residual below threshold
• Swamping: a clean measurement appears erroneous due to neighboring bad data Advanced techniques like Hypothesis Testing Identification (HTI) and combinatorial optimization address these cases by testing multiple measurement subsets simultaneously rather than sequentially.