Bad Data Detection

A global hypothesis test applied to the sum of weighted squared residuals from the state estimation solution. The test statistic follows a Chi-Square distribution with degrees of freedom equal to the measurement redundancy.

• Null Hypothesis (H₀): No bad data exists in the measurement set.
• Detection Logic: If the computed objective function exceeds a critical threshold (e.g., χ² at 95% confidence), the null hypothesis is rejected, indicating the presence of at least one gross error.
• Limitation: Identifies that bad data exists but does not locate which specific measurement is corrupted.

The primary method for identifying specific bad measurements after the Chi-Square test flags a problem. Each measurement residual is divided by its corresponding residual standard deviation to compute the normalized residual.

• Largest Normalized Residual (LNR) Test: The measurement with the highest absolute normalized residual is statistically the most likely to be bad data.
• Threshold Logic: If the LNR exceeds a critical value (typically 3.0 for a 3-sigma rule), it is flagged for removal.
• Iterative Process: The suspect measurement is removed and the state estimation is re-run until the Chi-Square test passes.

Bad data detection is formalized as a binary hypothesis test with controlled error probabilities to balance sensitivity against false alarms.

• Type I Error (α): False positive—flagging a valid measurement as bad. Typically set to 1% or 5%.
• Type II Error (β): False negative—failing to detect actual bad data. Minimized by maximizing measurement redundancy.
• Detection Probability: The power of the test increases with larger measurement errors and higher redundancy. A 20% gross error in a measurement with high sensitivity is detected with near certainty.

Bad data originates from multiple points in the measurement-to-control pipeline, each with distinct statistical signatures.

• Sensor Drift: Gradual calibration decay in current transformers (CTs) and potential transformers (PTs) producing biased measurements.
• Communication Noise: Bit errors in SCADA telemetry frames causing sporadic spikes in reported values.
• Time Skew: Mismatched timestamps between asynchronous SCADA scans creating apparent inconsistencies that are not physical.
• Topology Mismatch: An incorrect breaker status in the network model causes the estimator to solve the wrong topology, producing large residuals on multiple adjacent measurements.

An alternative estimation criterion that inherently rejects bad data without requiring iterative detection-removal cycles. Instead of minimizing the sum of squared residuals, LAV minimizes the sum of absolute residuals.

• Automatic Rejection: The LAV estimator naturally places zero weight on outlier measurements, effectively identifying and discarding bad data in a single solution.
• Breakdown Point: LAV can tolerate up to 50% contaminated measurements if redundancy is sufficient, far exceeding WLS robustness.
• Computational Cost: Historically more expensive than WLS, but modern interior-point linear programming solvers have made LAV practical for online distribution system state estimation.

The primary statistical hypothesis test for identifying gross measurement errors. The residual (difference between measured and estimated value) is divided by its standard deviation to create a normalized index. If this index exceeds a predefined statistical threshold (typically 3.0), the measurement is flagged as bad data and removed from the estimation process. This test relies on the assumption that measurement noise follows a Gaussian distribution.

A global detection test that evaluates the sum of squared weighted residuals against a Chi-Square distribution. If the objective function value exceeds a critical threshold determined by the degrees of freedom, the test indicates the presence of bad data somewhere in the measurement set. Unlike the normalized residual test, this method does not identify which specific measurement is erroneous, requiring subsequent largest normalized residual identification.

A robust state estimation criterion that minimizes the sum of absolute residuals rather than squared residuals. This approach automatically rejects outliers by assigning zero weight to measurements with large deviations, effectively performing bad data suppression without iterative re-weighting. LAV is particularly effective in distribution systems where communication noise and sensor drift produce non-Gaussian error distributions.

A maximum-likelihood-type estimator that bridges the gap between WLS and LAV. It applies quadratic weighting to small residuals (maintaining Gaussian efficiency) and linear weighting to large residuals (providing outlier resilience). The transition point is controlled by a tuning constant. This prevents a single bad measurement from corrupting the entire state estimate while preserving statistical optimality for clean data.

A cyber-physical attack vector where an adversary manipulates measurement data in a way that deliberately bypasses conventional bad data detection. By constructing an attack vector that lies in the column space of the Jacobian matrix, the injected errors cancel out in the residual calculation. This corrupts the state estimate without triggering the Chi-Square or normalized residual alarms, requiring advanced measurement redundancy analysis for mitigation.

The engineering practice of ensuring sufficient critical measurement pairs and critical sets exist to detect and identify bad data. A measurement is critical if its removal renders the system unobservable, meaning bad data on that measurement cannot be detected. Redundancy analysis identifies these vulnerabilities and guides the placement of additional sensors or pseudo-measurements to strengthen the bad data detection capability of the state estimator.