CUSUM

CUSUM operates on sequential data, analyzing each new observation in real-time as it arrives. Unlike batch methods, it doesn't require a fixed window size. The algorithm maintains a running cumulative sum of deviations, making it highly sensitive to small, sustained shifts that might be missed by point-in-time tests like the Z-score.

• Key Mechanism: It accumulates evidence over time: S_t = max(0, S_{t-1} + (x_t - μ_0 - k)).
• Advantage: Excellent for online monitoring of production data streams, manufacturing processes, or financial metrics where early detection is critical.

The primary strength of CUSUM is its ability to detect small but persistent mean shifts—often as small as 0.5 to 2 standard deviations. It achieves this by accumulating small deviations that individually would not trigger an alert. A single large outlier has less impact than a consistent, subtle bias.

• Contrast with Shewhart Charts: While Shewhart charts are good for detecting large single-point shifts, CUSUM excels at identifying gradual process drift.
• Parameter 'k': The reference value (k), often set to half the smallest shift one wishes to detect, controls sensitivity. A smaller k makes the chart more sensitive to tiny shifts.

Standard CUSUM implementations use two one-sided statistics to monitor for shifts in both positive and negative directions from the target mean (μ_0).

• Upper CUSUM (C+): Accumulates evidence for an increase in the process mean: C+t = max[0, x_t - (μ_0 + K) + C+{t-1}].
• Lower CUSUM (C-): Accumulates evidence for a decrease in the process mean: C- t = max[0, (μ_0 - K) - x_t + C-{t-1}].
• Decision Rule: An alarm is triggered when either C+_t or C- _t exceeds a predefined decision interval (H). This allows for comprehensive monitoring of process degradation or improvement.

CUSUM performance is defined by its Average Run Length (ARL), which is the expected number of samples before a false alarm (ARL0) or before detecting a true shift of a given size (ARL1). The decision interval (H) is the control limit chosen to achieve a desired ARL0.

• Design Principle: Engineers select H based on an acceptable false positive rate. A larger H results in fewer false alarms but slower detection of real shifts.
• Performance Metric: ARL provides a direct, interpretable measure of the algorithm's speed and reliability, linking statistical design to operational requirements.

CUSUM can be implemented in two equivalent forms:

• Tabular CUSUM: The modern, computational form described in other cards. It uses the recursive formulas for C+ and C- and a fixed threshold H. This is the standard implementation in software.
• V-Mask CUSUM: A graphical procedure where a V-shaped mask is overlaid on a plot of cumulative sums. If any previous point lies outside the arms of the V, a change is signaled. While visually intuitive, the tabular form is preferred for automated systems.

In modern data pipelines, CUSUM is a foundational algorithm for statistical process control (SPC) of data quality metrics.

• Use Cases: Detecting silent failures like gradual data drift in feature distributions, sustained increases in null rates, creeping latency in pipeline execution, or subtle drops in record counts.
• Integration: It is often deployed as a lightweight, stateful monitor within data observability platforms, triggering alerts when cumulative evidence of degradation passes a threshold, allowing remediation before downstream models or reports are affected.

Changepoint detection is the process of identifying points in time-series or sequential data where the statistical properties, such as the mean, variance, or slope, change abruptly. CUSUM is a classic algorithm for this task, specifically optimized for detecting small, persistent shifts in the process mean. Other methods include:

• PELT (Pruned Exact Linear Time): An optimal partitioning method.
• Binary Segmentation: A recursive search algorithm.
• Bayesian Changepoint Detection: Uses probabilistic models to infer changepoints. These techniques are critical for monitoring industrial processes, financial markets, and sensor networks.

Statistical Process Control (SPC) is a method of quality control that uses statistical methods to monitor and control a process. CUSUM is a key SPC chart, alongside:

• Shewhart Control Charts (X-bar, R charts): Best for detecting large shifts.
• EWMA (Exponentially Weighted Moving Average) Charts: Sensitive to smaller shifts, similar to CUSUM, but uses a weighted average. SPC aims to ensure a process operates predictably and within defined limits. CUSUM's strength is its cumulative memory, allowing it to detect small, sustained drifts that Shewhart charts might miss.

Exponential smoothing is a time series forecasting method that applies exponentially decreasing weights to past observations. While used for prediction, its residuals (differences between forecasts and actuals) are a common input for anomaly detection. Key variants include:

• Simple Exponential Smoothing: For data with no trend or seasonality.
• Holt's Linear Trend Method: Captures trend.
• Holt-Winters Method: Captures both trend and seasonality. Unlike CUSUM, which monitors the raw process mean, exponential smoothing-based detection monitors deviations from a dynamically updated forecast, making it powerful for non-stationary data.

A Z-score (or standard score) is a statistical measurement that describes a data point's relationship to the mean of a group of values, expressed in terms of standard deviations. It's calculated as (x - μ) / σ. While CUSUM cumulatively sums deviations, the Z-score is a point-in-time measure used for univariate outlier detection.

• A common rule flags points where |Z-score| > 3 as outliers.
• It assumes the data is roughly normally distributed. Z-score is simple and fast but lacks CUSUM's sequential memory, making it less sensitive to small, gradual shifts.

Concept drift is a phenomenon in machine learning where the statistical properties of the target variable a model is trying to predict change over time in unforeseen ways, degrading model performance. CUSUM can be applied to monitor model prediction errors or performance metrics (like accuracy) to detect such drifts.

• Real Drift: Change in P(Y|X), the conditional distribution.
• Virtual Drift: Change in P(X), the input feature distribution (also called covariate shift). Detecting concept drift triggers model retraining or adaptation, making CUSUM a valuable tool in Continuous Model Learning Systems.

Local Outlier Factor (LOF) is a density-based algorithm for unsupervised anomaly detection. It calculates the local density deviation of a given data point relative to its neighbors. Unlike CUSUM, which is designed for sequential/temporal data, LOF is for multivariate, non-sequential data.

• Points with a significantly lower density than their neighbors are considered outliers.
• It can detect outliers in clusters of varying densities. While CUSUM excels at detecting mean shifts in a stream, LOF is used for finding anomalous individual observations in static, multi-dimensional datasets.