Statistical Process Control (SPC)

A control chart is the primary visualization tool in SPC, plotting a monitored metric (e.g., prediction error rate, feature mean) over time against calculated control limits. These limits, typically set at ±3 standard deviations from a central line (the process mean), define the expected range of common-cause variation. Points outside the limits or forming non-random patterns signal special-cause variation, indicating potential concept drift or data drift. In ML, control charts are applied to model outputs and key input features to separate natural fluctuation from significant degradation.

The central line represents the expected value or mean of a process metric during a stable, in-control period, often established from a reference window of training or validation data. Control limits (Upper Control Limit - UCL, Lower Control Limit - LCL) are statistically derived boundaries, calculated as the central line ± (a multiplier * the process standard deviation). The standard multiplier of 3 creates a 99.7% confidence interval under a normal distribution. These limits are not specification limits but define the bounds of inherent process noise; violations suggest the process mean or variance has changed, triggering a drift investigation.

SPC distinguishes between two types of process variation:

• Common-cause variation is inherent, random noise within a stable system. It is always present and defines the baseline performance band within control limits.
• Special-cause variation is assignable, non-random deviation caused by an external factor. In ML, this corresponds to concept drift, data drift, or pipeline failures. The goal of SPC-based monitoring is not to eliminate common-cause variation but to detect the emergence of special-cause variation, which signifies a fundamental change requiring intervention like triggered retraining.

Beyond a single point outside control limits, SPC uses heuristic run rules to detect non-random patterns that indicate an unstable process. Common rules, formalized as the Western Electric rules, include:

• A point outside the 3-sigma control limits.
• 2 out of 3 consecutive points beyond the 2-sigma warning limits.
• 4 out of 5 consecutive points beyond the 1-sigma limit.
• 8 consecutive points on one side of the central line.
• 6 consecutive points steadily increasing or decreasing (a trend). These rules increase sensitivity to small, persistent shifts (detected by CUSUM-like logic) and reduce the false positive rate from reacting to single anomalies.

While control charts monitor stability, process capability analysis quantifies how well a stable process meets specification requirements. In ML, this translates to assessing whether a model's performance metrics (e.g., accuracy, F1-score) during a stable period consistently meet business-defined specification limits (e.g., accuracy > 95%). Indices like Cp and Cpk measure the potential and actual capability of the process relative to these specs. A stable process (in control) with poor capability indicates the model is consistently underperforming by design, necessitating architectural change, not just drift detection.

SPC is adapted for ML by applying its principles to model-specific signals:

• Supervised SPC: Monitoring the model's error rate or loss over time, as in the Drift Detection Method (DDM).
• Unsupervised SPC: Monitoring the distribution of input features or model confidence scores using statistics like the mean or variance, plotted on control charts.
• Multivariate SPC: Using techniques like Hotelling's T² statistic to monitor correlations between multiple features simultaneously. The reference window establishes the initial central line and control limits. The test window (recent data) is continuously compared against this baseline. A key challenge is balancing detection delay with false alarms.

A control chart is the primary graphical tool of SPC, used to monitor a process metric over time against statistically derived control limits. In ML monitoring, key metrics like prediction error, data distribution statistics, or model confidence scores are plotted. The chart visually signals special-cause variation (drift) when points fall outside the control limits or exhibit non-random patterns.

• Upper/Lower Control Limits (UCL/LCL): Calculated from historical data, typically at ±3 standard deviations from the process mean.
• Central Line: Represents the historical average or expected value of the monitored metric.
• Application: Used to track model accuracy, prediction latency, or feature means for unsupervised drift detection.

CUSUM is a sequential analysis technique used in SPC for detecting small, persistent shifts in the mean of a process. It works by accumulating the sum of deviations between observed values and a target value. A significant cumulative sum signals a change point.

• Mechanism: Calculates S_t = max(0, S_{t-1} + x_t - μ - k), where μ is the target mean and k is a sensitivity parameter. A drift is flagged when S_t exceeds a threshold h.
• Advantage: More sensitive to small, sustained drifts than Shewhart control charts.
• ML Use Case: Ideal for online drift detection of metrics like classification error rate or the mean of a critical feature in a data stream.

The Page-Hinkley test is an online change detection algorithm based on sequential analysis, closely related to CUSUM. It monitors the cumulative difference between observed values and their running average, flagging a change when this difference exceeds a dynamic threshold.

• Calculation: Maintains a cumulative variable m_t and the minimum value M_t seen so far. A drift is detected when (m_t - M_t) > λ.
• Characteristic: Designed to detect abrupt changes in the mean of a Gaussian signal.
• Application: A core algorithm in streaming ML libraries for real-time concept drift detection, often applied to model loss or error sequences.

A stationarity test is a statistical hypothesis test used to determine if the properties of a time series (mean, variance, autocorrelation) are constant over time. Non-stationarity is a primary indicator of underlying drift in the data-generating process.

• Common Tests: Augmented Dickey-Fuller (ADF) test (for unit roots), Kwiatkowski–Phillips–Schmidt–Shin (KPSS) test.
• Null Hypothesis: The time series is stationary.
• ML Relevance: Applied to model inputs or error sequences. Rejecting stationarity suggests the model is operating in a non-stable environment, necessitating drift adaptation or investigation.

Two-sample hypothesis testing is the statistical foundation for batch drift detection. It tests the null hypothesis that two samples (e.g., a reference training window and a recent production batch) are drawn from the same distribution.

• Core Question: "Has the data distribution changed?"
• Common Tests: Kolmogorov-Smirnov (KS) test for any distributional difference, Student's t-test for differences in means, Chi-squared test for categorical data.
• Process: 1. Define reference and test datasets. 2. Choose a test statistic. 3. Calculate p-value. 4. Reject the null hypothesis (signal drift) if p-value < significance level α (e.g., 0.05).

Process capability analysis in SPC uses indices like Cp and Cpk to quantify how well a stable process performs relative to its specification limits. In ML, this translates to measuring a model's operational envelope and its robustness to inherent data variation.

• Capability Indices: Cp = (USL - LSL) / (6σ) measures potential capability. Cpk accounts for process centering.
• ML Analogy: Specification limits can be defined as acceptable ranges for model metrics (e.g., accuracy > 90%, latency < 100ms).
• Use Case: Establishing a baseline for normal model operation. A significant drop in Cpk indicates the process (model performance) is drifting out of its capable range, even if it remains statistically "in control."