Statistical Process Control (SPC)

Control charts plot a model performance metric over time against a center line (historical mean) and control limits (typically ±3 sigma). When a metric breaches these limits, it signals a statistically significant shift rather than random noise. Common charts include:

• X-bar charts: Monitor the mean of a metric across batches
• R-charts: Track the range or variance within subgroups
• p-charts: Monitor proportions, such as fraud detection rate per hour
• CUSUM charts: Accumulate deviations to detect small, persistent shifts earlier than Shewhart charts

SPC's core distinction separates variation into two categories:

• Common cause variation: Inherent, random fluctuation within a stable process. Reacting to this noise causes tampering—unnecessary retraining that destabilizes the system
• Special cause variation: Assignable, non-random shifts caused by external factors like data pipeline failures, schema changes, or genuine fraud pattern evolution Only special cause variation warrants investigation and corrective action. This prevents alert fatigue and preserves MLOps team bandwidth.

Beyond simple limit breaches, the Western Electric Rules define pattern-based triggers that indicate process instability:

• Rule 1: Any single point beyond 3-sigma limits
• Rule 2: Two out of three consecutive points beyond 2-sigma on the same side
• Rule 3: Four out of five consecutive points beyond 1-sigma on the same side
• Rule 4: Eight consecutive points on the same side of the center line (a run)
• Rule 5: Six points in a row steadily increasing or decreasing (a trend) These rules detect subtle drift before catastrophic failure occurs.

Process capability indices quantify how well a model's performance meets defined specifications:

• Cp: Measures potential capability—the ratio of specification width to process spread. High Cp indicates the model's natural variation fits within acceptable bounds
• Cpk: Measures actual capability, accounting for centering. A model may have high Cp but low Cpk if its mean performance has drifted off-target
• Pp and Ppk: Long-term capability indices that include all sources of variation over extended periods In fraud detection, specification limits might define minimum acceptable recall or maximum tolerable false positive rate.

Rational subgrouping organizes inference requests into meaningful batches to maximize sensitivity to drift:

• Time-based subgroups: Group predictions by hour or minute to detect diurnal fraud pattern shifts
• Transaction-type subgroups: Separate wire transfers, card-not-present, and ACH transactions into distinct charts
• Score-band subgroups: Monitor high-risk and low-risk segments independently Poor subgroup selection can mask drift by mixing heterogeneous populations. The goal is to maximize variation between subgroups when drift occurs while minimizing variation within subgroups.

SPC operates as the statistical decision layer within a broader continuous evaluation framework:

• Metric ingestion: Performance metrics (AUC, precision, recall) stream into SPC monitors from the evaluation pipeline
• Automatic limit calculation: Control limits are computed from a stable baseline period, not arbitrary thresholds
• Alert routing: Special cause signals trigger retraining pipelines or model rollback procedures
• Feedback loop closure: Delayed ground truth labels update control chart baselines as chargebacks resolve This statistical rigor prevents the common anti-pattern of retraining on every metric fluctuation.

The core visualization tool of SPC, plotting a metric over time against a center line (mean) and control limits (typically ±3 sigma). In MLOps, a Shewhart chart tracks model accuracy or prediction drift. A point outside the limits signals an assignable cause of variation, triggering an alert. Western Electric rules add sensitivity by detecting non-random patterns like runs of 7 points on one side of the center line.

SPC distinguishes between two types of process variation:

• Common Cause: Inherent, random noise within a stable system. No action needed; tampering increases variance.
• Special Cause: An assignable, non-random event outside the system's normal behavior. Requires immediate investigation. In fraud models, a seasonal spending spike is common cause; a sudden data pipeline failure causing null features is special cause.

Indices measuring how well a process output fits within specification limits. Cpk assesses short-term capability relative to the process spread, while Ppk evaluates overall long-term performance. In ML monitoring, these metrics quantify if your model's error rate consistently stays within business-defined tolerance thresholds. A Cpk < 1.33 indicates the process is not capable of meeting specifications.

A control chart variant that applies exponentially decreasing weights to older observations, making it more sensitive to small, persistent shifts in the process mean than standard Shewhart charts. In drift detection, EWMA charts excel at identifying gradual concept drift—such as fraudsters slowly modifying their behavior—that might be missed by abrupt-change detectors like the Page-Hinkley test.

A sequential analysis technique that accumulates deviations from a target value over time. CUSUM is exceptionally sensitive to small, sustained shifts in the process mean. It plots the cumulative sum of residuals, triggering an alarm when the accumulation exceeds a decision interval. In production ML, CUSUM detects subtle performance degradation long before aggregate metrics like Population Stability Index (PSI) signal an alert.

The principle of selecting samples so that variation within a subgroup comes from a homogeneous source (common cause), while variation between subgroups captures special causes. In ML monitoring, rational subgroups might be hourly batches of predictions. Choosing the wrong subgroup size—such as daily aggregates that mask intraday drift—renders control charts ineffective and leads to silent failures.