Control Chart

The Central Line represents the process mean or median, calculated from historical data when the process is in a state of statistical control. It serves as the baseline for the expected performance of the metric being monitored.

• Purpose: Establishes the process center or target value.
• Calculation: Typically the mean (x̄) or median of the reference data.
• Example: In monitoring daily model inference latency, the CL would be the average latency observed during a stable performance period.

Upper and Lower Control Limits are statistically derived boundaries, typically set at ±3 standard deviations from the central line. They define the expected range of common cause (inherent) variation in the process.

• Purpose: Distinguish between common cause and special cause variation. Points outside these limits indicate an out-of-control process.
• Calculation: UCL = CL + (3 * σ), LCL = CL - (3 * σ), where σ is the estimated process standard deviation.
• Key Insight: These are not specification limits or business tolerances; they are purely statistical limits based on observed process behavior.

Upper and Lower Warning Limits are inner boundaries, often set at ±2 standard deviations. They provide an early signal of potential process instability before a point breaches the control limits.

• Purpose: Serve as a secondary, less severe alert zone. Multiple points near or beyond warning limits can signal developing drift.
• Common Rule: Two out of three consecutive points between the warning and control limits suggest the process may be shifting.
• Usage: While not always plotted, they enhance sensitivity for detecting small, persistent shifts in the process mean.

The plotted Data Points are the sequential measurements of the monitored metric (e.g., prediction error, data feature mean), ordered on the Time Axis (e.g., batch number, timestamp). This creates the time-series view essential for trend analysis.

• Purpose: Visualizes the process behavior over time.
• Data Types: Can be individual measurements (I-chart) or subgroup statistics like means (x̄-chart) or ranges (R-chart).
• Critical for Drift: The temporal ordering is what allows for the detection of trends, runs, or cycles that indicate non-random, special cause variation.

The area between the center line and control limits is often divided into three Zones on each side. Zone C is closest to the center line, Zone B is the middle, and Zone A is between the warning and control limits. These zones enable the application of Western Electric Rules.

• Purpose: Provides a structured framework for identifying non-random patterns.
• Rule Example: A run of 8 consecutive points on one side of the center line (in any zone) signals a shift in the process mean.
• Analytical Use: Pattern recognition using zones is more powerful than solely relying on points outside control limits, as it detects subtle, sustained drifts.

Western Electric Rules (or Nelson Rules) are a set of eight pattern-testing heuristics applied to a control chart to detect non-random, special cause variation. They formalize the interpretation of zones and sequences.

• Core Rules: Include 1) Point outside 3σ limits, 2) 9 points in a row on same side of center, 3) 6 points in a row steadily increasing/decreasing (trend), 4) 14 points alternating up and down.
• Purpose: Objectively identify when a process has statistically deviated from its stable, in-control state, triggering investigation.
• Drift Detection: These rules are directly analogous to statistical tests for concept drift or data drift in ML monitoring.

Statistical Process Control (SPC) is a method of quality control that uses statistical techniques to monitor and control a process. In ML, it is adapted to monitor model performance metrics (like accuracy or error rate) and input data distributions over time.

• Core Principle: Distinguishes between common-cause variation (inherent, random noise) and special-cause variation (assignable, significant shifts).
• ML Application: Provides a framework for determining if a drop in model performance is due to expected noise or a genuine concept drift event requiring intervention.
• Key Tools: Includes control charts, process capability analysis, and Pareto charts.

CUSUM (Cumulative Sum) is a type of control chart and sequential analysis technique designed to detect small, persistent shifts in the mean of a monitored process metric.

• Mechanism: It accumulates the sum of deviations of observations from a target value (e.g., a baseline error rate). A small, consistent bias will cause the cumulative sum to drift, triggering an alert.
• Advantage: More sensitive than Shewhart charts (like the basic control chart) for detecting subtle, sustained drifts.
• Use Case: Ideal for online drift detection where early warning of gradual performance degradation is critical.

Change Point Detection is the broader statistical task of identifying specific points in time where the properties of a sequence of observations change significantly.

• Relation to Control Charts: While control charts signal that a process is out of control, change point detection algorithms often aim to identify when the change began.
• Methods: Includes offline (batch) methods like PELT and online methods like the Page-Hinkley test.
• ML Application: Used to pinpoint the exact batch or time after which data drift or concept drift commenced, enabling more targeted model investigation and retraining.

A Stationarity Test is a statistical procedure used to determine if the properties of a time series (such as its mean, variance, and autocorrelation) are constant over time.

• Key Test: The Augmented Dickey-Fuller (ADF) test is a common unit root test for stationarity.
• Connection to Drift: Non-stationarity in model inputs or outputs is a primary indicator of drift. Control charts are a visual and statistical tool for monitoring stationarity.
• Usage: Often applied as a preliminary batch check on historical data before setting up continuous control chart monitoring.

Two-Sample Hypothesis Testing is a core statistical method for determining whether two data samples are drawn from the same underlying probability distribution.

• Foundation for Batch Drift Detection: Tests like the Kolmogorov-Smirnov test (for 1D distributions) or Maximum Mean Discrepancy (MMD) (for high-dimensional data) compare a reference window (training data) with a test window (recent production data).
• Contrast with Control Charts: Provides a batch-based, statistical significance answer, whereas control charts offer continuous, visual process monitoring.
• Common Metrics: p-values from these tests inform whether a detected difference is statistically significant.

Process Capability Analysis is an SPC technique that quantifies how well a stable process performs relative to its specified requirements or tolerance limits.

• Key Indices: Cp and Cpk indices measure the potential and actual capability of a process.
• ML Analogy: In model monitoring, 'specifications' could be minimum acceptable performance thresholds (e.g., accuracy > 95%, latency < 100ms).
• Workflow: First, a control chart establishes the process is in control (stable). Then, capability analysis measures if this stable performance meets business needs. A capable process that later goes 'out of control' signals drift.