Drift Detection

Drift detection relies on formal statistical tests to determine if observed changes are significant or due to random noise. Common methods include:

• Kolmogorov-Smirnov (KS) Test: A non-parametric test measuring the maximum distance between two cumulative distribution functions, ideal for continuous features.
• Chi-Squared Test: Assesses differences in categorical feature distributions by comparing observed and expected frequencies.
• Wasserstein Distance: Quantifies the minimum 'work' required to transform one distribution into another, offering a more geometrically intuitive measure than KS for high-dimensional data. These tests output a p-value; a value below a threshold (e.g., 0.05) triggers a drift alert.

The Population Stability Index (PSI) is the industry-standard metric for quantifying distributional shift, especially in financial services and credit risk modeling. It works by:

• Binning the values of a feature from a reference (training) dataset and a production dataset.
• Calculating the proportion of records in each bin for both samples.
• Computing a symmetric measure of the difference between these proportions, weighted by a logarithmic ratio. A PSI below 0.1 indicates no significant shift, 0.1–0.25 suggests moderate drift, and above 0.25 signals a major distribution change requiring immediate investigation.

Univariate tests on individual features can miss drift that only manifests in the joint distribution of multiple features. Multivariate methods detect these subtle, systemic shifts:

• Domain Classifier: A model is trained to distinguish between samples from the reference and production datasets. High classifier accuracy (e.g., AUC > 0.7) indicates the distributions are easily separable, confirming drift.
• Maximum Mean Discrepancy (MMD): A kernel-based method that compares the means of two distributions in a high-dimensional reproducing kernel Hilbert space, capable of detecting changes in correlations between features. This approach is critical for high-dimensional models like deep neural networks where feature interactions are complex.

Beyond input features, monitoring the output distribution of a model's predictions is a direct proxy for performance when ground truth labels are delayed or unavailable. Key techniques include:

• Tracking the mean, variance, and quantiles of prediction scores over time windows.
• Comparing the current prediction distribution to a baseline using the same statistical tests applied to input features (e.g., KS test, PSI).
• For classification models, monitoring the proportion of predictions for each class. A sudden drop in a minority class prediction rate can signal concept drift. A stable prediction distribution does not guarantee no drift, but a shifting one is a definitive red flag.

Static drift thresholds often generate false positives due to natural cyclical patterns (e.g., hourly or seasonal traffic). A robust system uses:

• Sliding Windows: Comparing the current production window (e.g., last hour) against a fixed reference window, or using a cumulative window that grows over time.
• Adaptive Thresholds: Dynamically adjusting alert thresholds based on historical volatility. For instance, using a rolling z-score where an alert fires only if the current drift metric exceeds 3 standard deviations of its own recent history.
• Grace Periods: Suppressing alerts for a defined period after a known change, such as a new model deployment or a major marketing campaign, to avoid unnecessary noise.

Detecting drift is the first step; diagnosing its source and impact is the operational goal. A mature system provides:

• Feature-Level Attribution: Ranking features by their individual drift score (e.g., PSI) to immediately pinpoint which inputs are shifting the most.
• Correlation with Business Metrics: Overlaying drift events on dashboards with operational metrics like click-through rate or conversion rate to assess business impact.
• Segmented Analysis: Drilling down into specific user segments (e.g., by geography or device type) to determine if drift is global or isolated to a particular cohort, which guides targeted retraining strategies.

The phenomenon where the statistical properties of the target variable change over time. Unlike data drift, this focuses on the relationship between inputs and the prediction target.

• Real-world example: A fraud detection model trained during a recession may fail when spending patterns normalize.
• Detection method: Monitor prediction distributions using the Population Stability Index (PSI) or divergence metrics like KL-Divergence.
• Impact: Directly degrades model accuracy as the fundamental mapping learned becomes obsolete.

A shift in the distribution of input features between training and production. The model sees data unlike anything it was trained on.

• Real-world example: An e-commerce recommender trained on desktop traffic suddenly receives 90% mobile traffic with different session lengths.
• Detection method: Two-sample statistical tests like the Kolmogorov-Smirnov (KS) test or monitoring the Population Stability Index (PSI) per feature.
• Key distinction: Data drift does not always cause model degradation if the shifted features are not predictive, but it is a leading indicator of risk.

The gradual decline in a model's predictive performance over time due to environmental changes. Drift detection is the diagnostic tool; model decay is the disease.

• Synonyms: Model staleness, model rot.
• Causes: Unaddressed concept drift, data drift, or shifts in the competitive landscape.
• Mitigation: Automated retraining pipelines triggered by a performance degradation threshold are the standard defense against decay.

A symmetric metric quantifying how much a variable's distribution has shifted from a reference (training) period to a monitoring (production) period.

• Interpretation: PSI < 0.1 indicates no significant shift; 0.1-0.25 is a moderate shift; > 0.25 is a significant shift requiring investigation.
• Application: Commonly applied to both individual input features (data drift) and model prediction scores (concept drift).
• Formula: PSI = sum( (Actual% - Expected%) * ln(Actual% / Expected%) ) across binned distributions.

The automated MLOps practice of regularly retraining models with fresh data, the direct operational response to a drift alert.

• Architecture: An automated retraining pipeline ingests new data, validates it, trains a new model candidate, evaluates it against the champion, and deploys if performance improves.
• Deployment patterns: Often uses Champion/Challenger or Canary Deployment to safely introduce the retrained model.
• Goal: To close the loop between drift detection and remediation, minimizing the time a degraded model serves predictions.

A discrepancy between the data processing code paths used during training and inference, causing a form of drift that is engineered, not environmental.

• Common cause: A feature transformation is implemented differently in a batch training pipeline (e.g., Spark) than in the real-time serving stack (e.g., Python).
• Prevention: The principle of Offline/Online Consistency mandates using identical feature engineering logic in both environments, often enforced by a Feature Store.
• Detection: Logging and comparing feature values at training time vs. serving time for the same entity.