Data Drift

The most common form of data drift, where the distribution of independent variables P(X) changes but the conditional relationship P(Y|X) remains stable.

• Example: A demand forecasting model trained on pre-pandemic shopping patterns encounters a permanent shift in online traffic volume, even though the relationship between traffic and purchases stays the same.
• Impact: The model's predictions become miscalibrated because it is extrapolating into regions of the feature space it was never trained on.
• Detection: Use the Kolmogorov-Smirnov test or Population Stability Index (PSI) to compare training and production feature distributions.

Occurs when the distribution of the target variable P(Y) changes independently of the input features. This is common in classification tasks but also affects demand forecasting when product mix shifts.

• Example: A retailer's sales mix shifts from 70% low-margin essentials to 60% high-margin discretionary items due to a seasonal trend, altering the aggregate demand distribution.
• Key Distinction: The input features may look identical, but the base rate of the outcome has fundamentally changed.
• Mitigation: Apply importance reweighting or recalibrate model outputs using production label data.

A gradual, continuous shift in feature distributions driven by long-term trends, seasonality, or slow environmental changes rather than abrupt events.

• Example: A 3% year-over-year increase in mobile traffic share slowly shifts the distribution of device-type features, degrading a model trained on a static snapshot from 18 months ago.
• Why It's Dangerous: Temporal drift is insidious because the degradation is incremental and often goes unnoticed until a quarterly audit reveals a significant accuracy drop.
• Monitoring Strategy: Track the Hellinger distance between reference and production windows on a weekly cadence to catch slow-moving shifts.

Drift can manifest at the granularity of individual features or across the entire multivariate distribution, and these require different detection strategies.

• Univariate Drift: A single feature, such as avg_session_duration, shifts due to a UX change. Detected with KS tests or Jensen-Shannon divergence per feature.
• Multivariate Drift: The joint distribution P(X1, X2, ..., Xn) changes even when marginal distributions appear stable. For example, the correlation between time_on_site and cart_adds breaks down.
• Detection: Use Maximum Mean Discrepancy (MMD) or train a domain classifier to distinguish between reference and production samples.

Effective drift monitoring requires defining a reference window (the baseline distribution from training data) and a production window (the current live data) for statistical comparison.

• Sliding Window: Compare the reference distribution against a rolling window of the last N days. Sensitive to recent shifts but noisy.
• Fixed Cumulative Window: Compare against all production data since deployment. More stable but slow to detect abrupt changes.
• Best Practice: Use a two-window approach—a short window for alerting and a long window for trend confirmation—to balance sensitivity and false positive control.

A critical operational distinction: drift is a persistent statistical change, while an anomaly is a transient outlier. Confusing the two leads to unnecessary model retraining or missed degradation.

• Anomaly: A Black Friday traffic spike is a known, temporary event. The distribution will revert. Do not retrain.
• Drift: A post-COVID permanent shift to curbside pickup changes the fulfillment_method feature distribution indefinitely. Retraining is required.
• Operational Rule: Trigger a retraining pipeline only when a drift metric exceeds a threshold for K consecutive detection windows, filtering out ephemeral anomalies.

A change in the fundamental relationship between input features and the target variable. Unlike data drift, the input distribution may appear stable, but the mapping to the output has changed.

• Example: A promotional discount that historically drove a 3x demand lift now only drives 1.5x due to market saturation
• Impact: Model predictions become systematically biased even when input data looks normal
• Detection: Requires monitoring prediction error against ground truth, not just input statistics

A specific type of data drift where the distribution of input features P(X) changes, but the conditional distribution P(Y|X) remains stable.

• Example: A forecasting model trained on pre-pandemic shopping patterns encounters post-pandemic basket compositions
• Key distinction: The underlying demand logic is intact, but the model sees unfamiliar input ranges
• Mitigation: Importance-weighting training samples or online retraining with recent data

The degradation of pre-computed features in a feature store that no longer reflect current reality, often caused by pipeline failures or delayed updates.

• Example: A 'customer lifetime value' feature that hasn't been recalculated in 30 days, missing recent high-value purchases
• Symptom: Model serves predictions using outdated inputs, producing drift-like errors
• Prevention: Feature freshness SLAs with automated monitoring and backfill triggers

A discrepancy between the data processing logic used during model training and the logic used during online inference, causing feature values to diverge.

• Example: Training pipeline normalizes price using a global mean; serving pipeline uses a rolling 7-day mean
• Root cause: Separate code paths for offline and online feature engineering
• Solution: Shared feature computation libraries and rigorous validation of feature parity

The gradual decay of a deployed model's predictive accuracy simply due to the passage of time, even without detectable drift in inputs or concepts.

• Example: A demand forecasting model trained 18 months ago no longer captures subtle shifts in seasonal buying patterns
• Measurement: Track prediction error over a rolling window and trigger retraining when it exceeds a threshold
• Strategy: Scheduled retraining cadences combined with performance-based triggers

Statistical techniques used to automatically identify when data drift has occurred in production pipelines.

• Population Stability Index (PSI): Measures distribution shift for categorical and binned continuous features
• Kolmogorov-Smirnov Test: Non-parametric test comparing two continuous distributions
• Wasserstein Distance: Measures the 'earth mover's distance' between distributions, sensitive to both location and shape changes
• Best practice: Run multivariate drift detection across feature groups, not just univariate checks