Data Drift

The most common form of drift, where the distribution of input features P(X) changes, but the conditional relationship P(Y|X) remains stable.

• Example: A credit model trained on applicants aged 20–40 suddenly receives a surge of applicants aged 60+ due to a marketing pivot.
• Mechanism: The model encounters data in regions of the feature space it never learned to generalize over.
• Detection: Use univariate statistical tests like the Kolmogorov-Smirnov test or Population Stability Index (PSI) on individual features.

Occurs when the distribution of the target variable P(Y) changes, but the class-conditional densities P(X|Y) remain fixed.

• Example: A fraud detection model trained on a 1% fraud rate suddenly faces a coordinated attack, spiking the fraud rate to 15%.
• Impact: The model's precision-recall trade-off breaks because the base rate assumption is invalidated.
• Remediation: Adjust the decision threshold using Bayesian updating or recalibrate the model's output probabilities.

The fundamental relationship between inputs and outputs P(Y|X) changes. What was true is no longer true.

• Example: A spam filter trained to block "Nigerian prince" emails fails when spammers switch to AI-generated, grammatically perfect phishing lures.
• Severity: This is the most dangerous form of drift because it invalidates the model's core logic.
• Response: Requires online learning, periodic retraining, or a complete model redesign.

A critical operational distinction for monitoring pipelines:

• Feature Drift: A change in the input data distribution. This is a leading indicator—it signals potential future performance decay before ground truth labels arrive.
• Target Drift: A change in the distribution of the actual outcomes. This is a lagging indicator because labels are often delayed by days or weeks.
• Strategy: Monitor feature drift for real-time alerts; use target drift for post-hoc root cause analysis.

Drift manifests with distinct temporal signatures that dictate the architectural response:

• Sudden Drift: An abrupt shift caused by an external event (e.g., a global pandemic breaking demand forecasting models). Requires immediate model rollback or A/B switching.
• Gradual Drift: A slow, continuous change (e.g., user preference evolution over years). Managed via scheduled retraining pipelines.
• Recurring/Seasonal Drift: Cyclical patterns (e.g., holiday shopping behavior). Addressed by time-aware features or ensembling seasonal sub-models.

A specific engineering failure often mistaken for data drift. It occurs when the feature transformation code in the training pipeline differs from the code in the serving pipeline.

• Example: Training normalizes age using min-max from the training set, but serving uses a fixed [0,100] range.
• Distinction: This is not a change in the world but a bug in the infrastructure.
• Prevention: Use a centralized Feature Store to guarantee identical transformation logic in both environments.

A distinct failure mode where the statistical relationship between input features and the target variable changes. Unlike data drift, the input distribution may appear stable, but the underlying meaning shifts. For example, a 'high value' transaction threshold might change due to inflation, rendering a fraud model's historical patterns obsolete. Key distinction: Data drift is a change in P(X); concept drift is a change in P(Y|X).

A discrepancy between the data processing logic used during model training and the logic used during inference. This is an engineering error, not a statistical phenomenon. Common causes include:

• Feature transformation code that differs between training and production pipelines
• Missing default values for new categories
• Inconsistent timestamp formatting Unlike drift, skew is deterministic and immediately correctable through rigorous feature store governance.

The automated monitoring of data pipelines to detect anomalies and lineage breaks before they degrade downstream model performance. Key pillars include:

• Freshness: Is data arriving on schedule?
• Distribution: Have statistical properties shifted?
• Volume: Has throughput changed unexpectedly?
• Schema: Have field types or structures changed?
• Lineage: What upstream sources are impacted?

A symmetric metric quantifying how much a variable's distribution has shifted between a reference (training) sample and a production sample. Calculated as: PSI = Σ (Actual% - Expected%) × ln(Actual% / Expected%). Industry thresholds:

• PSI < 0.1: No significant shift
• 0.1 ≤ PSI < 0.25: Moderate shift, investigate
• PSI ≥ 0.25: Significant shift, model retraining likely required

A specific type of data drift where the distribution of input features P(X) changes, but the conditional distribution of labels given features P(Y|X) remains constant. This is the most common form of drift in production ML systems. Example: A credit model trained on pre-pandemic income distributions encounters post-pandemic economic conditions. Mitigation strategies include importance weighting and regular retraining on recent data windows.

A centralized repository for storing, managing, and serving consistent machine learning features. It serves as a critical defense against drift by:

• Versioning feature definitions and transformations
• Enforcing consistency between training and serving pipelines
• Monitoring feature distributions over time
• Providing point-in-time correctness for historical retraining Popular implementations include Feast and Tecton.