Data Drift

The most common form of data drift, covariate shift, occurs when the distribution of independent variables $P(X)$ changes, but the conditional distribution $P(Y|X)$ remains constant.

• Example: A credit model trained on pre-pandemic income data encounters a post-pandemic population with systematically lower FICO scores.
• Impact: The model applies correct logic to a new reality, but the input space has shifted, leading to unexpected rejection rates.
• Detection: Measured via Population Stability Index (PSI) or Kullback-Leibler Divergence between training and production feature histograms.

Also known as label shift, this occurs when the distribution of the target variable $P(Y)$ changes, but the likelihood $P(X|Y)$ remains fixed.

• Example: A disease prediction model trained during an epidemic sees a sudden drop in disease prevalence in production.
• Impact: The model's precision collapses because the base rate of the positive class has fundamentally changed.
• Correction: Requires calibration adjustment using production label feedback, often via importance re-weighting of the training likelihoods.

A gradual degradation where specific features lose predictive power over time due to evolving user behavior or external market conditions.

• Example: A recommendation engine relying on a 'holiday season' feature becomes stale when consumer shopping patterns shift to year-round micro-seasons.
• Monitoring: Track the information value (IV) or Shapley value variance of individual features across time windows.
• Mitigation: Implement feature freshness metrics and automated retraining triggers when a feature's statistical weight drops below a defined threshold.

While univariate drift checks individual features, multivariate drift detects changes in the joint distribution or correlation structure between features.

• Example: The correlation between 'account age' and 'transaction volume' inverts in production, breaking a critical interaction term in a logistic regression model.
• Detection: Use Maximum Mean Discrepancy (MMD) or domain classifier-based drift detection, which learns to distinguish training from production samples.
• Significance: Univariate checks can pass while the underlying data topology has completely warped, making this a critical silent failure mode.

A deliberate, malicious manipulation of input data to exploit model vulnerabilities, distinct from natural statistical drift.

• Example: Fraudsters systematically alter transaction velocity and merchant category codes to mimic legitimate patterns, bypassing a fraud detection model.
• Detection: Requires adversarial robustness testing and monitoring for inputs that fall in high-confidence but low-density regions of the training manifold.
• Response: Combine drift monitoring with outlier detection and adversarial training techniques to harden the production pipeline against adaptive attackers.

It is critical to distinguish data drift from concept drift. Data drift is a change in $P(X)$; concept drift is a change in $P(Y|X)$.

• Data Drift: The input 'average purchase amount' increases. The model still correctly identifies high-value customers.
• Concept Drift: The same 'average purchase amount' no longer correlates with customer loyalty due to inflation.
• Diagnosis: Use performance monitoring (accuracy, F1-score) alongside distribution tests. If distributions shift but performance holds, it's pure data drift. If performance degrades, concept drift is likely present.