Distribution Shift

A change in the distribution of the input variables P(X) while the conditional relationship P(Y|X) remains constant.

• Example: A vision model trained on high-resolution studio photos fails on low-light, grainy security camera footage. The mapping of 'object to label' is still correct, but the input pixels look different.
• Mechanism: The model receives data from a region of the feature space it never saw during training, leading to unpredictable extrapolation.
• Detection: Use two-sample statistical tests (like Maximum Mean Discrepancy) between training and serving feature distributions.

A change in the distribution of the output variable P(Y) while the likelihood P(X|Y) remains fixed.

• Example: A disease prediction model deployed during an epidemic. The prevalence of the disease (Y) spikes, but the symptoms (X) given the disease stay the same.
• Impact: The model's prior assumptions about class frequency are violated, breaking calibration and requiring threshold adjustment.
• Correction: Apply importance re-weighting using the ratio of target to source class probabilities.

The most severe shift where the fundamental relationship between input and output P(Y|X) changes. The definition of the target variable itself evolves.

• Example: A fraud detection model where fraudsters change their tactics. The same transaction pattern (X) that was previously 'safe' is now 'fraudulent' (Y).
• Example: A sentiment model where slang evolves. The word 'sick' previously meant 'ill' but now means 'excellent'.
• Mitigation: Requires continuous online learning, windowed retraining, or human-in-the-loop relabeling to adapt to the new reality.

A specific case of label shift where the base rate of the target class changes, but the feature distributions within each class remain identical.

• Scenario: A churn prediction model where the business suddenly runs a retention campaign. The overall churn rate drops, but the profile of a churning customer looks the same.
• Fix: Adjust the decision threshold based on the new prior probability rather than retraining the entire model.
• Formula: Recalibrate posterior probabilities using Bayes' theorem with the updated prior.

A shift driven purely by the passage of time, often seen in consumer behavior, financial markets, and sensor degradation.

• Example: A recommendation engine trained on summer fashion trends failing in winter. The data is not 'wrong,' it is just stale.
• Example: A predictive maintenance model where physical sensors corrode over months, causing a slow, systematic bias in readings.
• Architecture: Implement time-based train/test splits (backtesting) rather than random splits to simulate this degradation accurately before deployment.

A mismatch between the source domain (lab/training) and the target domain (production) due to environmental or hardware changes.

• Example: A speech recognition system trained on US English accents failing on UK regional dialects.
• Example: A medical imaging model trained on MRI scans from a Siemens machine performing poorly on scans from a GE machine due to different sensor physics.
• Solution: Domain adaptation techniques (adversarial training, domain-invariant feature learning) force the model to ignore domain-specific noise.

A specific type of distribution shift where the input feature distribution P(X) changes between training and production, but the conditional target distribution P(Y|X) remains constant.

• Example: A self-driving car model trained on sunny California highways deployed in snowy Michigan. The visual features (X) change, but the rules of the road (Y|X) do not.
• Detection: Monitored via population stability index (PSI) on input features.

A more dangerous form of shift where the relationship between inputs and the target variable P(Y|X) itself changes. The meaning of the data evolves.

• Example: A fraud model where the definition of 'fraudulent behavior' changes as criminals adapt their tactics. The same transaction pattern (X) that was safe yesterday is risky today.
• Mitigation: Requires online learning or frequent retraining pipelines.

Occurs when the distribution of the target class P(Y) changes, but the class-conditional feature distributions P(X|Y) remain static. Often seen in medical diagnosis.

• Example: A disease screening model deployed in a general population clinic vs. a specialist ward. The prevalence (P(Y)) is drastically different, breaking calibrated probability thresholds.
• Fix: Adjusting decision thresholds using Bayes' theorem with the new prevalence rate.

The continuous statistical analysis of production feature distributions against a baseline training snapshot to trigger alerts before model accuracy decays.

• Metrics: Population Stability Index (PSI), Kullback-Leibler Divergence, Wasserstein Distance.
• Architecture: Typically implemented via feature stores that log histograms and compare them using scheduled batch jobs or streaming windows.

The engineering discipline of training models to be robust to distribution shift by design, rather than relying on post-deployment detection.

• Techniques: Invariant risk minimization (IRM), data augmentation with domain randomization, and adversarial training.
• Goal: Learning representations that are stable across environments, so the model does not latch onto spurious correlations.