Drift Detection

Concept drift occurs when the statistical relationship between the model's input features and the target variable it predicts changes over time. The model's learned mapping becomes outdated, even if the input data distribution remains stable.

• Real-world example: A fraud detection model trained on historical transaction patterns may degrade as criminals develop new techniques, changing the fundamental 'concept' of what constitutes fraud.
• Detection methods: Monitor for a sustained drop in performance metrics (e.g., accuracy, F1-score) or use statistical tests on the joint distribution of inputs and labels, if labels are available with low latency.

Data drift, also known as covariate shift, is a change in the statistical distribution of the input features served to a production model compared to the data it was trained on. The underlying concept being predicted remains the same.

• Real-world example: An e-commerce recommendation model trained on user data from North America may experience data drift when launched in Europe, where user demographics and purchasing habits differ.
• Detection methods: Use statistical tests like Population Stability Index (PSI), Kullback-Leibler (KL) divergence, or Kolmogorov-Smirnov tests to compare feature distributions between a training reference dataset and recent production data.

Label drift refers to a change in the distribution of the target variable (the label) the model is trying to predict. This is a specific, observable form of concept drift where the change manifests in the ground truth.

• Real-world example: A model predicting customer churn may experience label drift if the overall market churn rate increases due to a new competitor, changing the prior probability of the 'churn' class.
• Detection methods: Monitor the distribution of labels in newly acquired ground truth data. This requires a reliable, timely feedback loop to obtain true labels, which can be a limiting factor.

Prediction drift is a change in the statistical distribution of a model's output predictions over time. It is a proxy signal that can indicate underlying concept or data drift, as changes in inputs or relationships will manifest in the outputs.

• Real-world example: A credit scoring model that gradually shifts from outputting a balanced range of scores to predominantly 'high-risk' scores indicates a shift in its decision boundary, warranting investigation.
• Detection methods: Monitor the distribution of prediction scores or classes. This is often the most practical signal, as model outputs are always available without needing immediate ground truth labels.

Upstream data drift occurs due to changes or failures in the data pipelines and pre-processing steps that feed the model, rather than a change in the raw source data's inherent distribution. This includes schema changes, broken feature encoders, or corrupted data.

• Real-world example: A sensor's firmware update changes the data format from integer to float, causing a feature engineering step to fail and pass default values, altering the input distribution.
• Detection methods: Implement data observability checks at various pipeline stages, monitoring for schema violations, sudden spikes in null values, or out-of-range values for engineered features.

A type of model decay where the fundamental relationship between the input features and the target variable changes over time, rendering the model's learned mapping obsolete. Unlike data drift, which concerns input distribution shifts, concept drift indicates that the same input now maps to a different output.

• Example: A credit scoring model trained during economic stability may fail when a recession changes the relationship between income and default risk.
• Detection: Monitors for sustained degradation in model performance metrics (e.g., accuracy, F1-score) that cannot be explained by input data shifts alone.

A change in the statistical properties of the input data served to a production model compared to the data it was trained on. This is a primary cause of model performance decay.

• Types: Covariate Shift (change in feature distribution P(X)) and Label Shift (change in target distribution P(y)).
• Detection Methods: Statistical tests (e.g., Kolmogorov-Smirnov, Population Stability Index), divergence measures (e.g., Jensen-Shannon, Kullback-Leibler), and monitoring descriptive statistics (mean, variance).

A predefined, automated condition that initiates the retraining of a production model. It is the operational link between drift detection and model correction.

• Common Triggers:
  • Performance metrics fall below a defined threshold.
  • Statistical drift scores exceed a configured limit.
  • A scheduled time interval elapses (time-based).
  • Volume of new data reaches a certain size.
• Implementation: Part of an MLOps pipeline, where a triggered event launches a new training job, validation, and potential deployment.

An operational pattern where models are automatically and periodically retrained on fresh data to maintain predictive accuracy. This creates a closed feedback loop between monitoring and model improvement.

• Architecture: Integrates drift detection, retraining triggers, and CI/CD for ML.
• Challenges: Requires robust data pipelines, experiment tracking, and strategies to prevent catastrophic forgetting.
• Goal: To transition from static, decaying models to adaptive, self-correcting systems.

A deployment strategy where a new challenger model processes live production requests in parallel with the current champion model, but its predictions are logged and not served to users. This is a critical technique for validating model performance and detecting drift in a risk-free manner.

• Use Case for Drift: The challenger model, potentially trained on newer data, can be monitored for performance on live traffic. Discrepancies between champion and challenger predictions can signal emerging concept drift.
• Advantage: Provides a real-world performance baseline without impacting user experience.

A benchmark metric or model performance level established under controlled conditions, used as a reference point for evaluating production models and detecting decay.

• Establishment: Typically defined during model validation on a held-out test set that reflects the expected production data distribution.
• Role in Drift Detection: Serves as the comparison point for ongoing model performance monitoring. Sustained deviation from the baseline triggers investigation for drift.
• Dynamic Baselines: In continuous retraining systems, the baseline may be updated periodically to reflect the latest validated model performance.