Model Monitoring

The phenomenon where the statistical properties of the target variable change over time, breaking the fundamental assumption that the future will resemble the past. Unlike data drift, concept drift means the relationship between inputs and outputs has shifted—what once indicated a purchase now signals abandonment.

• Sudden Drift: Abrupt change due to external shock (e.g., COVID-19 lockdowns instantly altering buying patterns)
• Gradual Drift: Slow evolution of user preferences over months or years
• Recurring Drift: Seasonal patterns like holiday shopping behavior that cycle predictably

Detection requires monitoring prediction distributions against ground truth labels as they become available, often using Population Stability Index (PSI) or Kullback-Leibler divergence on prediction scores.

A shift in the distribution of input features between training and production environments. Even if the underlying relationship holds, a model trained on one data regime will produce unreliable predictions when faced with unfamiliar feature values.

• Covariate Shift: Change in P(X) while P(Y|X) remains stable—the most common form
• Multivariate Drift: Joint distribution changes invisible when examining features independently
• Detection Methods: Kolmogorov-Smirnov test for continuous features, Chi-squared test for categorical features, and Wasserstein distance for high-dimensional embeddings

Data drift is often the earliest warning sign of impending model decay, detectable before ground truth labels arrive.

The insidious discrepancy between the data processing code paths used during training and those executing during inference. A feature transformation that normalizes differently, a missing one-hot encoding category, or a timestamp parsed in the wrong timezone can silently corrupt every prediction.

• Feature Engineering Mismatch: Training uses Pandas, serving uses SQL—subtle logic differences compound
• Default Value Handling: Missing values imputed differently in batch vs. real-time pipelines
• Prevention: Implement offline/online consistency by using shared feature libraries or a Feature Store that guarantees identical transformation logic in both environments

This is the most dangerous form of model failure because it produces no obvious errors—just quietly wrong predictions.

A symmetric statistical metric that quantifies how much a variable's distribution has shifted from a reference baseline to a production window. PSI is the workhorse metric for data drift detection in financial services and is increasingly standard in MLOps monitoring stacks.

• Formula: PSI = Σ(Aᵢ - Bᵢ) × ln(Aᵢ / Bᵢ) where A and B are bin proportions
• Interpretation: PSI < 0.1 indicates no significant shift; 0.1–0.25 suggests moderate drift; > 0.25 signals major distribution change requiring investigation
• Application: Calculate PSI on each input feature independently and on prediction score distributions to detect both data and concept drift

PSI is binned by decile or quantile and works best on continuous and high-cardinality categorical features with sufficient volume.

The inevitable, gradual erosion of a model's predictive power over time as the world changes around it. Also called model staleness, this is the symptom that monitoring systems are designed to detect before business impact occurs.

• Causes: Concept drift, data drift, changes in competitive landscape, evolving user behavior, or shifts in the population being served
• Measurement: Track key performance metrics (AUC, precision, recall) on delayed ground truth labels and compare against a performance degradation threshold
• Mitigation: Automated continuous training pipelines that retrain on recent data windows, combined with champion/challenger deployment patterns to validate new models before promotion

Without monitoring, model decay is silent—revenue declines, churn increases, and no alert fires.

The systematic application of statistical hypothesis tests to production data streams to identify when a model's operating environment has diverged from its training baseline. Effective drift detection balances sensitivity (catching real problems) against specificity (avoiding alert fatigue from false positives).

• Univariate Detection: Apply Kolmogorov-Smirnov, Chi-squared, or Jensen-Shannon distance to each feature independently
• Multivariate Detection: Use Maximum Mean Discrepancy (MMD) or domain classifier-based methods to detect joint distribution shifts invisible to univariate tests
• Adaptive Thresholding: Dynamically adjust alert thresholds based on historical false positive rates and business impact tolerance

Modern MLOps platforms like Evidently AI, WhyLabs, and NannyML provide out-of-the-box drift detection with configurable statistical tests and alerting integrations.