Model Drift Monitoring

The phenomenon where the statistical relationship between input features and the target variable changes over time. In lead time prediction, this occurs when a previously reliable supplier begins experiencing raw material shortages, fundamentally altering the connection between order volume and delivery speed. Unlike data drift, concept drift directly degrades model accuracy even when input distributions remain stable. Detection methods include:

• Monitoring prediction error rates over sliding windows
• Comparing model performance on recent vs. legacy data batches
• Using drift detection algorithms like ADWIN or Page-Hinkley

A shift in the distribution of input features fed into a deployed model, independent of whether the underlying relationship with the target has changed. For example, a sudden shift to air freight from ocean freight due to port congestion represents data drift—the model sees carrier types it was not trained on. Key monitoring techniques:

• Kolmogorov-Smirnov test for continuous feature distribution comparison
• Chi-squared test for categorical feature shifts
• Population Stability Index (PSI) for measuring overall distribution divergence
• Tracking multivariate drift using Maximum Mean Discrepancy (MMD)

Changes in the distribution of model outputs over time, often the first visible symptom of underlying concept or data drift. In lead time forecasting, prediction drift manifests as a systematic shift toward over-optimistic delivery estimates. Monitoring prediction drift is computationally lightweight since it requires only model outputs, not ground truth labels. Critical metrics:

• Tracking the mean and variance of predictions over rolling windows
• Monitoring the rate of extreme predictions (tail behavior)
• Comparing prediction distributions between training and production using Wasserstein distance

A specialized monitoring approach that tracks shifts in SHAP values or other feature importance metrics rather than raw features or predictions. This technique reveals when the model's decision-making logic itself has changed—for instance, when a lead time model suddenly weights carrier type more heavily than seasonality, indicating a structural shift in the supply chain. Implementation steps:

• Compute SHAP values on a reference dataset at deployment time
• Continuously calculate SHAP values on production inference data
• Monitor the rank correlation of feature importance between windows
• Alert when top features change by more than a predefined threshold

Automated conditions that initiate model retraining pipelines when drift exceeds acceptable thresholds. Effective trigger design balances model freshness against computational cost and stability. Common trigger strategies include:

• Performance-based: Retrain when MAPE exceeds a 10% degradation threshold
• Schedule-based: Weekly retraining regardless of drift metrics
• Data-volume-based: Retrain after accumulating a minimum number of new labeled examples
• Hybrid approaches: Combine drift severity scores with business impact assessments before triggering resource-intensive retraining jobs

The challenge of obtaining ground truth labels in production to calculate actual model performance. In lead time prediction, the true delivery date is only known after the shipment arrives—creating a delayed feedback problem where drift may go undetected for weeks. Mitigation strategies:

• Using proxy metrics like shipment milestone check-ins as early indicators
• Implementing performance estimation techniques that predict model accuracy without labels
• Building shadow deployments where new model versions run alongside production to compare outputs
• Establishing data SLAs with logistics partners to minimize label latency