Model Drift

Occurs when the fundamental statistical relationship between the input features and the target variable changes. The model's learned mapping from inputs to outputs is no longer valid.

Example: A new raw material supplier introduces a visually distinct but acceptable surface texture. The model, trained on the old texture, incorrectly flags this as a scratch defect, increasing the False Reject Rate (FRR).

• P(Y|X) changes: The conditional probability of the label given the features shifts.
• Sudden Drift: An abrupt change, like a new product SKU.
• Incremental Drift: A gradual change, like slow tool wear altering part geometry.

A change in the statistical distribution of the input features themselves, even if the decision boundary remains valid. The model sees data unlike what it was trained on.

Example: Gradual degradation of factory floor lighting shifts the pixel intensity histogram of captured images. The model, trained on bright images, becomes uncertain on darker inputs.

• P(X) changes: The probability distribution of the input features shifts.
• Sensor Decay: Physical camera degradation, lens dust accumulation, or vibration misalignment.
• Environmental Shift: Changes in ambient temperature, humidity, or background clutter.

A change in the distribution of the target variable itself, independent of the input features. The prior probability of seeing a certain class changes in production.

Example: A successful process improvement reduces the actual defect rate from 5% to 0.1%. The model, calibrated for a higher base rate, now over-predicts defects relative to the new reality.

• P(Y) changes: The marginal probability of the target label shifts.
• Class Imbalance Shift: The ratio of defective to non-defective parts changes dramatically.
• Seasonality: Cyclical changes in production volume or product mix that alter defect frequency.

A specific form of drift where the relevance of data decays purely as a function of time, often due to cyclical patterns or obsolescence of the training data's context.

Example: A model trained on summer production data fails in winter because material viscosity changes with temperature, altering the visual appearance of acceptable parts.

• Non-Stationarity: The data-generating process is not constant over time.
• Recurring Contexts: Drift that follows a predictable pattern (hourly, daily, seasonal).
• Model Staleness: The training data simply becomes too old to represent the current process.

A primary metric for quantifying data drift by comparing the distribution of a feature in the production data against its distribution in the training baseline. It measures the divergence between two distributions.

Interpretation Guide:

• PSI < 0.1: Insignificant drift. No action required.
• 0.1 ≤ PSI < 0.25: Moderate drift. Requires monitoring and investigation.
• PSI ≥ 0.25: Significant drift. Model retraining or feature engineering is likely necessary.

PSI is calculated by binning the feature values and summing the logarithmic difference between the actual and expected percentages in each bin.

The most robust defense against drift is a continuous feedback loop that samples production inferences, sends them for human or high-precision verification, and compares the model's prediction against the newly established ground truth.

Process:

• Shadow Mode: Run the new model in parallel, logging predictions without actioning them.
• Champion/Challenger: A/B test a retrained model against the current production model.
• Active Learning: A human operator labels only the most uncertain or high-value samples flagged by the model to efficiently create a targeted retraining dataset.

A change in the statistical distribution of the input features (P(X)) between training and production. Unlike concept drift, the relationship between inputs and outputs remains stable, but the incoming data looks different.

• Example: A vision inspection system trained on parts under fluorescent light begins receiving images under new LED lighting, shifting pixel intensity distributions.
• Detection: Use Kolmogorov-Smirnov tests or Population Stability Index (PSI) to compare feature distributions over time.
• Impact: Causes silent degradation as the model extrapolates into unfamiliar regions of feature space.

A change in the underlying relationship between inputs and target outputs (P(Y|X)). The same input now maps to a different correct prediction, rendering the model's learned decision boundary obsolete.

• Example: A new defect type appears on the production line that the model has never seen, or a previously critical scratch defect is reclassified as cosmetic by engineering.
• Detection: Requires ground truth labels for production data. Monitor accuracy, precision, and recall against a delayed labeling pipeline.
• Challenge: Harder to detect than data drift because it requires ongoing human annotation of production samples.

A specific form of data drift where the distribution of input variables changes, but the conditional distribution of outputs given inputs remains constant. This is the most common drift type in manufacturing vision systems.

• Example: A camera's auto-exposure algorithm is updated, systematically darkening all captured images. The physical relationship between a visible crack and the label 'defective' hasn't changed, but the input pixel values have.
• Mitigation: Histogram equalization and data augmentation during training can build robustness to covariate shift.
• Formal definition: P_s(X) ≠ P_t(X) but P_s(Y|X) = P_t(Y|X), where s is source (training) and t is target (production).

A change in the prior distribution of target classes (P(Y)) in production. The proportion of defective vs. non-defective parts shifts, which can skew metrics and trigger false alarms in monitoring dashboards.

• Example: A supplier quality issue causes a sudden spike in the defect rate from 0.5% to 5%. The model's precision and recall remain unchanged, but the raw count of false positives increases proportionally.
• Monitoring: Track prediction distribution over time. A sudden change in the ratio of predicted classes often signals label drift.
• Note: Label drift alone does not indicate model degradation, but it often co-occurs with concept drift.

Operational approaches to restoring model performance after drift is detected, balancing freshness against stability and annotation cost.

• Periodic Retraining: Retrain on a fixed cadence (e.g., weekly) using the most recent labeled data. Simple but may lag behind sudden drift.
• Trigger-Based Retraining: Automatically initiate retraining when a drift metric exceeds a threshold. Requires robust monitoring and a human-in-the-loop validation gate.
• Online Learning: Update model weights incrementally with each new labeled example. Enables rapid adaptation but risks catastrophic forgetting of rare defect types.
• Champion-Challenger: Deploy a new candidate model in shadow mode alongside the production model. Promote the challenger only after it demonstrates statistically significant improvement on live data.