Data Drift

A symmetric metric that quantifies how much a variable's distribution has shifted from a reference period to a monitoring period. PSI bins the data and compares the proportion of observations in each bin.

• PSI < 0.1: Insignificant drift
• 0.1 ≤ PSI < 0.25: Moderate drift, warrants investigation
• PSI ≥ 0.25: Significant drift, model likely degraded

Widely used in financial services for model validation, PSI is intuitive but sensitive to binning strategy and struggles with high-cardinality features.

A non-symmetric measure of how one probability distribution diverges from a reference distribution. In data drift detection, KL divergence quantifies the information lost when approximating the production distribution with the training distribution.

• Sensitive to small shifts in continuous features
• Unbounded score (0 to ∞), making threshold setting challenging
• Non-symmetric: D_KL(P || Q) ≠ D_KL(Q || P)

Often used alongside Jensen-Shannon divergence, its symmetric and bounded variant, for more interpretable monitoring dashboards.

Hypothesis tests that determine whether two samples are drawn from the same underlying distribution. Common choices for multivariate drift detection:

• Kolmogorov-Smirnov Test: Non-parametric, compares cumulative distribution functions. Sensitive to location and shape changes.
• Wasserstein Distance: Earth Mover's Distance; measures the minimum 'work' to transform one distribution into another. Robust for high-dimensional data.
• Maximum Mean Discrepancy (MMD): Kernel-based method that compares mean embeddings in a reproducing kernel Hilbert space. Effective for detecting subtle multivariate shifts.

P-values from these tests provide a statistical confidence level for drift alerts.

A machine learning-based method that trains a binary classifier to distinguish between samples from the reference (training) and production datasets. If a classifier can easily tell them apart, significant drift is present.

• AUC-ROC ≈ 0.5: No discernible drift (classifier performs at chance)
• AUC-ROC > 0.7: Meaningful distributional shift detected

This method is inherently multivariate, capturing complex interactions between features that univariate tests miss. The classifier's feature importance scores can also help localize which features are drifting.

An online drift detection algorithm designed for streaming data that dynamically adjusts a sliding window. ADWIN maintains a variable-length window of recent observations and shrinks it when a statistically significant change in the average is detected.

• No fixed threshold or window size required
• Provides theoretical guarantees on false positive and false negative rates
• Computationally efficient for high-velocity telemetry streams

Ideal for continuous monitoring in network telemetry pipelines where batch processing is impractical and immediate drift alerts are critical.

Leverages the internal representations of a neural network to detect semantic drift invisible in raw feature space. By comparing the distribution of embeddings from a reference and production set, subtle shifts in the latent structure of data are revealed.

• Use a pre-trained autoencoder or the model's own penultimate layer
• Apply MMD or k-NN distance metrics in the embedding space
• Detects covariate shift that raw feature statistics miss

Particularly effective for unstructured data like images, audio, and text where pixel-level or token-level comparisons are meaningless.

A change in the statistical relationship between input features and the target variable. Unlike data drift, the input distribution may appear stable, but the underlying mapping has shifted.

• Example: A credit risk model where the same income level now correlates with higher default probability due to an economic downturn.
• Detection: Requires access to ground truth labels, often delayed in production.
• Impact: Directly invalidates the model's learned decision boundary.

A specific type of data drift where the distribution of input features P(X) changes, but the conditional distribution P(y|X) remains constant.

• Example: A network traffic classifier trained on daytime patterns encounters nighttime traffic with different protocol mixes.
• Mitigation: Importance-weighting training samples or periodic retraining.
• Common in: Image recognition deployed in new lighting conditions or sensor telemetry from upgraded hardware.

The gradual erosion of predictive performance over time due to the cumulative effect of data drift, concept drift, or changes in the operational environment.

• Monitoring: Tracked via metrics like Population Stability Index (PSI) and Kullback-Leibler divergence.
• Remediation: Automated retraining pipelines triggered when drift metrics exceed predefined thresholds.
• Business impact: Undetected decay in a RAN anomaly detector can lead to increased network outages.

A granular form of data drift that tracks distribution changes in individual input features rather than the joint distribution.

• Example: A single telemetry counter, such as avg_rrc_connected_users, shifts its mean and variance after a network upgrade.
• Detection: Univariate statistical tests like Kolmogorov-Smirnov or Wasserstein distance per feature.
• Root cause: Often the first signal for diagnosing why a model is degrading, enabling targeted retraining or feature engineering.

A discrepancy between the data used during model training and the data encountered during inference, often caused by bugs in the feature pipeline rather than natural environmental change.

• Example: A preprocessing step normalizes a feature differently in the training notebook than in the production serving function.
• Detection: Log and compare summary statistics from the training set against live inference requests.
• Prevention: Feature stores and unified preprocessing libraries that guarantee consistency.

A symmetric metric quantifying how much a variable's distribution has shifted between a reference (training) and a monitoring (production) sample.

• Formula: Sum of (Actual% - Expected%) * ln(Actual% / Expected%) across bins.
• Interpretation: PSI < 0.1 indicates no significant shift; PSI > 0.25 signals a major distribution change requiring investigation.
• Application: Widely used in financial services model governance and increasingly in MLOps monitoring dashboards.