Data Drift

Quantifying drift requires statistical distance measures that compare training and production distributions. Population Stability Index (PSI) bins continuous variables and measures the logarithmic difference between expected and actual percentages in each bin, with values above 0.25 indicating significant shift. Kullback-Leibler Divergence provides an asymmetric measure of information loss when approximating one distribution with another, while Wasserstein Distance calculates the minimum 'earth mover's cost' to transform one distribution into another, offering a more geometrically intuitive metric for high-dimensional feature spaces.

Not all data drift is identical. Covariate shift occurs when P(X) changes but P(Y|X) remains constant—the input features drift but the relationship to fraud labels stays intact. For example, transaction amounts may shift seasonally without changing what constitutes fraud at each amount level. Prior probability shift affects P(Y) directly, such as a sudden surge in the base fraud rate from 0.1% to 2% during a coordinated attack. Distinguishing between these drift types is critical because covariate shift may require feature adaptation, while prior shift demands threshold recalibration.

Formal statistical tests provide rigorous drift detection with confidence bounds. The Kolmogorov-Smirnov (KS) Test compares empirical cumulative distribution functions of training and production samples, outputting a p-value that indicates whether the two samples likely originate from the same distribution. Maximum Mean Discrepancy (MMD) embeds distributions into a reproducing kernel Hilbert space and tests for differences in mean embeddings, offering sensitivity to higher-order moments beyond simple mean shifts. These tests are typically applied per-feature in batch monitoring pipelines.

A particularly insidious form of data drift arises from training-serving skew—discrepancies between the feature engineering code executed during training and the code running in production inference. Common causes include:

• Feature transformation mismatches: Different normalization constants or encoding logic
• Timestamp handling errors: Incorrect timezone conversions or date boundary logic
• Missing value defaults: Inconsistent imputation strategies between pipelines Unlike natural distributional evolution, this skew is a deterministic engineering defect that produces systematic, repeatable drift patterns.

Adversarial validation provides a practical, model-based approach to detecting distributional shift. The technique trains a binary classifier to distinguish between training samples and production samples using the feature values alone. If the classifier achieves high accuracy—significantly above random chance—it confirms that the two datasets are drawn from different distributions. The features with the highest importance scores in this classifier pinpoint exactly which variables are drifting, enabling targeted investigation rather than broad statistical monitoring across all features.

Aggregate drift metrics can mask critical shifts occurring in specific data segments. Slice-based evaluation partitions production traffic by meaningful dimensions—merchant category, transaction channel, geographic region, or user segment—and computes drift metrics independently for each slice. A model may appear stable globally while experiencing severe drift in a high-risk segment like cross-border wire transfers. This granular approach prevents silent failures where aggregate monitoring dashboards show green status while specific cohorts experience degraded fraud detection.