Covariate Shift

A symmetric metric that quantifies distributional shift by binning a continuous variable and comparing the proportion of observations in each bin between a reference (training) and production dataset.

• Formula: PSI = Σ (Actual% - Expected%) * ln(Actual% / Expected%)
• Interpretation: PSI < 0.1 indicates no significant shift; 0.1–0.25 suggests moderate shift; > 0.25 signals a major distribution change requiring intervention.
• Common Application: Heavily used in financial risk modeling to monitor credit score drift, and in AdTech to track changes in user demographic features like age or income buckets.

A non-symmetric measure from information theory that quantifies how much a production probability distribution P(x) diverges from a reference training distribution Q(x).

• Asymmetry Note: D_KL(P || Q) ≠ D_KL(Q || P). The direction matters; typically, you measure the information lost when using the training distribution to approximate the production distribution.
• Continuous Features: Requires density estimation via histograms or kernel density estimation before calculation.
• Advantage: More sensitive to subtle shape changes in the distribution tail than PSI, making it useful for detecting shifts in high-value user segments.

Frequentist hypothesis tests that determine if two samples—one from training and one from a production window—are drawn from the same underlying distribution.

• Kolmogorov-Smirnov (KS) Test: Non-parametric test that measures the maximum distance between two empirical cumulative distribution functions. Sensitive to location and shape differences.
• Wasserstein Distance (Earth Mover's Distance): Measures the minimum 'work' required to transform one distribution into another. Unlike KS, it provides a scalar value that reflects the magnitude of the shift, not just a p-value.
• Practical Use: Run these tests on a scheduled basis (e.g., hourly) on key features like 'user_avg_session_duration' or 'item_price' to trigger automated alerts.

A black-box detection method that trains a binary classifier to distinguish between instances from the training dataset (label=0) and the production dataset (label=1).

• Mechanism: If a simple model like logistic regression can easily discriminate between the two datasets, significant covariate shift is present. The classifier's AUC serves as the drift score.
• Feature-Level Insight: By examining the learned feature weights of the domain classifier, you can identify exactly which specific features are driving the distributional divergence.
• Multivariate Detection: Unlike univariate tests, this method captures shifts in the joint distribution of features, detecting scenarios where individual features appear stable but their correlations have changed.

The ultimate operational indicator of covariate shift, detected by monitoring the divergence between offline evaluation metrics and live performance.

• Proxy Metrics: When ground-truth labels are delayed (e.g., conversions), monitor the average predicted CTR. A steady, unexplained drift in the mean prediction score often signals input distribution shift before labeled data confirms accuracy loss.
• Slice-Based Analysis: Track AUC or log loss across key feature slices (e.g., device_type='mobile', region='Southeast Asia'). A performance drop isolated to a specific slice pinpoints the sub-population experiencing shift.
• Guardrail Metrics: Set thresholds on business metrics like 'revenue-per-mille' to trigger a rollback to a static baseline model if covariate shift silently degrades the primary model.

Unlike covariate shift, concept drift occurs when the fundamental relationship between the input features (X) and the target variable (y) changes. The distribution of P(y|X) is no longer stable.

• Example: A user's threshold for considering a product a 'good deal' changes due to economic inflation, altering the purchase probability for the same price point.
• Key Distinction: Covariate shift is a change in P(X); concept drift is a change in P(y|X).

Label shift describes a change in the distribution of the target variable P(y) while the class-conditional feature distribution P(X|y) remains constant.

• Example: A sudden viral trend causes a specific product category to experience a massive spike in click-through rates, changing the overall base rate of positive labels.
• Key Distinction: This is the inverse of covariate shift, focusing on changes in the output distribution rather than the input distribution.

A classical statistical technique to correct covariate shift by assigning a weight to each training instance based on the ratio of its probability in the target domain to its probability in the source domain.

• Mechanism: Weights are calculated as w(x) = P_target(x) / P_source(x).
• Goal: Forces the model to pay more attention to training examples that are more representative of the serving environment.

A subfield of transfer learning that explicitly trains models to perform well on a target domain with a different data distribution than the source training domain.

• Adversarial Methods: Use a domain classifier to learn feature representations that are invariant to the shift between training and serving data.
• Application: Used when you know the serving distribution will be different but have unlabeled samples from it to guide training.

A broader MLOps term for any discrepancy between training and inference pipelines that degrades model performance. Covariate shift is a data-level cause of train-serving skew.

• Other Causes: Feature engineering code bugs, inconsistent data normalization, or stale lookup tables.
• Detection: Monitoring the statistical distance between training features and live inference features is the primary detection method.

A symmetric metric that quantifies the degree of distribution shift for a single feature between a reference dataset (training) and a monitoring dataset (production).

• Formula: PSI = sum( (Actual% - Expected%) * ln(Actual% / Expected%) ) across binned values.
• Thresholds: A PSI < 0.1 indicates no significant shift; PSI > 0.25 signals a major distribution change requiring model retraining.