Drift Score

The Population Stability Index (PSI) is a widely adopted metric in finance and risk modeling for monitoring univariate feature drift. It measures the change in distribution of a categorical or binned continuous variable by comparing the percentage of observations in each bin between two datasets.

• Calculation: PSI = Σ ( (Actual % - Expected %) * ln(Actual % / Expected %) ).
• Interpretation: Common thresholds are PSI < 0.1 indicates no significant drift, 0.1 < PSI < 0.25 indicates moderate drift, and PSI > 0.25 indicates significant drift.
• Primary Use: Monitoring individual feature distributions over time, especially for model input features.

The Kolmogorov-Smirnov (KS) statistic is a non-parametric test for comparing continuous, one-dimensional probability distributions. The drift score is the maximum vertical distance between the two empirical cumulative distribution functions (ECDFs).

• Calculation: D = supₓ | F_reference(x) - F_production(x) |, where sup is the supremum.
• Interpretation: Values range from 0 (identical distributions) to 1 (completely different). It is sensitive to differences in both the location and shape of distributions.
• Primary Use: Detecting univariate drift in continuous features, often used for model predictions or scores.

Jensen-Shannon Divergence (JSD) is a symmetric, smoothed derivative of the Kullback-Leibler Divergence. It measures the similarity between two probability distributions and is bounded between 0 (identical) and 1 (maximally different) if using base-2 logarithm.

• Calculation: JSD(P || Q) = ½ * KL(P || M) + ½ * KL(Q || M), where M = ½(P + Q).
• Interpretation: Its symmetry and finite range make it more interpretable and stable than KL Divergence for drift detection.
• Primary Use: Quantifying drift for both discrete and continuous distributions; a common choice for multivariate drift detection when aggregating feature-level scores.

Wasserstein Distance, or Earth Mover's Distance, is a metric from optimal transport theory. It intuitively measures the minimum "work" required to transform one probability distribution into another, where work is defined as the amount of probability mass moved multiplied by the distance it is moved.

• Calculation: Computationally more intensive, often solved via linear programming or approximations.
• Interpretation: Particularly effective for continuous distributions and is sensitive to geometric shifts. It provides a more intuitive metric when distributions have non-overlapping support.
• Primary Use: Detecting drift in complex continuous distributions (e.g., embeddings, sensor data) and in generative model evaluation.

The Chi-Square test statistic is used to assess the independence between categorical distributions. For drift detection, it tests if the frequency distribution of a categorical feature in production is significantly different from the reference distribution.

• Calculation: χ² = Σ [ (O_i - E_i)² / E_i ], where O_i is the observed frequency in production and E_i is the expected frequency from the reference.
• Interpretation: A higher χ² value indicates greater divergence. The p-value derived from the statistic is used to determine statistical significance.
• Primary Use: Monitoring drift in high-cardinality categorical features, such as user country, product category, or error codes.

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test used to determine if two samples are drawn from different distributions. It measures the distance between the mean embeddings of the distributions in a high-dimensional reproducing kernel Hilbert space (RKHS).

• Calculation: MMD² = (1/m²) Σ k(x_i, x_j) + (1/n²) Σ k(y_i, y_j) - (2/mn) Σ k(x_i, y_j), where k is a kernel function like the Radial Basis Function (RBF).
• Interpretation: An MMD value close to zero suggests the distributions are similar. It is a powerful non-parametric multivariate test.
• Primary Use: Detecting multivariate drift in complex, high-dimensional data, such as image embeddings or feature vectors from deep learning models.

The Population Stability Index (PSI) is one of the most common statistical measures used to calculate a univariate drift score. It quantifies the shift between two distributions by comparing the proportion of observations in each bin (e.g., deciles).

• Calculation: PSI = Σ (Actual % - Expected %) * ln(Actual % / Expected %).
• Interpretation: Values < 0.1 indicate minimal change, 0.1-0.25 indicate some minor change, and > 0.25 signal a major population shift.
• Primary Use: It is extensively used in finance and credit risk modeling to monitor feature stability over time.

These are two fundamental types of data drift that a drift score may be trying to surface.

• Concept Drift: The statistical relationship between the input features (X) and the target variable (Y) changes. The mapping the model learned is no longer valid, even if the input distribution is stable.
• Covariate Shift: The distribution of the input features (X) changes between training and production, while the relationship P(Y|X) remains constant. The model's logic is still correct, but it's seeing unfamiliar inputs.

A high drift score on input features suggests covariate shift. A drop in model performance with a stable drift score may indicate concept drift.

A drift threshold is a predefined, configurable limit on a drift score that operationalizes drift detection. It transforms a continuous metric into a binary alert.

• Function: When a calculated drift score (e.g., PSI, JSD) exceeds this threshold, the system declares a significant drift event.
• Setting Thresholds: Thresholds are often set empirically based on historical data, domain tolerance for risk, and the cost of false alarms. For PSI, a common actionable threshold is 0.25.
• Automation: Exceeding a threshold can trigger automated alerts, dashboard warnings, or initiate an automated retraining trigger for a model.

This distinction defines the scope of what a drift score is measuring.

• Univariate Drift: Analysis is performed on individual features independently. Metrics like PSI or the Kolmogorov-Smirnov Test are applied to each column. It's computationally simple but can miss correlated shifts.
• Multivariate Drift: Analysis considers the joint distribution of multiple features simultaneously. Methods like Jensen-Shannon Divergence or model-based approaches (e.g., using a classifier) are used. It captures complex, interaction-based changes that univariate scores overlook.

A comprehensive monitoring system typically employs both, using univariate scores for root-cause analysis and a multivariate score for an overall system health signal.

These are the two core data entities required to compute any drift score.

• Reference Dataset: The baseline or benchmark distribution. This is typically the dataset used to train the model, or a trusted historical snapshot representing "normal" operation. It is considered the expected state.
• Production Dataset: The live, incoming data on which the deployed model performs inference. This can be analyzed as a batch (for offline detection) or as a streaming window (for online detection).

The drift score is fundamentally a function of the statistical distance between these two datasets. Training-serving skew is a specific failure mode where the production dataset diverges from the reference due to engineering inconsistencies, not world changes.

Model Performance Monitoring (MPM) is the complementary practice to data drift detection. While drift scores monitor the input (X), MPM directly monitors the model's output and accuracy.

• Direct Signal: MPM tracks metrics like accuracy, precision, recall, and F1-score using ground truth labels when available. A sustained drop is a direct indicator of model decay.
• Relationship to Drift: Concept drift will cause MPM metrics to degrade. Covariate shift may or may not degrade performance immediately, depending on if the model extrapolates well. Therefore, monitoring both drift scores and performance metrics provides a complete picture of model health.
• Challenge: MPM requires timely ground truth labels, which can be delayed or expensive, making proactive drift scores a critical early warning system.