Population Stability Index (PSI)

PSI calculates a symmetric score by comparing the proportion of observations in each bin of a reference distribution (typically training data) against a monitoring distribution (current production data). Unlike the Kullback-Leibler Divergence, PSI is symmetric, meaning PSI(A, B) = PSI(B, A). The formula sums the weighted logarithmic ratio of proportions across all bins:

• PSI < 0.1: No significant shift; the distribution is stable.
• 0.1 ≤ PSI < 0.25: Moderate shift; investigation is warranted.
• PSI ≥ 0.25: Significant shift; the feature distribution has materially changed, likely degrading model performance.

The accuracy of PSI depends heavily on the binning methodology used to discretize continuous variables. Common approaches include:

• Equal-width binning: Divides the range into intervals of identical size. Sensitive to outliers.
• Equal-frequency (quantile) binning: Each bin contains approximately the same number of observations from the reference set. Preferred for skewed financial data.
• Decision-tree binning: Uses a tree-based algorithm to find optimal split points that maximize information value.

A standard practice is to use 10-20 bins derived from the reference data's deciles or percentiles, then apply those fixed bin boundaries to the monitoring data.

In financial fraud detection, PSI is applied to individual feature distributions and the model score distribution to detect silent degradation:

• Feature-level PSI: Monitors input variables like transaction amount, time since last login, or merchant category code. A spike in PSI for transaction_amount may indicate a shift in customer spending behavior or a new fraud pattern.
• Score-level PSI: Tracks the distribution of the model's output probability scores. A high score-level PSI signals that the model is seeing a fundamentally different population, often triggering a champion-challenger evaluation.
• Characteristic Stability Index (CSI): A variant of PSI applied specifically to individual features, often used interchangeably in credit risk and fraud model governance.

While PSI is widely adopted, it has critical limitations that require complementary metrics:

• Insensitivity to bin-internal shifts: PSI only compares bin proportions. A distribution can shift dramatically within a bin without changing the bin's total count, yielding a low PSI.
• No directional information: PSI indicates that a shift occurred but not how the distribution moved. Pair PSI with Wasserstein Distance for directional insight.
• Small sample sensitivity: PSI is unreliable with low sample counts. A minimum of 50-100 observations per bin in the monitoring window is recommended.
• Seasonality confounding: Natural cyclical patterns (e.g., holiday spending spikes) can inflate PSI. Use a seasonally-adjusted reference window or compare against the same period from the prior year.

PSI is operationalized within continuous evaluation frameworks as an automated drift trigger:

• Scheduled batch calculation: PSI is computed daily or hourly on rolling windows of production traffic against a fixed training baseline stored in the model registry.
• Threshold-based alerting: When PSI exceeds 0.2 for a critical feature, an alert is routed to the MLOps team via Statistical Process Control (SPC) dashboards.
• Triggered retraining: A sustained PSI breach across multiple features can automatically initiate a triggered retraining pipeline, pulling fresh ground truth labels to rebuild the model.
• Slice-based PSI: Compute PSI on specific segments (e.g., mobile transactions, high-value wires) to detect localized drift masked by aggregate stability.

PSI is one component of a comprehensive drift detection suite. It is often used alongside:

• Kullback-Leibler Divergence: Asymmetric alternative that emphasizes regions where the monitoring distribution has density but the reference does not. Useful for detecting new modes.
• Kolmogorov-Smirnov Test (KS Test): A nonparametric hypothesis test that compares cumulative distributions. More sensitive to location shifts than PSI.
• Wasserstein Distance: Measures the minimum 'work' to reshape one distribution into another. Provides a geometrically interpretable drift magnitude.
• Maximum Mean Discrepancy (MMD): A kernel-based method that can detect subtle multivariate shifts without requiring binning, making it suitable for high-dimensional embedding spaces.