Population Stability Index (PSI)

A change in the statistical properties of input feature distributions between training and production data. While PSI quantifies the magnitude of this shift, data drift itself is the underlying phenomenon that silently degrades model performance. Key characteristics:

• Covariate shift: Input distribution P(X) changes, but P(Y|X) remains stable
• Detection methods: PSI, Kullback-Leibler divergence, Kolmogorov-Smirnov test, Jensen-Shannon distance
• Common causes: Seasonality, market regime changes, new product launches, or upstream data pipeline errors

Unlike concept drift, data drift does not necessarily indicate that the learned decision boundary is invalid—only that the model is now operating on unfamiliar input ranges.

A fundamental change in the underlying relationship between inputs and the target variable P(Y|X) over time. While PSI detects shifts in feature distributions, concept drift means the same feature values now map to different fraud probabilities.

• Real-world example: Fraudsters adapt their tactics, making previously suspicious transaction patterns now legitimate, or vice versa
• Detection requires ground truth: Unlike PSI which is unsupervised, concept drift detection needs labeled outcomes or proxy metrics like model performance degradation
• Types: Sudden (abrupt regime change), incremental (gradual evolution), recurring (seasonal patterns), and virtual (changes in the context not captured by features)

PSI serves as an early warning; concept drift confirms that the warning was material.

The feature-level decomposition of PSI, measuring distributional shift for each individual input variable. While PSI provides a single aggregate score, CSI identifies exactly which features are driving the instability.

• Calculation: Apply the same PSI formula independently to each feature's binned distribution
• Diagnostic workflow: High PSI triggers CSI analysis to isolate problematic features for root cause investigation
• Thresholds: Typically, CSI > 0.1 indicates a significant shift for that specific feature
• Use case: A fraud model's overall PSI may spike; CSI reveals that only the transaction_amount and merchant_category features shifted, while device_fingerprint remained stable

CSI transforms PSI from a red flag into an actionable diagnostic tool.

The process of comparing historical model predictions against actual realized outcomes to empirically measure predictive accuracy. While PSI is a forward-looking early warning, backtesting provides retrospective ground-truth validation.

• Methodology: Apply the model to a historical period where outcomes are now known, comparing predicted risk scores to actual fraud labels
• Key metrics: Area under the ROC curve (AUC), precision-recall curves, and Kolmogorov-Smirnov statistics over time windows
• Regulatory requirement: SR 11-7 mandates ongoing backtesting as part of model monitoring
• Limitation: Backtesting confirms past degradation but cannot prevent future losses; PSI complements it by providing real-time distributional monitoring

Together, PSI and backtesting form a complete monitoring strategy: PSI warns, backtesting confirms.

The independent, evidence-based evaluation of a model's conceptual soundness, performance, and limitations. PSI monitoring is an operational control; model validation is the governance function that assesses whether that control framework is adequate.

• Three pillars of validation: Conceptual soundness assessment, outcome analysis (including backtesting), and ongoing monitoring evaluation
• PSI's role in validation: Validators review PSI thresholds, monitoring frequency, and escalation procedures as part of the ongoing monitoring assessment
• Independence requirement: SR 11-7 mandates that validation be performed by qualified parties independent of model development and ownership
• Frequency: Initial validation before deployment, annual revalidation, and event-driven revalidation when PSI or other metrics breach thresholds

PSI is a tool; model validation ensures the tool is used correctly within a sound governance framework.

A controlled experimentation methodology where a live champion model runs in parallel with one or more challenger models on identical production traffic. When PSI indicates the champion model is operating on shifted data, the challenger framework provides a structured path to replacement.

• Implementation: Route a percentage of live traffic to challenger models, logging predictions without impacting decisions (shadow mode) or with controlled impact (A/B testing)
• PSI integration: A sustained PSI breach triggers the evaluation of challenger models that were trained on more recent data distributions
• Statistical validation: Challenger models must demonstrate statistically significant improvement over the champion before promotion
• Risk mitigation: Ensures a validated replacement is ready before retiring a degraded model, preventing gaps in fraud coverage

This framework operationalizes PSI alerts into a governed model replacement process.