Page-Hinkley Test

A change in the underlying relationship between input features and the target variable, rendering the model's learned decision boundary obsolete. Unlike covariate shift, concept drift means the fundamental mapping from transaction attributes to fraud probability has changed.

• Real-world trigger: Fraudsters adopt a new technique that the model's features fail to capture
• Detection challenge: Requires ground truth labels to confirm, unlike pure data drift
• Impact: Directly degrades recall and precision, leading to undetected fraud or increased false positives

An online drift detection algorithm that dynamically adjusts the size of a sliding window over a data stream to maintain a stable statistical reference. When the mean of a signal within two sub-windows diverges significantly, ADWIN shrinks the window to discard old data and triggers a drift alert.

• Key advantage: No fixed window size or threshold requires manual tuning
• Comparison to Page-Hinkley: ADWIN handles variable drift rates automatically; Page-Hinkley uses a fixed cumulative sum with user-defined sensitivity
• Use case: Monitoring prediction error rates on streaming transaction data

A quality control methodology using control charts and statistical limits to distinguish normal process variation from significant shifts. Applied to model monitoring, SPC tracks metrics like daily fraud detection rate against upper and lower control limits.

• Core concept: A process is 'out of control' when a metric exceeds 3-sigma limits or violates run rules
• Relationship to Page-Hinkley: SPC provides the broader framework; Page-Hinkley is a specific sequential test within that framework
• Practical application: Plotting model PSI scores or false positive rates on Shewhart control charts

An automated MLOps pipeline that initiates a new model training cycle in response to a specific event, such as a Page-Hinkley alert or a drop in a key performance indicator below a defined threshold.

• Pipeline components: Drift detection → alert generation → data labeling job → training job → model validation → shadow deployment
• Critical consideration: Must include a human-in-the-loop gate for fraud models due to regulatory requirements
• Anti-pattern: Retraining on data that already contains the drift without investigating root cause first

An asymmetric statistical measure of how one probability distribution diverges from a reference distribution. In drift monitoring, KL divergence quantifies the magnitude of shift between training and production feature distributions.

• Formula interpretation: D_KL(P||Q) measures the information lost when Q is used to approximate P
• Comparison to Page-Hinkley: KL divergence is a batch metric for distribution comparison; Page-Hinkley is a sequential test for mean change detection
• Threshold guidance: Values above 0.1 typically warrant investigation in fraud model monitoring

A symmetric metric that quantifies the shift in a variable's distribution over time by measuring divergence between expected and actual probability bins. PSI is the industry standard for feature drift monitoring in financial models.

• Interpretation scale: PSI < 0.1 (no significant shift), 0.1–0.25 (moderate shift), > 0.25 (significant shift requiring action)
• Complement to Page-Hinkley: PSI monitors input feature stability; Page-Hinkley monitors output error rate stability
• Implementation: Calculate PSI for each feature independently, then aggregate for a model health dashboard