Kolmogorov-Smirnov Test (KS Test)

The KS test makes no assumptions about the underlying data distribution. It does not require the data to follow a Gaussian curve, making it robust for financial transaction features that often exhibit heavy-tailed or multimodal distributions.

• Distribution-Free: Works directly on the empirical cumulative distribution function (ECDF).
• Versatile: Applicable to continuous variables like transaction amounts or risk scores.
• Limitation: Sensitivity is highest near the center of the distribution and lower at the tails.

The test statistic, D, is the maximum absolute vertical distance between the two empirical cumulative distribution functions (ECDFs).

• Calculation: D = max|F1(x) - F2(x)|.
• Interpretation: A larger D indicates a greater divergence between the reference and production data.
• Threshold: If D exceeds a critical value (derived from the sample size and significance level α), the null hypothesis that the samples are drawn from the same distribution is rejected.

In a continuous evaluation pipeline, the KS test is applied as a univariate drift detector on individual features.

• Reference Window: A baseline sample from the training data or a stable production period.
• Analysis Window: The most recent batch of production traffic.
• Alerting: A trigger is fired if the p-value falls below a strict threshold (e.g., p < 0.001), indicating a statistically significant shift that may degrade model performance.

The KS test is highly sensitive to large sample sizes. In big data streaming contexts, even trivially small distributional shifts can produce a statistically significant p-value.

• Practical Drift: A statistically significant result does not always imply practical model degradation.
• Mitigation: Combine the KS statistic with a business impact metric like the Population Stability Index (PSI) to filter out noisy alerts that have no material effect on fraud detection accuracy.

Unlike Kullback-Leibler Divergence, the KS test is symmetric and does not require binning continuous data.

• Symmetry: KS(P, Q) = KS(Q, P), unlike KL divergence.
• No Binning: Operates directly on raw values, avoiding information loss from discretization.
• Trade-off: KL divergence can be more sensitive to differences in the tails of a distribution, whereas the KS test focuses on the median body of the data.

The KS test has two primary variants for drift detection:

• Two-Sample KS Test: Compares two empirical distributions (e.g., training data vs. production data). This is the standard for data drift monitoring.
• One-Sample KS Test: Compares a sample against a specific theoretical distribution (e.g., testing if errors are normally distributed). This is less common in raw feature monitoring but useful for residual analysis.

A symmetric metric that quantifies the shift in a variable's distribution over time by measuring the divergence between expected and actual probability bins. PSI is the industry standard for monitoring feature drift in financial services.

• Calculation: PSI = Σ (Actual% - Expected%) * ln(Actual% / Expected%)
• Interpretation: PSI < 0.1 indicates no significant shift; PSI > 0.25 signals major drift requiring investigation
• Relationship to KS: While KS measures maximum vertical distance between cumulative distributions, PSI measures information divergence across binned ranges

Two distinct failure modes that the KS test helps diagnose. Data drift (covariate shift) occurs when input feature distributions change—detectable by applying the KS test to each feature independently. Concept drift occurs when the relationship between features and the target variable changes, requiring monitoring of prediction error rates.

• Example: A KS test on transaction amounts may reveal data drift if spending patterns shift seasonally
• Example: Concept drift occurs when previously fraudulent patterns become legitimate without feature distribution changes
• Diagnostic workflow: Use KS for feature-level monitoring; use Page-Hinkley or ADWIN for prediction error monitoring

An asymmetric statistical measure of how one probability distribution diverges from a second reference distribution. KL divergence is often preferred over KS when the direction of drift matters.

• Asymmetry: D_KL(P || Q) ≠ D_KL(Q || P), making it useful when the reference distribution has a specific meaning
• Limitation: Undefined when Q(x) = 0 but P(x) > 0, requiring smoothing in sparse regions
• Complement to KS: KS provides a bounded [0,1] interpretable statistic; KL provides an unbounded information-theoretic measure sensitive to tail behavior

A metric measuring the minimum 'cost' to transform one distribution into another. Wasserstein distance captures geometric relationships between distributions that the KS test may miss.

• Advantage over KS: Sensitive to both location and shape changes, not just maximum vertical separation
• Interpretation: Literally the amount of probability mass multiplied by the distance it must be moved
• Use case: Particularly effective for monitoring continuous features like transaction amounts where the 'effort' to shift mass is meaningful

A kernel-based statistical test that distinguishes two probability distributions by comparing their embeddings in a reproducing kernel Hilbert space. MMD generalizes the KS test to high-dimensional and structured data.

• Kernel flexibility: Using a Gaussian kernel allows MMD to detect any difference between distributions with sufficient data
• Multivariate capability: Unlike KS which is primarily univariate, MMD can test for joint distribution shifts across multiple features simultaneously
• Statistical power: Often more sensitive than KS for detecting subtle, complex distributional changes in high-dimensional feature spaces

An automated MLOps process that persistently monitors deployed model performance against validation baselines. The KS test serves as a core statistical trigger within this framework.

• Pipeline components: Feature validation → KS-based drift detection → Performance metric calculation → Alert generation
• Threshold calibration: KS statistic thresholds must be tuned per feature to balance sensitivity against alert fatigue
• Automated response: When KS exceeds threshold, trigger champion-challenger evaluation or triggered retraining workflows to maintain fraud detection efficacy