Kullback-Leibler Divergence (KL Divergence)

KL Divergence is fundamentally asymmetric: D_KL(P || Q) ≠ D_KL(Q || P). This means the divergence from distribution P to Q is not the same as from Q to P.

• Forward KL: D_KL(P || Q) penalizes Q for placing low probability where P has high probability (mode-covering).
• Reverse KL: D_KL(Q || P) penalizes Q for placing high probability where P has low probability (mode-seeking).

In drift detection, this asymmetry is critical. Using the training distribution as P and production data as Q measures a different effect than the reverse, influencing whether you detect new modes or missing modes.

KL Divergence is always non-negative: D_KL(P || Q) ≥ 0. This is guaranteed by Gibbs' inequality.

• D_KL(P || Q) = 0 if and only if P and Q are identical almost everywhere.
• A value of zero indicates perfect distributional match.
• Any deviation results in a positive value, making it a natural measure of drift magnitude.

This property allows it to serve as a lower-bound metric in optimization, such as in variational inference where minimizing KL divergence approximates a target distribution.

KL Divergence is not a true distance metric because it violates both symmetry and the triangle inequality.

• The triangle inequality states that for a distance metric d, d(A, C) ≤ d(A, B) + d(B, C).
• KL Divergence does not satisfy this, meaning you cannot bound the divergence between two distributions by summing intermediate divergences.

This is why metrics like the Jensen-Shannon Divergence were developed—they symmetrize and bound KL Divergence to create a proper distance metric suitable for applications requiring metric properties.

KL Divergence is defined only when P is absolutely continuous with respect to Q: P(x) > 0 implies Q(x) > 0 for all x.

• If Q assigns zero probability to an event where P assigns non-zero probability, D_KL(P || Q) → ∞.
• This is a practical hazard in drift monitoring: a single new categorical value in production data can cause the divergence to explode.

Mitigation strategies include:

• Laplace smoothing or additive smoothing of probability estimates.
• Using alternative metrics like Population Stability Index (PSI) which handles zero bins more gracefully.
• Clipping probability values to a small epsilon.

KL Divergence decomposes into cross-entropy minus entropy: D_KL(P || Q) = H(P, Q) - H(P).

• H(P) is the entropy of the true distribution P, representing its inherent uncertainty.
• H(P, Q) is the cross-entropy, measuring the average code length needed to encode events from P using a code optimized for Q.
• KL Divergence thus quantifies the extra bits required due to using the wrong distribution.

In machine learning, minimizing cross-entropy loss is equivalent to minimizing KL divergence since H(P) is constant with respect to model parameters.

KL Divergence is invariant under reparameterization. If you apply an invertible transformation to the random variable, the divergence between distributions remains unchanged.

• This property makes it a geometrically meaningful measure of distributional difference.
• It does not depend on the coordinate system used to represent the data.
• This contrasts with metrics like mean squared error, which change under non-linear transformations.

For drift detection, this means KL Divergence captures fundamental distributional shifts regardless of feature scaling or encoding choices, provided the transformation is bijective.

A symmetric metric derived directly from KL Divergence, PSI quantifies the shift in a variable's distribution by summing the logarithmic difference between expected and actual proportions across binned data.

• Formula: PSI = Σ (Actual% - Expected%) * ln(Actual% / Expected%)
• Interpretation: PSI < 0.1 indicates no significant shift; PSI > 0.25 signals major drift requiring investigation
• Use Case: The financial industry standard for monitoring feature drift in credit scoring and fraud models
• Key Difference: Unlike raw KL Divergence, PSI is symmetric and operates on discretized bins, making it robust to small sample sizes

Also known as Earth Mover's Distance, this metric measures the minimum 'cost' to transform one probability distribution into another by moving probability mass. Unlike KL Divergence, it is a true metric satisfying symmetry and the triangle inequality.

• Advantage: Remains stable when distributions have non-overlapping support, where KL Divergence becomes undefined or infinite
• Interpretation: Directly corresponds to physical distance between distribution shapes
• Use Case: Preferred for continuous feature monitoring in fraud models where KL Divergence's asymmetry complicates threshold setting

A nonparametric hypothesis test that determines whether two samples are drawn from the same underlying distribution by measuring the maximum vertical distance between their empirical cumulative distribution functions.

• Test Statistic: D = max |F₁(x) - F₂(x)|, the supremum of absolute difference between ECDFs
• Output: A p-value indicating whether the null hypothesis (same distribution) can be rejected
• Use Case: Applied to individual feature monitoring in production fraud pipelines; a significant KS statistic triggers drift alerts
• Limitation: More sensitive to shifts near the center of distributions than at the tails, which can miss fraud-relevant edge cases

A kernel-based statistical test that distinguishes probability distributions by comparing the means of their embeddings in a reproducing kernel Hilbert space (RKHS). MMD equals zero if and only if the two distributions are identical.

• Kernel Choice: Gaussian RBF kernel is standard; the bandwidth parameter controls sensitivity to different scales of variation
• Advantage: Detects subtle multivariate distribution shifts that univariate tests like KS miss entirely
• Use Case: Applied to high-dimensional feature vectors in fraud detection to detect coordinated drift across multiple transaction attributes simultaneously
• Relationship to KL: MMD avoids KL Divergence's density estimation requirement, making it more practical for high-dimensional data

A symmetrized and smoothed version of KL Divergence that addresses its asymmetry and undefined regions. JSD calculates the average KL Divergence of each distribution from their mixture distribution.

• Formula: JSD(P||Q) = ½ KL(P||M) + ½ KL(Q||M), where M = ½(P + Q)
• Range: Bounded between 0 and ln(2) when using natural log, making thresholds easier to interpret
• Use Case: Preferred over raw KL Divergence when a symmetric, bounded metric is required for automated drift alerting in production MLOps pipelines
• Square Root: The square root of JSD yields the Jensen-Shannon distance, a true metric satisfying the triangle inequality

A practical technique that trains a binary classifier to distinguish between training and production data samples. If the classifier achieves high accuracy, significant distributional shift exists between the two sets.

• Mechanism: Combine training and production data, label them (0 for training, 1 for production), shuffle, and train a classifier; ROC-AUC above 0.7 indicates problematic drift
• Advantage: Captures complex multivariate shifts without requiring explicit distribution modeling or density estimation
• Use Case: Deployed as a pre-inference gate in fraud pipelines; if adversarial validation detects drift, the system can trigger shadow evaluation or fallback to a rules-based engine
• Relationship to KL: Provides a practical, model-agnostic alternative when KL Divergence is computationally intractable for high-dimensional feature spaces