Kullback-Leibler Divergence (KL Divergence)

KL Divergence is not symmetric: (D_{KL}(P || Q) \neq D_{KL}(Q || P)). This is its most defining property.

• Interpretation: (D_{KL}(P || Q)) measures the information lost when distribution Q is used to approximate the true distribution P. Reversing the arguments asks a different question.
• Implication for Drift: This makes directionality critical. In drift detection, (P) is typically the baseline/reference distribution (e.g., training data), and (Q) is the current/target distribution (e.g., production data). The divergence quantifies the cost of assuming the current data comes from the old distribution.

KL Divergence is always greater than or equal to zero: (D_{KL}(P || Q) \geq 0).

• Equality Condition: (D_{KL}(P || Q) = 0) if and only if the two distributions (P) and (Q) are identical (almost everywhere).
• Practical Use: This property provides a clear, interpretable baseline. Any positive value indicates a measurable divergence. In production monitoring, a sustained value > 0 signals a distributional shift requiring investigation.

KL Divergence has a foundational interpretation in information theory. It measures the expected extra number of bits required to encode samples from distribution (P) using a code optimized for distribution (Q).

• From Cross-Entropy: (D_{KL}(P || Q) = H(P, Q) - H(P)), where (H(P)) is the entropy of (P) (inherent randomness) and (H(P, Q)) is the cross-entropy.
• In Model Evaluation: When (P) is the true data distribution and (Q) is the model's distribution, minimizing KL divergence is equivalent to maximizing the model's log-likelihood of the data.

KL Divergence is highly sensitive to differences where (P(x)) is non-zero but (Q(x)) is very small or zero.

• The Log Penalty: The formula ( \sum P(x) \log(\frac{P(x)}{Q(x)}) ) includes a (\log(\frac{1}{Q(x)})) term. If (Q(x) = 0) for an event where (P(x) > 0), the divergence becomes infinite.
• Engineering Consideration: This makes it a conservative metric for drift detection. It will flag scenarios where the current distribution fails to account for events that were possible in the baseline. This often requires smoothing (e.g., adding epsilon) to handle finite samples.

KL Divergence is one member of the f-divergence family. Its properties differ from other common metrics:

• vs. Jensen-Shannon Divergence: JS Divergence is a symmetrized and smoothed version of KL, bounded between 0 and 1, and avoids infinite values.
• vs. Total Variation Distance: TV Distance measures the largest possible difference in probability assigned to any event, providing a more robust but less information-theoretic view.
• vs. Wasserstein Distance: Wasserstein (Earth Mover's Distance) is a true metric (symmetric, obeys triangle inequality) and is less sensitive to absolute support differences, making it useful for high-dimensional or continuous drift detection.

In MLOps, KL Divergence is applied as a univariate drift detector for categorical or discretized continuous features.

• Typical Workflow: 1. Discretize a continuous feature into bins using the baseline data. 2. Compute the frequency distribution for the baseline (P) and a recent window (Q). 3. Calculate (D_{KL}(P || Q)). 4. Trigger an alert if the value exceeds a threshold.
• Advantage: Provides an information-theoretic measure of shift magnitude.
• Limitation: As a univariate measure, it cannot capture multivariate or correlation drift. It is often used in conjunction with metrics like PSI or Wasserstein Distance for a comprehensive view.

A symmetric and smoothed alternative to KL Divergence, defined as the average of the KL Divergence from each distribution to their midpoint. It is calculated as JSD(P || Q) = 1/2 * KL(P || M) + 1/2 * KL(Q || M), where M = (P + Q)/2. Its key properties are:

• Symmetry: JSD(P || Q) = JSD(Q || P).
• Bounded Range: Values are always between 0 and 1 (or log(2) if using base-2 log), making it interpretable as a distance metric.
• Square Root is a Metric: The square root of JSD satisfies the triangle inequality, which KL Divergence does not. It is often preferred for drift detection when a true distance metric is required.

A metric that measures the minimum "cost" of transforming one probability distribution into another, conceptualized as moving piles of earth. Unlike KL Divergence, it is defined even when distributions have non-overlapping support. Its properties include:

• Metric Properties: It is symmetric and satisfies the triangle inequality.
• Sensitivity to Geometry: Accounts for the distance between points in the sample space, making it robust for multivariate drift detection.
• Computational Cost: Calculating the exact Wasserstein distance is more computationally intensive than KL Divergence, often requiring linear programming or approximations like the Sinkhorn algorithm. It is particularly useful for comparing high-dimensional or continuous distributions where KL Divergence may be infinite or unstable.

A widely used metric in finance and risk modeling to quantify the shift between two distributions, often for monitoring feature or score distributions. It is closely related to KL Divergence and is calculated by binning data and summing: PSI = Σ (Actual% - Expected%) * ln(Actual% / Expected%). Key characteristics:

• Practical Interpretation: Values < 0.1 indicate insignificant change, 0.1-0.25 indicate moderate change, and > 0.25 indicate major shift.
• Categorical & Continuous: Applied to binned continuous variables or categorical features.
• Operational Use: A cornerstone of batch drift detection in production ML systems, often computed weekly or monthly to monitor model input stability.

A simple, interpretable distance metric between probability distributions, defined as half the sum of absolute differences: TV(P, Q) = 1/2 * Σ |P(x) - Q(x)|. It represents the largest possible difference in probability that the two distributions can assign to the same event. Important aspects are:

• Bounded: Ranges from 0 (identical) to 1 (completely disjoint).
• Relationship to KL: Pinsker's Inequality states that TV(P, Q) ≤ √( KL(P || Q) / 2 ), providing a theoretical link—a large KL Divergence implies a large Total Variation distance.
• Computational Simplicity: Easy to compute for discrete distributions, but can be challenging for continuous ones without discretization. It provides a robust, worst-case measure of distributional difference.

A broad family of divergence measures, of which KL Divergence is a specific instance. An f-divergence is defined as D_f(P || Q) = Σ Q(x) * f( P(x) / Q(x) ), where f is a convex function with f(1) = 0. Common examples include:

• KL Divergence: f(t) = t log(t).
• Reverse KL Divergence: f(t) = -log(t).
• Total Variation Distance: f(t) = 0.5 * |t - 1|.
• Pearson χ² Divergence: f(t) = (t - 1)².
• Hellinger Distance: f(t) = (√t - 1)². This framework unifies many drift detection metrics, allowing theoreticians and engineers to select the divergence whose properties (e.g., sensitivity to tails, symmetry) best suit the monitoring task.

A kernel-based statistical test used to determine if two samples are drawn from different distributions. It measures the distance between the mean embeddings of the distributions in a Reproducing Kernel Hilbert Space (RKHS). Key advantages for drift detection:

• No Density Estimation: Works directly with samples, avoiding the need to estimate probability densities, which is difficult in high dimensions.
• Multivariate Capability: Effectively handles high-dimensional data like embeddings or image features.
• Test Statistic: Provides a well-defined statistical test; a large MMD value rejects the null hypothesis that the samples are from the same distribution. It is a powerful non-parametric alternative to KL Divergence, especially for complex, modern data types where traditional divergence measures struggle.