Maximum Mean Discrepancy (MMD)

MMD is a non-parametric statistical test, meaning it makes no assumptions about the specific form (e.g., Gaussian) of the underlying data distributions. This is crucial for real-world data where distributions are often unknown and complex.

• Flexibility: Works on any data type where a kernel can be defined (images, text, graphs).
• No Density Estimation: Avoids the difficult problem of explicitly estimating probability densities, especially in high dimensions.
• Direct Sample Comparison: The test statistic is computed directly from the samples, making it practical for finite datasets.

MMD leverages the kernel trick to map data into a high-dimensional Reproducing Kernel Hilbert Space (RKHS). In this space, the distance between the mean embeddings of two distributions is computed.

• Mean Embedding: Each distribution is represented by a single point (its mean) in the RKHS.
• Distance Metric: MMD is the distance between these mean embeddings. If the distributions are identical, their mean embeddings coincide, and MMD = 0.
• Rich Representations: Using powerful kernels like the Gaussian RBF kernel allows MMD to capture complex, non-linear differences between distributions.

MMD operates as a rigorous two-sample hypothesis test.

• Null Hypothesis (H₀): The two samples (e.g., training data vs. production data) are drawn from the same distribution.
• Test Statistic: The calculated MMD value.
• Statistical Significance: A p-value is estimated, often via a permutation test, to determine if the observed MMD is statistically significant. A low p-value (e.g., < 0.05) provides evidence to reject H₀ and signal drift.

A major advantage of MMD for drift detection is that it is fundamentally unsupervised. It requires only the input features (X), not the labels or model predictions (Y).

• Early Warning: Can detect data drift (covariate shift) before it impacts model accuracy.
• Label-Efficient: Vital for production where true labels are scarce, delayed, or expensive to obtain.
• Versatility: Applicable to any model type (classifier, regressor) and any machine learning pipeline stage.

MMD can be computed efficiently using the quadratic-time unbiased estimator or the linear-time estimator, making it scalable.

• Unbiased Estimator: Common formula: MMD² = (1/m²)Σk(x_i, x_j) + (1/n²)Σk(y_i, y_j) - (2/mn)Σk(x_i, y_j) for samples X (size m) and Y (size n).
• Linear-Time MMD: Approximates the statistic in O(m+n) time, enabling application to very large datasets or high-frequency data streams.
• GPU Acceleration: Kernel matrix computations are highly parallelizable on modern hardware.

MMD is part of a family of integral probability metrics. Key comparisons:

• vs. KL Divergence: MMD is symmetric and always finite; KL Divergence can be infinite and requires density estimation.
• vs. Wasserstein Distance: Both are metrics. Wasserstein is based on optimal transport cost; MMD is based on RKHS distance. MMD is often easier to compute and differentiate.
• vs. Kolmogorov-Smirnov: KS is for one-dimensional distributions; MMD generalizes to multi-dimensional and structured data via kernels.

A kernel two-sample test is a statistical hypothesis test that uses a kernel function to determine if two sets of observations are drawn from the same probability distribution. MMD is a specific, powerful instance of this test.

• Core Mechanism: It embeds probability distributions into a Reproducing Kernel Hilbert Space (RKHS) and compares their means.
• Key Advantage: It can detect any type of distributional difference (mean, variance, shape) if a characteristic kernel (like the Gaussian RBF kernel) is used.
• Application in ML: This is the direct theoretical framework for using MMD to compare a reference window of training data against a test window of production data for unsupervised drift detection.

A Reproducing Kernel Hilbert Space (RKHS) is a Hilbert space of functions where point evaluation is a continuous linear functional. It is the mathematical foundation that gives kernel methods like MMD their power.

• The "Kernel Trick": An RKHS allows algorithms to operate in a high-dimensional (often infinite-dimensional) feature space without explicitly computing the coordinates of the data in that space, using only the kernel function.
• Role in MMD: MMD computes the distance between the mean embeddings of two distributions within this RKHS. The choice of kernel (e.g., Gaussian, Laplacian) defines the properties of the RKHS and what differences MMD is sensitive to.
• Implication: Understanding RKHS explains why MMD can capture complex, non-linear distribution differences that simpler metrics like Population Stability Index (PSI) cannot.

A characteristic kernel is a kernel function for which the map from a probability distribution to its mean embedding in the Reproducing Kernel Hilbert Space (RKHS) is injective. This property is critical for the effectiveness of MMD.

• Key Property: If a kernel is characteristic, then the MMD between two distributions is zero if and only if the distributions are identical. This makes MMD a metric on the space of probability distributions.
• Common Examples: The Gaussian Radial Basis Function (RBF) kernel and the Laplacian kernel are universal characteristic kernels.
• Practical Impact: Using a characteristic kernel ensures the MMD test is consistent against all alternatives, meaning it will correctly identify any type of drift given enough data.

Wasserstein Distance, also known as the Earth Mover's Distance, is a metric that quantifies the minimum cost of transforming one probability distribution into another. Like MMD, it is used for two-sample testing and drift detection.

• Intuition: It frames the difference between distributions as a transportation problem: how much "mass" must be moved and how far to turn one histogram into another.
• Comparison to MMD: While MMD operates in a kernel-induced feature space, Wasserstein distance works directly on the sample space. It is often more interpretable and can be better for distributions with non-overlapping support.
• Computational Note: Calculating the exact Wasserstein distance is often more computationally intensive than the unbiased quadratic-time MMD estimate, especially in high dimensions.

Kullback-Leibler Divergence is a non-symmetric measure of how one probability distribution diverges from a second, reference probability distribution. It is a foundational concept in information theory and statistics.

• Definition: It measures the expected excess surprise from using the reference distribution to model the true distribution. It is zero only if the distributions are identical.
• Contrast with MMD: Unlike MMD, KL Divergence is not a metric (it's asymmetric and doesn't satisfy the triangle inequality). It can be infinite if the distributions have non-overlapping support, making it less robust for empirical drift detection on finite samples compared to MMD.
• Usage: Often used in variational inference and as a loss function. In drift detection, it's typically estimated for discrete distributions or used with density estimation, whereas MMD is a direct non-parametric test.

The unbiased quadratic-time MMD estimate is the standard, computationally efficient estimator used to compute MMD from finite samples. It is the workhorse formula for practical drift detection.

• Formula: For two samples X (size m) and Y (size n), the estimate is: MMD²_u = (1/(m(m-1))) Σ_{i≠j} k(x_i, x_j) + (1/(n(n-1))) Σ_{i≠j} k(y_i, y_j) - (2/(mn)) Σ_{i,j} k(x_i, y_j).
• Key Features: It is an unbiased estimator of the population MMD². The computation is O((m+n)²), which is feasible for moderate batch sizes in monitoring pipelines.
• Hypothesis Testing: This estimate is used in a permutation test or with an asymptotic distribution to compute a p-value for the null hypothesis that the two samples are from the same distribution, forming the basis of the statistical drift detection decision.