Kolmogorov-Smirnov Test

The KS test is a nonparametric (distribution-free) test, meaning it makes no assumptions about the underlying distribution of the data (e.g., normality). It operates directly on the empirical cumulative distribution functions (ECDFs) of the samples.

• Key Advantage: Robustness to unknown or non-standard data distributions.
• Comparison: Unlike parametric tests (e.g., t-test), it does not rely on parameters like mean or variance, making it suitable for complex, real-world data where distributional assumptions are often violated.

The core of the KS test is the D-statistic, defined as the supremum (maximum vertical distance) between the two empirical cumulative distribution functions (ECDFs).

• Calculation: (D_{n,m} = \sup_x |F_{1,n}(x) - F_{2,m}(x)|), where (F_{1,n}) and (F_{2,m}) are the ECDFs of the two samples.
• Interpretation: A large D value indicates a greater discrepancy between the two sample distributions. The test's p-value is derived by comparing the observed D to its distribution under the null hypothesis (that the samples are from the same distribution).

While a one-sample KS test compares a sample to a reference distribution, the two-sample KS test is the primary tool for synthetic data fidelity assessment. It directly compares the ECDFs of the real dataset and the synthetic dataset.

• Primary Use Case: Quantifying the distributional shift or synthetic-to-real gap.
• Output: A p-value indicating whether the observed difference is statistically significant. A high p-value (> 0.05) suggests the synthetic data distribution is not significantly different from the real data distribution.

The KS test is sensitive to differences in the overall shape of the cumulative distribution, rather than specific moments like mean or variance.

• Detects: Differences in median, spread, skewness, and multimodality as reflected in the ECDF.
• Limitation: It may be less powerful than specialized tests for detecting specific differences (e.g., a t-test for mean differences). It is a global test of distributional equality.
• Example: It can effectively identify if synthetic data fails to capture the tail behavior or bimodal structure present in the real data.

The KS test is inherently visual. Plotting the two ECDFs and the point of maximum distance (D) provides an intuitive diagnostic.

• KS Plot: A graph showing the step functions of the two ECDFs. The D-statistic is the largest gap between them.
• Engineering Utility: This visualization allows data scientists to immediately see where in the distribution (e.g., at low values, high values) the synthetic data diverges most significantly from the real data, guiding iterative improvements to the generative model.

The KS test is one of several statistical distance metrics. Its characteristics differ from alternatives:

• vs. KL Divergence: KS is a metric (symmetric, satisfies triangle inequality) while KL is not. KL can be infinite if distributions don't share support.
• vs. Wasserstein Distance: Wasserstein (Earth Mover's Distance) considers the "cost" of moving probability mass and is often more intuitive for high-dimensional data, but is computationally more intensive.
• vs. Maximum Mean Discrepancy (MMD): MMD uses kernel methods and is generally more powerful for high-dimensional data, but requires kernel choice. The KS test is simpler and provides an easy-to-interpret scalar and visual.

The Kolmogorov-Smirnov test is a two-sample goodness-of-fit test that quantifies the maximum vertical distance between the Empirical Cumulative Distribution Functions (ECDFs) of two datasets. It answers the question: "Are my real training data and my synthetic data drawn from the same underlying distribution?"

• Null Hypothesis (H₀): The two samples come from the same distribution.
• Test Statistic (D): The supremum (greatest) distance between the two ECDFs.
• P-value: The probability of observing a D statistic as extreme as the one calculated, assuming H₀ is true. A low p-value (e.g., < 0.05) provides evidence to reject H₀, indicating a statistically significant distributional shift.

In synthetic data generation, the KS test is a primary metric for quantitative fidelity assessment. It is applied feature-by-feature (univariately) to compare the distribution of each column (e.g., age, income, sensor reading) between real and synthetic datasets.

• Process: For each continuous feature, calculate the KS statistic (D) and p-value. A suite of tests across all features provides a multidimensional fidelity profile.
• Interpretation: A high p-value across most features suggests the synthetic data generator (e.g., a GAN, Variational Autoencoder, or diffusion model) is effectively capturing the marginal distributions of the source data.
• Limitation: The standard two-sample KS test is univariate and does not capture multivariate correlations, which must be assessed with other metrics.

The KS test is deployed in MLOps monitoring pipelines as a drift detection mechanism. It compares the distribution of incoming production data against a reference set (e.g., training data or a previous time window).

• Covariate Shift Detection: Applied to input feature distributions to alert when the data a model receives has changed.
• Prior Shift Detection: Applied to the distribution of model predictions or target labels in classification tasks.
• Operational Use: Automated KS tests run on batched or streaming data, triggering alerts or model retraining pipelines when the D statistic exceeds a predefined threshold, indicating significant data drift.

The KS test occupies a specific niche within the toolkit of statistical distance and divergence metrics. Its properties dictate its use cases:

• Strengths: Nonparametric (makes no distributional assumptions), easy to interpret (D is in the original data's units), and sensitive to differences in both location and shape of distributions.
• Vs. KL Divergence: KS is a true metric (symmetric in the two-sample test) and always finite, unlike the asymmetric, potentially infinite KL Divergence.
• Vs. Wasserstein Distance: KS is less sensitive to subtle differences in the tails of distributions compared to Wasserstein, which considers the "cost" of moving probability mass.
• Vs. Maximum Mean Discrepancy (MMD): KS operates on 1D ECDFs, while kernel-based MMD can capture high-dimensional distributional differences in a feature space.

Implementing the KS test requires careful consideration of data scale, dimensionality, and hypothesis testing pitfalls.

• Scalability: The test is computationally efficient, with a time complexity of O(m*n) for sample sizes m and n, often optimized to O((m+n) log(m+n)).
• Multiple Testing Problem: When testing hundreds of features, the chance of false positives increases. Corrections like the Bonferroni correction or False Discovery Rate (FDR) control must be applied.
• Categorical/Binned Data: The standard KS test is for continuous data. For ordinal or heavily binned data, the Chi-squared test or Cramér–von Mises criterion may be more appropriate.
• Visual Companion: The KS statistic is perfectly visualized by plotting the two ECDFs; the D statistic is the largest gap between them.

The primary limitation of the standard two-sample KS test is its univariate nature. High-fidelity synthetic data must preserve not just marginal distributions but also joint distributions and correlations.

• The Curse of Dimensionality: Applying a univariate test to each of 1,000 features gives no guarantee about their 1,000-dimensional joint distribution.
• Complementary Techniques: To assess multivariate fidelity, the KS test is used in conjunction with:
  • Dimension Reduction: Apply KS test to principal components from PCA or embeddings from UMAP/t-SNE.
  • Domain Classifier Test: Train a model to distinguish real from synthetic; a low AUC indicates good multivariate fidelity.
  • Downstream Task Performance: The ultimate test—train a model on synthetic data and evaluate it on held-out real data.

A quantitative measure of dissimilarity between two probability distributions. It is the foundational concept underlying all synthetic data fidelity metrics.

• Core Metrics: Includes Kullback-Leibler Divergence, Jensen-Shannon Divergence, Wasserstein Distance, and Maximum Mean Discrepancy.
• Purpose: Provides a scalar value summarizing how far a synthetic data distribution is from the real data distribution.
• Selection: The choice of distance metric depends on the data type (continuous vs. discrete) and the specific properties being compared (e.g., mass transport vs. density ratios).

A statistical hypothesis test used to determine if two sets of observations are drawn from the same underlying probability distribution. The Kolmogorov-Smirnov Test is a prominent nonparametric example.

• Null Hypothesis: The two samples originate from identical distributions.
• Output: Produces a p-value; a low p-value (e.g., < 0.05) provides evidence to reject the null hypothesis, indicating a statistically significant difference.
• Alternatives: Other two-sample tests include the Anderson-Darling test (more sensitive to tails) and the Cramér–von Mises criterion.

A kernel-based statistical test for determining if two samples are from different distributions. It compares the means of the samples after mapping them into a high-dimensional reproducing kernel Hilbert space (RKHS).

• Advantages over KS: Can be applied to multivariate and high-dimensional data, not just univariate. More powerful for complex, structured data like images.
• Mechanism: If the means of the two mapped distributions are close, the samples are likely from the same distribution.
• Use Case: Commonly used to evaluate the fidelity of synthetic datasets in complex domains where KS is insufficient.

A distance metric between two probability distributions based on optimal transport theory. It measures the minimum "cost" of transforming one distribution into the other.

• Intuition: Imagine piles of earth (probability mass); the distance is the minimum amount of work needed to move earth from one pile configuration to another.
• Properties: Symmetric, satisfies the triangle inequality, and is sensitive to the geometry of the underlying space.
• Application: The basis for the Fréchet Inception Distance (FID), the standard metric for evaluating synthetic image quality.

A practical method to detect distributional shift between two datasets (e.g., real vs. synthetic). A classifier is trained to distinguish between the two sources.

• Interpretation: High classifier accuracy indicates the two datasets are easily separable, revealing significant distributional differences. Low accuracy suggests they are statistically similar.
• Process: 1. Label real data as 0, synthetic as 1. 2. Train a model (e.g., XGBoost) to predict the label. 3. Evaluate on a held-out set.
• Utility: Provides a model-based, often multivariate, assessment of fidelity that complements statistical tests like KS.

A framework that decomposes generative model evaluation into two separate metrics: quality (precision) and coverage/diversity (recall) of the generated data.

• Precision: The fraction of synthetic samples that lie within the support of the real data distribution (are realistic).
• Recall: The fraction of real data modes that are captured by the synthetic data distribution.
• Significance: Addresses the limitation of single-score metrics. A model can have high precision but low recall (mode collapse), or vice-versa. This framework reveals such trade-offs explicitly.