Statistical Distance

A symmetric distance satisfies D(P, Q) = D(Q, P). This property is crucial for direct comparison, as it ensures the distance from the real to synthetic distribution is the same as from synthetic to real. Jensen-Shannon Divergence and Wasserstein Distance are symmetric. Kullback-Leibler Divergence is famously asymmetric, measuring the information loss when Q is used to approximate P, which is not the same as the reverse.

A true metric satisfies four axioms: non-negativity, identity of indiscernibles (D(P,Q)=0 iff P=Q), symmetry, and the triangle inequality. Distances with these properties, like Wasserstein Distance, enable reliable geometric reasoning in the space of probability distributions. Many divergences, like KL, are not metrics. The triangle inequality (D(P,R) ≤ D(P,Q) + D(Q,R)) is particularly important for tasks like interpolation and proving convergence guarantees.

This property defines how a distance behaves when distributions have non-overlapping regions of probability mass. KL Divergence becomes infinite if P assigns probability zero where Q assigns positive probability, making it extremely sensitive. Wasserstein Distance, in contrast, provides a smooth, finite measure based on the 'cost' of moving probability mass to align the supports. This makes Wasserstein more robust for comparing distributions that may have little overlap, a common scenario in early-stage synthetic data generation.

The feasibility of calculating the distance from finite samples is a primary engineering concern. Wasserstein Distance has a well-defined sample-based estimator but can be computationally expensive for high-dimensional data. Maximum Mean Discrepancy (MMD) offers a kernel-based estimator that is often more scalable. Jensen-Shannon Divergence can be estimated via density models or classifier-based approximations (like the Domain Classifier Test), trading off accuracy for speed.

The scale and meaning of the distance value matter for setting thresholds and communicating results. Wasserstein Distance in one dimension has intuitive units (e.g., dollars, meters) as it computes the cost of moving earth. KL Divergence is measured in bits or nats (units of information). Jensen-Shannon Divergence is bounded between 0 and 1, providing a normalized score. Unbounded measures like KL can be difficult to contextualize without a baseline.

This refers to how well the empirical estimate of the distance, computed from a finite number of samples, converges to the true population distance. Distances with poor sample efficiency require prohibitively large sample sizes for reliable estimates in high dimensions. Kernel-based measures like MMD often have favorable convergence properties. Understanding this is key for designing statistically powerful two-sample tests to reliably detect distributional shifts between real and synthetic datasets.

An asymmetric statistical distance that measures the information lost when one probability distribution is used to approximate another. It is defined as the expected logarithmic difference between the distributions. A key property is that KL(P||Q) ≠ KL(Q||P), making it sensitive to the choice of reference distribution. It is widely used in variational inference and model training but can be infinite if the support of Q does not cover P.

A metric from optimal transport theory that measures the minimum "cost" of transforming one probability distribution into another. Intuitively, it calculates the least amount of probability mass that must be moved, multiplied by the distance it is moved. Unlike KL divergence, it is symmetric and provides a smooth, meaningful gradient even when distributions have non-overlapping support. It is the foundation for the Fréchet Inception Distance (FID) metric for images.

A kernel-based statistical test that determines if two samples are from different distributions by comparing their means in a high-dimensional reproducing kernel Hilbert space (RKHS). If the means of the embedded samples are close, the distributions are deemed similar. MMD provides a unified framework for a two-sample test and is differentiable, making it useful as a loss function in generative models to match real and synthetic data distributions.

A statistical hypothesis test used to determine whether two sets of observations are drawn from the same underlying probability distribution. The null hypothesis is that the samples are from the same distribution. Common nonparametric tests include:

• Kolmogorov-Smirnov Test: Measures the maximum distance between two empirical cumulative distribution functions.
• Permutation Tests: Compute a test statistic by randomly shuffling data labels. These tests are foundational for adversarial validation and detecting distributional shift.

A framework that decomposes generative model evaluation into two separate metrics, extending the concepts from information retrieval to probability distributions.

• Precision: Measures the quality of generated samples (what fraction of the synthetic distribution is contained within the real distribution).
• Recall: Measures the coverage of the real data (what fraction of the real distribution is covered by the synthetic distribution). This approach provides a more nuanced diagnosis than a single distance metric, revealing issues like mode collapse (high precision, low recall).

A practical method to detect distributional shift between two datasets (e.g., training vs. test, real vs. synthetic). The procedure is:

1. Label data from one domain as 0 and the other as 1.
2. Train a classifier (e.g., a gradient-boosted tree) to distinguish between them.
3. Evaluate the classifier's performance (e.g., AUC-ROC). A high-performing classifier indicates the domains are statistically distinguishable, signaling a significant shift that may degrade model performance. It is a crucial diagnostic before model deployment.