Two-Sample Test

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test for determining if two samples originate from the same distribution. It works by comparing the means of the two samples after mapping them into a high-dimensional Reproducing Kernel Hilbert Space (RKHS).

• Mechanism: Computes the distance between the mean embeddings of the two distributions. A low MMD value suggests the distributions are similar.
• Key Advantage: It is a non-parametric test that can detect complex, non-linear differences.
• Common Use: A standard metric for evaluating the fidelity of synthetic data in high-dimensional spaces, such as images or embeddings.

Train-on-Synthetic Test-on-Real (TSTR) is a pragmatic utility evaluation protocol. It measures the practical value of synthetic data by training a downstream machine learning model (e.g., a classifier) entirely on the synthetic dataset and then evaluating its performance on a held-out set of real data.

• Primary Metric: The performance score (e.g., accuracy, F1-score) on the real test set.
• Interpretation: High TSTR performance indicates the synthetic data has preserved the statistical patterns necessary for the model to generalize to reality.
• Critical Insight: This is often considered the ultimate test of synthetic data utility for a specific task.

The Domain Classifier Test is an adversarial evaluation method. A discriminative model (the classifier) is trained to distinguish between samples from the real and synthetic datasets.

• Evaluation Metric: The classifier's accuracy. Near 50% accuracy (random guessing) indicates the two datasets are statistically indistinguishable.
• Process: If the classifier fails to learn a decision boundary, it suggests high fidelity in the synthetic data.
• Relation to GANs: This principle is directly inspired by the training dynamics of Generative Adversarial Networks, where a discriminator's failure signals generator success.

Precision and Recall for Distributions (P&R) is a two-dimensional metric that separately quantifies the quality and diversity/coverage of a generative model's output.

• Precision: Measures the fraction of synthetic samples that lie within the support of the real data distribution (are realistic). High precision indicates high quality.
• Recall: Measures the fraction of real data modes that are captured by the synthetic distribution. High recall indicates good coverage and diversity.
• Visualization: Results are often plotted on a 2D graph, providing a more nuanced view than a single-score metric and helping diagnose issues like mode collapse (high precision, low recall).

Wasserstein Distance, also known as the Earth Mover's Distance, is a metric from optimal transport theory that quantifies the minimum "cost" of transforming one probability distribution into another.

• Intuition: It conceptualizes distributions as piles of earth; the distance is the minimum amount of work needed to move earth from one pile to shape the other.
• Advantages: Unlike Kullback-Leibler (KL) Divergence, it is a true metric (symmetric, obeys triangle inequality) and can handle distributions with non-overlapping supports.
• Application in ML: The basis for Wasserstein GANs (WGANs) and a robust measure for comparing high-dimensional real and synthetic distributions.

Adversarial Validation is a practical technique to quantify the distribution shift between two datasets, such as training and validation sets, or real and synthetic data. It involves training a classifier (e.g., XGBoost) to discriminate between the two sets.

• Output Metric: The classifier's Area Under the ROC Curve (AUC). An AUC of 0.5 suggests no detectable shift. An AUC approaching 1.0 indicates the sets are easily separable, signaling a significant distribution mismatch.
• Proactive Use: Commonly used in machine learning pipelines to detect data drift before model training, ensuring validation sets are representative.

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test used to determine if two samples (e.g., real and synthetic) are drawn from the same distribution. It computes the distance between the means of the two samples after mapping them into a high-dimensional Reproducing Kernel Hilbert Space (RKHS).

• Key Advantage: Non-parametric and can detect complex, non-linear distribution differences.
• Common Use: A popular test statistic within two-sample testing frameworks for high-dimensional data like images.
• Relation to Two-Sample Test: MMD provides the test statistic; the hypothesis test (Two-Sample Test) uses it to calculate a p-value.

Fréchet Inception Distance (FID) is a specialized metric for evaluating synthetic image quality. It is essentially a parametric, efficient approximation of the Wasserstein-2 distance between multivariate Gaussian distributions fitted to the features of real and generated images, extracted by a pre-trained Inception-v3 network.

• How it works: Lower FID scores indicate that the synthetic image distribution is closer to the real image distribution.
• Limitation: Assumes features are normally distributed, which is a simplification.
• Relation to Two-Sample Test: FID is a metric derived from distribution distance, while a two-sample test is a formal hypothesis test. FID gives a distance score; a two-sample test provides a statistical significance (p-value).

Precision and Recall for Distributions (P&R) is a two-dimensional metric that separately evaluates the quality and diversity/coverage of a generative model's output.

• Precision: Measures what fraction of the synthetic data lies within the support of the real data (are the generated samples realistic?).
• Recall: Measures what fraction of the real data is covered by the support of the synthetic data (does the generator capture all modes of the real distribution?).
• Key Insight: A two-sample test asking "are the distributions the same?" is a one-dimensional question. P&R decomposes this into two critical, often competing, axes of performance, providing more nuanced diagnostic power.

Adversarial Validation is a practical technique to detect distribution shift. A domain classifier (e.g., a neural network) is trained to discriminate between samples from the real and synthetic datasets.

• Interpretation: If the classifier achieves near-random accuracy (e.g., 50% AUC), it suggests the two datasets are indistinguishable, indicating high synthetic data fidelity.
• Process: This is an empirical two-sample test. The classifier's performance is the test statistic, and its failure is the desired outcome.
• Utility: Directly measures whether a downstream model could detect the data as synthetic, which is a strong proxy for utility in training.

Train-on-Synthetic Test-on-Real (TSTR) is the ultimate utility-focused evaluation protocol. It measures the practical value of synthetic data for downstream machine learning tasks.

• Protocol: 1. Train a model (e.g., a classifier) exclusively on the synthetic dataset. 2. Evaluate the trained model's performance on a held-out set of real data.
• Outcome: High performance indicates the synthetic data has preserved the statistical relationships necessary for learning. This is a more task-specific and stringent test than general distribution similarity.
• Relation: A two-sample test establishes fidelity (distributions match). TSTR establishes utility (the data is useful for a specific purpose).

Wasserstein Distance (Earth Mover's Distance) is a metric that quantifies the minimum "cost" of transforming one probability distribution into another. It is well-suited for distributions with non-overlapping support.

• Intuition: Imagine piles of earth (one distribution) that must be moved to match another pile. The Wasserstein distance is the minimum amount of "work" (mass × distance) required.
• Advantage over KL Divergence: It remains finite and continuous even when distributions have little overlap.
• Role in Validation: Used as the test statistic in optimal transport-based two-sample tests (e.g., Sinkhorn divergence). It's also the theoretical foundation for metrics like FID and is central to training Generative Adversarial Networks (GANs).