Two-Sample Test

Parametric two-sample tests assume the data is drawn from a specific probability distribution, typically a normal distribution. The independent two-sample t-test is the most common, comparing the means of two groups. It requires assumptions of normality and homogeneity of variances (equal variance between groups).

• Use Case: Comparing the average pixel intensity of real vs. synthetic images.
• Variant: Welch's t-test, which does not assume equal variances.
• Key Statistic: Computes a t-value based on the difference between sample means, scaled by the variability in the data.

Nonparametric two-sample tests make no assumptions about the underlying data distribution, making them robust for non-normal or ordinal data. The Mann-Whitney U test (also called Wilcoxon rank-sum test) determines if one sample is stochastically greater than the other by comparing the ranks of the observations.

• Use Case: Comparing the median sentiment scores from two different text generators.
• Advantage: Robust to outliers and applicable to ranked data.
• Method: Ranks all observations from both groups together, then compares the sum of ranks for each group.

The chi-square test of independence is used for two-sample comparisons when the data is categorical. It assesses whether the distribution of categorical outcomes differs between two groups by comparing observed frequencies to expected frequencies under the null hypothesis of no association.

• Use Case: Testing if the proportion of 'approved' vs. 'denied' loan decisions is the same in real and synthetic financial datasets.
• Requirement: Adequate sample size with expected cell counts typically >5.
• Output: A chi-square statistic that measures the divergence of observed counts from the expected distribution.

Kernel-based tests like Maximum Mean Discrepancy (MMD) are powerful, nonparametric methods for comparing high-dimensional or complex distributions. MMD computes the distance between the means of two samples after mapping them into a high-dimensional reproducing kernel Hilbert space (RKHS).

• Use Case: The definitive test for detecting differences between the multivariate distributions of real and synthetic tabular or feature data.
• Advantage: Can capture higher-order statistical moments and complex dependencies.
• Connection: Forms the theoretical basis for training Generative Adversarial Networks (GANs).

These tests compare the full empirical cumulative distribution functions (ECDFs) of two samples. The Kolmogorov-Smirnov (K-S) test statistic is the maximum vertical distance between the two ECDFs. It is sensitive to differences in location, shape, and spread of the distributions.

• Use Case: Detecting if the distribution of transaction amounts in synthetic data matches the real data's distribution.
• Property: Nonparametric and applicable to any continuous distribution.
• Limitation: Less powerful for detecting differences in the tails of distributions compared to the center.

Tests based on optimal transport theory measure the minimum 'work' required to transform one distribution into another. The Wasserstein distance (Earth Mover's Distance) is a metric between distributions. Relatedly, the energy distance is a statistical distance that can be used for hypothesis testing.

• Use Case: Quantifying the fidelity of synthetic image datasets, as used in the Fréchet Inception Distance (FID) metric.
• Advantage: Provides a geometrically intuitive and differentiable distance metric.
• Application: Central to modern generative model evaluation and training paradigms like Wasserstein GANs.

A quantitative measure of the dissimilarity between two probability distributions. It is the foundational concept underlying all two-sample tests and synthetic data fidelity metrics.

• Purpose: Provides a single number summarizing how different two datasets are.
• Examples: Includes metrics like Kullback-Leibler Divergence, Wasserstein Distance, and Maximum Mean Discrepancy.
• Use in Fidelity Assessment: A smaller distance indicates higher synthetic data fidelity.

A kernel-based statistical test used to determine if two samples are drawn from different distributions. It is a popular non-parametric two-sample test in machine learning.

• Mechanism: Compares the means of the two samples after mapping them into a high-dimensional reproducing kernel Hilbert space (RKHS).
• Advantages: Can detect complex, nonlinear differences between distributions.
• Application: Directly used as a loss function in some generative models to encourage distributional alignment.

A nonparametric two-sample test that quantifies the distance between the empirical cumulative distribution functions (ECDFs) of two one-dimensional samples.

• Test Statistic: The KS statistic is the maximum vertical distance between the two ECDFs.
• Limitation: Primarily designed for univariate data. For multivariate data, it is often applied feature-by-feature.
• Interpretation: A large KS statistic (and small p-value) provides evidence that the two samples come from different distributions.

A practical method for detecting distributional shift by training a binary classifier to distinguish between samples from two datasets (e.g., real vs. synthetic).

• Procedure: Combine and label the datasets, train a classifier (e.g., a gradient boosting machine), and evaluate its accuracy.
• Interpretation: Classifier accuracy near 50% indicates the datasets are indistinguishable. High accuracy indicates a significant, detectable shift.
• Advantage: Leverages powerful discriminative models to find complex, multivariate differences.

A framework for evaluating generative models that decomposes fidelity into two separate components: quality and coverage.

• Precision: Measures the fraction of synthetic samples that are realistic (i.e., fall within the support of the real data distribution). High precision indicates high quality.
• Recall: Measures the fraction of real data modes that are captured by the synthetic data. High recall indicates good coverage.
• Insight: A two-sample test might pass if recall is high but precision is low (generated data covers real data but includes many unrealistic outliers). This framework provides a more nuanced view.

The ultimate, application-driven evaluation of synthetic data fidelity. It measures how well a model trained on synthetic data performs on its intended real-world task.

• Principle: High-fidelity synthetic data should preserve the task-relevant statistical relationships of the original data.
• Methodology: 1) Train Model A on real data. 2) Train Model B on synthetic data. 3) Evaluate both models on a held-out real test set. Compare performance metrics (e.g., accuracy, F1-score).
• Gold Standard: If Model B's performance is statistically indistinguishable from Model A's, the synthetic data has high fidelity for that task.