Fidelity Score

These metrics quantify the distance between the probability distributions of real and synthetic data.

• Maximum Mean Discrepancy (MMD): A kernel-based test that compares the means of the two datasets in a high-dimensional feature space. A low MMD score indicates high fidelity.
• Wasserstein Distance (Earth Mover's Distance): Measures the minimum 'cost' of transforming the synthetic distribution into the real distribution. It is more robust than metrics like KL divergence when distributions have non-overlapping support.
• Kullback-Leibler (KL) Divergence: An information-theoretic measure of how one distribution diverges from a second, reference distribution. It is asymmetric and can be unstable if the synthetic distribution assigns zero probability to regions where real data exists.

Specialized for visual data, these metrics use pre-trained neural networks to extract and compare high-level features.

• Fréchet Inception Distance (FID): The most widely adopted metric for image generation. It calculates the Wasserstein-2 distance between multivariate Gaussians fitted to the feature activations of real and synthetic images from a pre-trained Inception-v3 network. Lower FID is better.
• Kernel Inception Distance (KID): An unbiased alternative to FID that computes the squared Maximum Mean Discrepancy using a polynomial kernel on the Inception features. It does not assume a Gaussian distribution of features.
• Precision & Recall for Distributions: A two-dimensional metric that separates quality from coverage. Precision measures the fraction of synthetic images that are within the manifold of real images. Recall measures the fraction of real images that are within the manifold of synthetic images.

The most pragmatic fidelity tests measure how well synthetic data performs in its intended application.

• Train-on-Synthetic, Test-on-Real (TSTR): A primary utility test. A model (e.g., a classifier) is trained exclusively on synthetic data and evaluated on a held-out real test set. High performance indicates the synthetic data preserved the feature-target relationships.
• Train-on-Real, Test-on-Synthetic (TRTS): A complementary fidelity test. A model trained on real data is evaluated on synthetic data. Minimal performance drop suggests the synthetic data's internal relationships mirror reality.
• Domain Classifier Test: A discriminative model is trained to distinguish real from synthetic samples. If the classifier's accuracy is near 50% (random chance), it indicates the distributions are indistinguishable, signaling high fidelity.

Fidelity must be balanced with privacy and the avoidance of bias amplification. These audits ensure synthetic data is safe and equitable.

• Membership Inference Attack (MIA): Attempts to determine if a specific real data record was in the generator's training set. A robust synthetic dataset should resist such attacks, showing no more leakage than random guessing.
• Differential Privacy (DP) Audit: Empirically tests if the synthesis mechanism meets a formal (ε, δ)-DP guarantee by attempting to detect statistical differences in output with and without a specific record.
• Bias and Fairness Metrics: Compare statistical parity (demographic parity) and other fairness metrics (e.g., equalized odds) between models trained on real vs. synthetic data to ensure biases are not amplified.

Validating synthetic tabular data requires checking statistical properties, correlations, and logical constraints.

• Column-Wise Statistics: Compare means, medians, standard deviations, and value ranges for each column between real and synthetic data.
• Pairwise Correlation Preservation: Assess if the correlation matrix (Pearson, Spearman) of the synthetic data matches that of the real data. Critical for multivariate analysis.
• Semantic Integrity & Plausibility: Ensure generated records obey domain-specific rules (e.g., age > years of experience, pregnancy flag matches gender). Violations indicate poor conditional sampling fidelity.
• Support Coverage: Evaluate if the synthetic data covers the full range (min/max) and rare combinations of categories present in the real data.

Visual and diagnostic methods provide intuitive checks for distribution alignment and failure modes.

• t-SNE / UMAP Visualization: Dimensionality reduction techniques plot real and synthetic data points in 2D/3D. Overlapping, intermingled clusters indicate good fidelity; separate clusters reveal distribution shift.
• Adversarial Validation: A robust method to detect any systematic distribution shift. A powerful model (e.g., XGBoost) is trained to classify 'real' vs. 'synthetic'. An AUC-ROC near 0.5 suggests the datasets are indistinguishable.
• Detection of Mode Collapse & Out-of-Distribution (OOD) Samples: Visualize the diversity of synthetic samples and use OOD detectors to flag unrealistic extrapolations that fall outside the real data's support.

Fréchet Inception Distance (FID) is a cornerstone metric for evaluating the quality of synthetic images. It calculates the Wasserstein-2 distance between the feature distributions of real and generated data, as extracted by a pre-trained Inception-v3 network. A lower FID score indicates the synthetic images are statistically closer to the real images. It is widely used but has limitations, such as sensitivity to the number of samples and the specific features captured by the Inception network.

• Mechanism: Embeds images into a feature space and compares the multivariate Gaussian distributions.
• Use Case: Standard benchmark for GANs and diffusion models in computer vision.

Precision and Recall for Distributions (P&R) is a two-dimensional metric that separately quantifies the quality and diversity of generated data. Unlike a single scalar score, it provides a more nuanced diagnostic.

• Precision: Measures the fraction of synthetic samples that lie within the support of the real data distribution. High precision indicates generated samples are realistic.
• Recall: Measures the fraction of real data modes that are covered by the support of the synthetic distribution. High recall indicates the model captures the full diversity of the real data.

This breakdown helps diagnose specific failure modes like mode collapse (high precision, low recall).

Train-on-Synthetic Test-on-Real (TSTR) is a utility-focused evaluation protocol. It directly measures how useful synthetic data is for training downstream machine learning models.

Procedure:

1. Train a predictive model (e.g., a classifier) entirely on the synthetic dataset.
2. Evaluate the model's performance on a held-out set of real data.

Interpretation: High performance indicates the synthetic data has preserved the statistical relationships necessary for the task. This is often considered the ultimate test of synthetic data utility for a specific application, moving beyond pure distribution matching.

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test used to determine if two samples (real and synthetic) are drawn from the same distribution. It works by comparing the mean embeddings of the datasets in a Reproducing Kernel Hilbert Space (RKHS).

• Mechanism: If the means of the two distributions are close in this high-dimensional space, the distributions are considered similar. A low MMD score indicates high fidelity.
• Advantage: It is a non-parametric test that can be applied to any data type (images, text, tabular) with an appropriate kernel.
• Relation: The Kernel Inception Distance (KID) metric is derived from MMD, using a polynomial kernel on Inception network features.

A Domain Classifier is a discriminative model (e.g., a neural network) trained to distinguish between samples from the real dataset and the synthetic dataset. Its performance is used as a proxy metric for fidelity.

Evaluation:

• If the classifier achieves near-random accuracy (e.g., 50% for binary classification), it cannot tell the datasets apart, suggesting high fidelity.
• If accuracy is high, the synthetic data is easily distinguishable, indicating a distribution gap.

This approach is conceptually linked to the adversarial training in GANs, where the generator's goal is to fool a discriminator.

The Privacy-Utility Frontier is a conceptual curve that illustrates the fundamental trade-off in synthetic data generation. It plots the level of privacy protection (e.g., a Differential Privacy ε budget) against the statistical utility or fidelity of the output dataset.

• High Privacy, Low Utility: Strong privacy guarantees often require adding more noise or constraints, which degrades the dataset's statistical similarity to the original.
• Low Privacy, High Utility: A very accurate synthetic dataset may risk leaking information about individuals in the training data.

Effective synthetic data generation involves explicitly defining a target operating point on this frontier based on project requirements.