Synthetic Data Fidelity

Statistical fidelity measures how well the synthetic data preserves the marginal and joint probability distributions of the real data. This is the foundational layer of assessment, ensuring basic statistical properties like means, variances, and correlations are maintained.

• Core Metrics: Statistical distances like Wasserstein Distance, Jensen-Shannon Divergence, and Maximum Mean Discrepancy (MMD) are used to quantify distributional similarity.
• Validation: Techniques include two-sample tests (e.g., Kolmogorov-Smirnov) and training a domain classifier; if a classifier cannot distinguish real from synthetic samples, statistical fidelity is high.
• Pitfall: Perfect statistical match on training data can indicate overfitting by the generator, not generalizable fidelity.

Semantic fidelity assesses whether each synthetic data point is realistic and contextually meaningful within the problem domain, beyond just statistical likelihood. It ensures data points are not statistical outliers or nonsensical combinations.

• Evaluation Methods: Uses domain-specific rules, anomaly detection models, or discriminator networks from GANs to flag implausible samples.
• Example: In medical data, a synthetic record with a pregnancy flag for a male patient lacks semantic fidelity, even if the marginal distributions of gender and pregnancy are correct.
• Connection: Low semantic fidelity directly contributes to the synthetic-to-real gap, as models learn on invalid data patterns.

Relational fidelity evaluates the preservation of complex dependencies and multi-way interactions between features, as well as the topological structure of the data manifold. It is critical for datasets with intricate correlations or graph-like relationships.

• Advanced Metrics: Techniques like persistent homology from topological data analysis can reveal if the synthetic data has the same "shape" (e.g., clusters, loops) as the real data.
• Dimensionality: Comparing the intrinsic dimension of real and synthetic datasets can reveal if the generator has collapsed or altered the underlying data manifold.
• Importance: Failure here leads to mode collapse in generative models, where diversity is lost.

Downstream task fidelity is the ultimate validation metric, measured by the performance of a machine learning model trained exclusively on synthetic data when evaluated on a held-out set of real data. It directly tests the synthetic data's utility.

• Primary Measure: The performance delta (e.g., accuracy, F1-score) between a model trained on real data and one trained on synthetic data for the same task.
• Benchmarking: Requires establishing a model benchmarking suite for the target application (e.g., image classification, fraud detection).
• Outcome: High downstream task fidelity indicates the synthetic data has preserved the features most relevant for the model's learning objective.

The fidelity-privacy trade-off describes the fundamental tension between creating highly realistic synthetic data and guaranteeing the privacy of individuals in the source dataset. Increasing one typically reduces the other.

• Privacy Mechanisms: Techniques like differential privacy are explicitly designed to bound privacy loss but introduce statistical noise, reducing fidelity.
• Attack Resilience: High-fidelity synthetic data is more vulnerable to membership inference attacks, where an adversary can determine if a specific person's data was in the training set.
• Engineering Goal: The objective is to find the optimal point on this Pareto frontier for a given use case, maximizing utility while meeting privacy guarantees.

Temporal fidelity assesses how well synthetic data generation captures time-dependent patterns, trends, and concept drift present in real-world sequential or time-series data. It ensures the synthetic data is not just a static snapshot.

• Challenge: Must replicate autocorrelation, seasonality, and evolving relationships (concept drift).
• Evaluation: Compare the synthetic and real data's performance in forecasting future values or in detecting distributional shift over simulated time windows.
• Use Case: Critical for generating synthetic data for financial markets, IoT sensor streams, or customer behavior logs where timing is intrinsic to the signal.

A quantitative measure of the dissimilarity between two probability distributions, used as the mathematical foundation for assessing synthetic data fidelity. Key metrics include:

• Kullback-Leibler Divergence (KL Divergence): An asymmetric measure of how one distribution diverges from a reference.
• Jensen-Shannon Divergence: A symmetric, bounded version of KL divergence.
• Wasserstein Distance (Earth Mover's Distance): Measures the minimum 'cost' to transform one distribution into another, often more robust for high-dimensional data.

A de facto standard metric for evaluating the fidelity of synthetic images. It calculates the Wasserstein-2 distance between feature distributions extracted from real and generated images using a pre-trained Inception-v3 network. A lower FID score indicates the synthetic data is closer to the real data in the feature space of a powerful visual classifier.

A framework that decomposes generative model evaluation into two distinct aspects, providing more nuanced insight than a single score.

• Precision (Quality): Measures what fraction of the synthetic data lies within the support of the real data distribution. High precision means generated samples are highly realistic.
• Recall (Coverage/Diversity): Measures what fraction of the real data distribution is covered by the synthetic data. High recall means the synthetic data captures the full variability of the real data.

A practical method to detect distributional shift between real and synthetic datasets. A classifier (e.g., a neural network) is trained to distinguish between samples from the two domains. If the classifier achieves high accuracy, it indicates the distributions are easily separable, signaling low synthetic data fidelity. The goal is to produce synthetic data that 'fools' this classifier.

The observed performance degradation when a model trained exclusively on synthetic data is deployed on real-world data. This gap is the ultimate, practical consequence of imperfect fidelity. Closing it is the primary engineering goal of synthetic data generation, often validated through downstream task performance benchmarks.

The fundamental tension in synthetic data generation: maximizing statistical fidelity often increases the risk of privacy leakage (e.g., via membership inference attacks), while strong differential privacy guarantees typically introduce noise that reduces fidelity. Engineering solutions must explicitly balance these competing objectives based on the use case.