Data Plausibility

Statistical fidelity ensures the synthetic data preserves the core statistical properties of the source distribution. This is the foundational requirement for plausibility.

• Key metrics include comparing marginal distributions (for individual features), joint distributions (for feature relationships), and summary statistics (mean, variance, covariance).
• High fidelity means a domain classifier or a two-sample test like Maximum Mean Discrepancy (MMD) cannot reliably distinguish real from synthetic batches.
• Failure results in data drift, where models trained on synthetic data fail on real-world inputs.

Semantic integrity ensures that generated data obeys the logical, physical, and business rules of the domain. A sample can be statistically similar yet semantically nonsense.

• For tabular data, this means respecting constraints (e.g., age ≥ 0), functional dependencies (e.g., zip_code implies city), and causal relationships.
• In images, it requires physically possible object arrangements and lighting.
• Violations are often detected via rule-based checks or out-of-distribution (OOD) detection against known valid states. This characteristic is critical for structured data generation.

Plausible data must adequately cover the support—the full range of possible values—of the real data distribution without introducing unrealistic outliers.

• This guards against mode collapse, where a generator produces limited, repetitive samples.
• Metrics like Precision and Recall for Distributions (P&R) separately measure quality (are generated samples realistic?) and coverage (does the synthetic set represent all real modes?).
• t-SNE visualizations are commonly used to visually assess the overlap and gaps between real and synthetic clusters in a reduced space.

The ultimate test of plausibility is utility: can the data be used to train a model that performs well on real-world tasks? This moves beyond statistical similarity to functional validity.

• Evaluated via the Train-on-Synthetic Test-on-Real (TSTR) protocol. A high-performing model indicates the synthetic data captured the salient patterns for the task.
• The inverse, Train-on-Real Test-on-Synthetic (TRTS), checks if a real-world model generalizes to the synthetic set, further validating distribution alignment.
• This characteristic directly ties plausibility to business objectives and model performance.

For conditional generation models, plausibility requires that generated samples accurately reflect the specified input conditions or attributes.

• Examples include generating a synthetic chest X-ray that correctly shows the pathology described in a conditional label, or creating a customer profile with a specified income bracket and purchase history.
• Conditional sampling fidelity is measured by training a classifier on the conditions and testing it on the synthetic data; high accuracy indicates the generator correctly implemented the conditioning.
• This is essential for generating data for specific, targeted edge cases or class-balancing.

A plausible synthetic sample must not be a memorized or easily invertible copy of a real individual's data. Plausibility requires novelty that protects privacy.

• This is assessed through privacy attacks like membership inference, which tries to determine if a specific real record was in the training set.
• Differential Privacy (DP) audits provide formal guarantees, but any synthesis process must be evaluated for unintended data leakage.
• The goal is to navigate the privacy-utility frontier, maximizing plausibility and utility without compromising the privacy of the source data.

A fidelity score is a quantitative metric measuring how closely a synthetic dataset matches the statistical properties of a target real dataset. It is an umbrella term for various statistical tests and distance measures.

• Core Purpose: To provide a single-number summary of distributional similarity.
• Common Metrics: Includes Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and Wasserstein Distance.
• Limitation: A high fidelity score does not guarantee the synthetic data is useful for downstream tasks; it only confirms statistical resemblance.

Precision and Recall for Distributions (P&R) is a two-dimensional metric that separately quantifies the quality and diversity of generated data.

• Precision: Measures the fraction of synthetic samples that are realistic (i.e., within the support of the real data). High precision indicates high sample quality.
• Recall: Measures the fraction of real data modes that are captured by the synthetic data. High recall indicates good coverage and diversity.
• Advantage: Provides a more nuanced view than a single score, helping diagnose issues like mode collapse (high precision, low recall).

Downstream task performance is the ultimate validation metric, measuring the utility of synthetic data for its intended application.

• Evaluation Protocols:
  • Train-on-Synthetic Test-on-Real (TSTR): A model is trained on synthetic data and tested on real data. High performance indicates the synthetic data preserved task-relevant information.
  • Train-on-Real Test-on-Synthetic (TRTS): A model trained on real data is tested on synthetic data. Performance degradation can reveal distributional gaps.
• Interpretation: Strong TSTR results are the strongest indicator that synthetic data is a viable substitute for real data for model training.

A two-sample test is a statistical hypothesis test used to determine if two sets of observations (real vs. synthetic data) are drawn from the same underlying distribution.

• Null Hypothesis: The real and synthetic data distributions are identical.
• Common Tests: Maximum Mean Discrepancy (MMD) and Kolmogorov-Smirnov tests are frequently used.
• In Practice: Failure to reject the null hypothesis (high p-value) suggests the synthetic data is statistically indistinguishable from the real data, supporting its plausibility and fidelity.

Support coverage evaluates the extent to which a synthetic dataset spans the entire range of possibilities (the support) present in the real data distribution.

• Relation to Plausibility: A sample can be plausible (realistic) but still fail to cover rare edge cases in the real data's support.
• Measurement: Often assessed via Precision and Recall for Distributions, where recall directly measures coverage.
• Importance: Incomplete coverage can lead to models that perform poorly on underrepresented scenarios, a critical failure for safety-critical applications like autonomous driving or healthcare.

Semantic integrity is a quality dimension for structured or relational synthetic data, ensuring that logical relationships and constraints between attributes are preserved.

• Examples:
  • A generated patient record should not have an Age of 5 and a Job Title of "CEO."
  • In relational data, foreign key relationships must remain valid.
• Validation: Requires domain-specific rules or constraint-checking algorithms beyond statistical tests.
• Distinction from Plausibility: While plausibility asks "Is this sample believable?", semantic integrity asks "Is this sample internally consistent and logically valid?"