Train-on-Real Test-on-Synthetic (TRTS)

TRTS is a fidelity assessment protocol, not a utility test. The core premise is that a model trained on real data has learned the true data manifold. If the synthetic data is a high-fidelity replica, this model should perform well on it. A significant performance drop indicates the synthetic data has distributional shift or lacks key statistical properties captured by the real-trained model.

• Primary Goal: Quantify how well synthetic data mimics the statistical essence of real data.
• Contrast with TSTR: TRTS tests fidelity; Train-on-Synthetic Test-on-Real (TSTR) tests utility.
• Interpretation: High TRTS performance suggests the synthetic data's feature space aligns with reality, a prerequisite for downstream training tasks.

The protocol requires three datasets: a real training set (R_train), a real test set (R_test) for baseline, and the synthetic test set (S_test).

1. Train Classifier: A model (e.g., a ResNet for images, XGBoost for tabular data) is trained on R_train.
2. Establish Baseline: Evaluate the model on R_test to get the real-world performance ceiling (e.g., 95% accuracy).
3. Synthetic Evaluation: Evaluate the same, frozen model on S_test.

Key Metrics:

• Performance Gap: The absolute difference in accuracy/F1-score/AUC between R_test and S_test.
• Relative Performance: S_test performance as a percentage of R_test performance (e.g., 92% of real performance).
• Per-Class Analysis: Breakdown of performance gap across different classes or segments to identify where synthesis fails.

TRTS provides a direct, model-based measure of distributional similarity that is often more sensitive than pure statistical tests.

• Model as a Distributional Probe: Leverages the model's learned representation, which can capture complex, high-order statistical relationships that simple moment-matching misses.
• Task-Relevant Fidelity: Measures fidelity in the context of a specific modeling task (e.g., classification), which is often the end goal of the data.
• Interpretable Output: A percentage or score gap is easily communicated and tracked over time.
• Detects Subtle Flaws: Can reveal issues like semantic corruption (an image looks right to a human but the classifier fails) or covariate shift that simpler metrics might not catch.

TRTS is a powerful tool but must be applied with an understanding of its constraints.

• Dependence on Base Model Quality: If the model trained on R_train is poor or underfitted, the TRTS metric becomes unreliable.
• Not a Measure of Diversity: A model could score highly on TRTS even if the synthetic data suffers from mode collapse, as long as the collapsed modes are perfectly replicated.
• Task Specificity: A good TRTS score for one task (e.g., object classification) does not guarantee fidelity for another (e.g., object segmentation).
• Baseline Requirement: Requires a held-out real test set (R_test), which consumes valuable real data that could otherwise be used for training.

TRTS should never be used in isolation. A robust synthetic data validation pipeline combines it with other metrics to form a complete picture.

• Pair with TSTR: While TRTS measures fidelity, Train-on-Synthetic Test-on-Real (TSTR) measures utility. The ideal dataset scores highly on both.
• Statistical Metrics: Use Fréchet Inception Distance (FID), Kernel Inception Distance (KID), or Maximum Mean Discrepancy (MMD) to measure distributional overlap directly.
• Diversity Metrics: Use Precision and Recall for Distributions (P&R) to ensure the synthetic data covers the real distribution (recall) without generating outliers (precision).
• Visual Inspection: t-SNE visualizations remain crucial for human-in-the-loop spotting of global clustering issues.

Consider generating synthetic chest X-rays to augment a dataset for detecting pneumonia.

1. Real Data: 10,000 labeled real X-rays (R_train=8,000, R_test=2,000).
2. Synthetic Data: Generator produces 20,000 synthetic X-rays (S_test).
3. TRTS Execution:
   • Train a DenseNet classifier on the 8,000 real images.
   • Baseline AUC on R_test: 0.96.
   • AUC on S_test: 0.92.
4. Interpretation: The TRTS performance gap is 0.04 (or 96% of real performance). This indicates high fidelity. However, a per-class analysis might reveal the model struggles more with synthetic "pneumonia" images, prompting a review of the conditional generation for that class. This finding would be correlated with TSTR results and FID scores for a full assessment.

The inverse protocol to TRTS. TSTR measures the utility of synthetic data by training a model entirely on the generated dataset and evaluating its performance on a held-out set of real data. A high-performing model indicates the synthetic data has captured the essential patterns needed for the task.

• Primary Use: Assessing the training utility of synthetic data.
• Key Insight: If a model trained on synthetic data generalizes well to real data, the synthetic data is useful for model development.

A standard metric for evaluating the quality and diversity of synthetic images. FID calculates the Wasserstein-2 distance between the feature distributions of real and generated images, as extracted by a pre-trained Inception-v3 network. Lower scores indicate better fidelity.

• Mechanism: Compares multivariate Gaussians fitted to the feature vectors.
• Limitation: Requires a large sample size and is sensitive to image count.

A two-dimensional metric that separately quantifies the quality (precision) and coverage/diversity (recall) of a generative model's output. It addresses the limitation of single-score metrics like FID.

• Precision: The fraction of synthetic samples that lie within the support of the real data manifold (are realistic).
• Recall: The fraction of real data manifold covered by the synthetic samples.

A discriminative evaluation method. A classifier (e.g., a neural network) is trained to distinguish between samples from the real and synthetic datasets. The failure of this classifier—achieving near-random (50%) accuracy—is used as a proxy for high synthetic data fidelity.

• Interpretation: If a powerful classifier cannot tell real from fake, the distributions are well-aligned.
• Advantage: Can be applied to any data modality (tabular, text, images).

A kernel-based statistical test used to determine if two samples (real and synthetic) are drawn from the same distribution. MMD computes the distance between the mean embeddings of the distributions in a Reproducing Kernel Hilbert Space (RKHS).

• Application: A core component of metrics like Kernel Inception Distance (KID).
• Property: It is zero if and only if the distributions are identical.

An automated system that applies a suite of statistical, fidelity, utility, and privacy metrics to assess generated datasets. A robust pipeline typically sequences tests:

1. Statistical Tests: MMD, KL-divergence on marginal distributions.
2. Fidelity Tests: TRTS, Domain Classifier, FID.
3. Utility Tests: TSTR, Downstream Task Performance.
4. Privacy Tests: Differential Privacy audit, Membership Inference Attack resistance.