Train-Synthetic-Test-Real (TSTR)

TSTR follows a strict three-phase protocol:

• Phase 1 — Train: A machine learning model is trained exclusively on a synthetic dataset generated by a model like a GAN or VAE. No real data is seen during training.
• Phase 2 — Test: The trained model is evaluated on a held-out set of real data that was never used to train the generator.
• Phase 3 — Compare: The TSTR performance is compared against a baseline Train-Real-Test-Real (TRTR) score to compute a utility gap.

A small gap indicates high synthetic data utility; a large gap signals that the synthetic data failed to capture predictive patterns present in the real distribution.

TSTR measures utility, not just statistical similarity. This distinction is critical:

• Statistical fidelity metrics (e.g., Jensen-Shannon divergence, Wasserstein distance) compare column distributions and pairwise correlations. They can be gamed by generators that memorize marginal statistics.
• TSTR evaluates whether synthetic data preserves the conditional relationships that matter for prediction. A dataset can pass statistical tests yet fail TSTR if the generator drops subtle feature interactions.
• TSTR is task-specific: the same synthetic dataset may score well for classification but poorly for regression, revealing utility is not a monolithic property.

Every TSTR evaluation requires a TRTR (Train-Real-Test-Real) reference point:

• TRTR trains and tests on real data splits, establishing the upper bound of achievable performance given the model architecture and task.
• The Utility Gap = TRTR Score − TSTR Score. This gap quantifies the degradation caused by substituting real data with synthetic data.
• A gap approaching zero indicates the synthetic data is a drop-in replacement for real data in that specific downstream task.
• In regulated industries like healthcare, a utility gap below a predefined threshold (e.g., < 5% AUC drop) is often required before synthetic data can be released for external research.

TSTR is the primary tool for navigating the privacy-utility Pareto frontier:

• Differential Privacy (DP) guarantees are strengthened by increasing noise in the generator, which degrades synthetic data quality.
• TSTR provides a quantitative utility readout at each privacy level (ε), enabling data custodians to select an operating point where utility remains acceptable.
• Membership inference attack resistance can be plotted against TSTR scores to demonstrate that strong privacy does not necessarily destroy predictive value.
• This framework is essential for clinical data access committees deciding whether synthetic patient records are fit for purpose.

TSTR is powerful but has known blind spots:

• Mode collapse: If a GAN generates only a subset of the real distribution, TSTR may still score well if the test set happens to fall within that subset. Complement TSTR with coverage metrics like recall.
• Task selection bias: TSTR results are only valid for the specific model and task tested. Strong TSTR performance on logistic regression does not guarantee utility for a deep neural network or a different prediction target.
• Out-of-distribution detection: TSTR does not measure whether synthetic data introduces novel artifacts that a model might confidently misclassify. Pair TSTR with adversarial validation to detect distribution shift.
• Computational cost: Running full TSTR evaluations for every generator checkpoint is expensive, motivating the use of cheaper proxy metrics during training.

TSTR is becoming a standard evidence component for regulatory and compliance submissions:

• FDA submissions for AI/ML-enabled medical devices increasingly expect TSTR results when synthetic data is used for pre-market validation.
• HIPAA Safe Harbor compliance can be demonstrated by showing that TSTR-trained models do not memorize individual records, complementing formal privacy metrics like k-anonymity.
• Model cards for synthetic data generators now routinely include TSTR scores across multiple downstream tasks, providing transparency for downstream consumers.
• In federated learning scenarios, TSTR validates that synthetic data generated from aggregated teacher models preserves the predictive signal of the decentralized real data.

A composite metric evaluating synthetic data across three critical dimensions: statistical fidelity to the original data, utility for downstream machine learning tasks, and privacy protection against re-identification. High-quality synthetic data must balance all three, as optimizing solely for fidelity can compromise privacy. The score often aggregates sub-metrics like the propensity score divergence and nearest neighbor distance ratios.

A privacy metric that measures the difficulty of distinguishing real from synthetic records. It calculates the ratio of cases where a synthetic record's nearest neighbor in a combined real-synthetic dataset is another synthetic record. An NNAA score near 0.5 indicates that an adversary can do no better than random guessing, signifying strong privacy. Scores deviating significantly from 0.5 suggest identifiability risk.

A technique for detecting distribution shift between training and test sets by training a classifier to distinguish them. In the TSTR context, it is used to evaluate whether synthetic training data faithfully represents the real test data distribution. If a classifier can easily tell them apart (high ROC-AUC), the synthetic data has failed to capture the true data manifold, indicating poor utility for model training.

A metric for evaluating synthetic data quality by comparing the distribution of features extracted from a pre-trained model. It measures the Wasserstein-2 distance between multivariate Gaussians fitted to real and generated feature embeddings. Lower FID scores indicate higher visual fidelity and diversity. While originating in computer vision, the concept is adapted for tabular and time-series data using domain-specific feature extractors.

The degree to which synthetic medical data adheres to established physiological constraints, medical ontologies like SNOMED CT, and realistic disease progression pathways. In a TSTR evaluation for healthcare, a model trained on synthetic data must not only achieve high accuracy but also produce predictions that are clinically coherent and free of nonsensical correlations that violate established medical knowledge.