Train-Synthetic-Test-Real (TSTR)

TSTR establishes a strict separation between training and testing domains. A model is trained exclusively on synthetic data** and then evaluated against a held-out real dataset that was never seen during generation or training.

• Training Phase: Model learns patterns, distributions, and relationships from synthetic samples only
• Testing Phase: Performance is measured on real-world data to assess generalization
• Key Metric: The closer TSTR performance is to TRTR (Train-Real-Test-Real), the higher the utility of the synthetic data

The definitive utility score is the ratio of TSTR performance to TRTR performance. A ratio approaching 1.0 indicates the synthetic data captures the predictive signal of the original data.

• TRTR Baseline: Model trained and tested on real data — the theoretical upper bound
• TSTR Score: Model trained on synthetic, tested on real — the practical utility measure
• Interpretation: A TSTR/TRTR ratio of 0.95 means synthetic data preserves 95% of real data's predictive power
• Statistical Significance: Multiple runs with different synthetic datasets are required to establish confidence intervals

TSTR provides an empirical framework for measuring what is lost when privacy protections are applied. As differential privacy budgets (ε) decrease, TSTR performance typically degrades — TSTR quantifies exactly how much.

• ε vs. Utility Curve: Plot TSTR scores across decreasing privacy budgets to find the optimal operating point
• Regulatory Justification: Provides auditable evidence that synthetic data meets both utility requirements and privacy guarantees
• Federated Context: In federated GAN deployments, TSTR validates that decentralized generation preserves cross-institutional signal without exposing patient records

TSTR extends beyond simple accuracy metrics to domain-specific evaluation criteria critical in healthcare and regulated industries.

• Clinical TSTR: Evaluate synthetic EHR data by training a mortality prediction model on synthetic records and testing on real patient outcomes
• Radiological TSTR: Train a segmentation model on synthetic medical images and measure Dice coefficient on real scans
• Time-Series TSTR: Assess synthetic ICU waveforms by training an early warning system on artificial vitals and testing on real patient deterioration events
• Fairness TSTR: Measure whether models trained on synthetic data exhibit the same demographic parity as models trained on real data

When TSTR performance is poor, the failure mode provides diagnostic information about the synthetic data generation process.

• Mode Collapse: Synthetic data lacks diversity; model overfits to a narrow distribution and fails on real edge cases
• Distributional Shift: Synthetic data captures marginal distributions but fails to preserve joint distributions and feature interactions
• Temporal Leakage: Synthetic data inadvertently encodes future information, inflating TSTR scores artificially — requires careful temporal splitting
• Out-of-Distribution Detection: Use TSTR confidence scores to identify real samples that fall outside the synthetic training distribution

In federated data augmentation workflows, TSTR serves as the acceptance gate before synthetic data is shared across nodes or used for local training.

• Node-Level Validation: Each participating institution runs TSTR locally to verify that federated synthetic generation preserves their site-specific statistical properties
• Global Utility Aggregation: TSTR scores from all nodes are aggregated to produce a federated utility metric without centralizing test data
• Continuous Monitoring: TSTR is re-run periodically to detect concept drift — if synthetic data utility degrades, it signals that the underlying real data distribution has shifted and regeneration is required

A quantitative measure of how well a synthetic dataset preserves the statistical relationships and predictive performance of the original real-world data. TSTR is the primary metric for assessing this utility.

• High Utility: A model trained on synthetic data performs comparably to one trained on real data.
• Low Utility: Synthetic data fails to capture critical feature-label relationships, leading to poor real-world test performance.
• Measurement: Often expressed as the ratio of TSTR accuracy to Train-Real-Test-Real (TRTR) accuracy.

A dual-network architecture where a generator creates synthetic data and a discriminator evaluates its authenticity. The adversarial process iteratively improves realism.

• TSTR Application: GANs are a common source of synthetic training data evaluated by the TSTR paradigm.
• Key Variants: CTGAN for tabular data, medGAN for discrete EHR records, and Federated GANs for decentralized generation.
• Failure Mode: Mode collapse, where the generator produces limited varieties of samples, is directly exposed by poor TSTR performance.

A generative model that learns a compressed latent representation of input data and reconstructs new samples by sampling from that learned distribution.

• Mechanism: Encoder compresses data to a latent vector; decoder reconstructs from sampled points in latent space.
• TSTR Relevance: VAEs produce smoother, less sharp synthetic data than GANs. TSTR evaluation reveals whether the probabilistic reconstruction preserves class boundaries.
• Advantage: Generally more stable to train than GANs, making them reliable baselines for TSTR benchmarks.

A change in the statistical properties of data that degrades model performance. TSTR is a direct test of whether synthetic data introduces a harmful distributional gap from real data.

• Covariate Shift: Input feature distributions differ between synthetic training and real test sets.
• Concept Drift: The relationship between features and labels changes, invalidating learned patterns.
• Detection: A significant drop from TRTR to TSTR performance signals a distributional shift introduced by the generative process.

An adversarial technique that determines whether a specific patient's record was used in training a model. TSTR provides a privacy-utility trade-off benchmark.

• Risk: Overfitted generative models may memorize and reproduce training samples.
• Mitigation: Differential privacy guarantees can be layered onto GANs and VAEs.
• TSTR's Role: A model that performs well on TSTR while the generator resists membership inference demonstrates a successful privacy-utility balance.

A decentralized generative adversarial network where the discriminator and generator are trained across multiple institutions without sharing underlying patient data.

• Architecture: Local discriminators train on-site; a global generator synthesizes data from aggregated gradient updates.
• TSTR Validation: The global generator's output is tested via TSTR to ensure it captures cross-institutional data distributions.
• Challenge: Non-IID data across silos can cause the global generator to produce unrealistic or biased samples.