Train on Synthetic, Test on Real (TSTR)

The TSTR protocol is a direct, task-oriented evaluation method. Its primary purpose is to answer a critical question: Can a model trained on synthetic data solve real-world problems as effectively as one trained on real data?

• Procedure: A machine learning model (e.g., a classifier or regressor) is trained exclusively on the synthetic dataset. Its performance is then evaluated on a completely separate, unseen test set composed of real data.
• Objective: To quantify the downstream utility and statistical fidelity of the synthetic data. High TSTR performance indicates the synthetic data has preserved the predictive relationships necessary for the target task.

TSTR is almost always benchmarked against the Train on Real, Test on Real (TRTR) baseline, which represents the performance ceiling using the original data.

• TRTR Baseline: A model is trained on a subset of the real data and tested on the held-out real test set. This establishes the optimal achievable performance.
• Interpretation Gap: The performance difference between TRTR and TSTR scores reveals the utility gap. A small gap indicates high-fidelity synthesis.
• Critical Insight: A synthetic dataset achieving perfect statistical similarity scores might still fail the TSTR test if it hasn't captured subtle, task-relevant correlations, making TSTR a more pragmatic metric than distribution distance alone.

TSTR evaluation is essential in scenarios where synthetic data is intended for direct model training.

• Privacy-Preserving ML: Validating that synthetic data generated under differential privacy constraints retains utility for building accurate models.
• Data Augmentation for Imbalanced Classes: Assessing whether synthetic oversampling techniques (e.g., SMOTE, CTGAN) improve model performance on minority classes without degrading overall accuracy.
• Pre-training and Domain Adaptation: Evaluating if synthetic data can effectively pre-train a model before fine-tuning on limited real data, reducing data acquisition costs.
• Benchmarking Generative Models: Providing a concrete, application-driven metric to compare different tabular generative models (e.g., TVAE vs. TabDDPM).

The choice of metric in a TSTR evaluation is dictated by the downstream machine learning task.

• Classification Tasks: Use Accuracy, F1-Score, Precision-Recall AUC, or ROC-AUC.
• Regression Tasks: Use Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), or R-squared.
• Statistical Comparison: The TSTR metric is reported alongside the TRTR baseline. The relative percentage drop (e.g., "TSTR accuracy within 2% of TRTR") is a common summary.
• Multi-Metric Analysis: It's often necessary to evaluate performance across different data subgroups (e.g., per-class accuracy) to ensure the synthetic data hasn't introduced or amplified biases.

While crucial, TSTR has limitations that must be accounted for in a robust evaluation framework.

• Task Dependency: TSTR results are specific to the model architecture and task used for evaluation. A dataset that performs well for a logistic regression task may fail for a gradient boosting model.
• Does Not Guarantee Privacy: A successful TSTR outcome measures utility, not privacy. A synthetic dataset must be separately evaluated for privacy leakage using metrics like membership inference attack resilience.
• Held-Out Real Data Requirement: The protocol requires a reserved set of real data, which can be problematic when real data is extremely scarce.
• May Miss Multi-Task Utility: A single task evaluation might not capture all relational properties needed for other unforeseen downstream uses.

TSTR should be part of a comprehensive validation suite, not used in isolation.

• Train on Synthetic, Test on Synthetic (TSTS): A sanity check to ensure a model can learn from the synthetic data itself.
• Statistical Fidelity Tests: Measures like Wasserstein Distance, Kolmogorov-Smirnov tests, and correlation matrix similarity assess low-order moment matching.
• Discriminative Evaluation: Training a model to distinguish between real and synthetic samples (e.g., using a post-hoc classifier). A high classification accuracy indicates the synthetic data is easily distinguishable, suggesting poor fidelity.
• Privacy Audits: Conducting membership inference or attribute inference attacks on the synthetic data to quantify privacy risk.

A robust synthetic data pipeline will report TSTR alongside these other metrics to provide a complete picture of quality, utility, and safety.

Synthetic data utility quantifies how well a generated dataset preserves the statistical properties and machine learning task performance of the original, real data it is intended to replace or augment. It is the core objective measured by the TSTR protocol.

• Primary Metrics: Performance parity (e.g., accuracy, F1-score) of a downstream model trained on synthetic vs. real data.
• Statistical Fidelity: Preservation of marginal distributions, correlations, and higher-order joint distributions.
• Task-Agnostic vs. Task-Specific: Utility can be measured generally (e.g., using statistical distances) or for a specific predictive task (e.g., classification).

The Wasserstein Distance, or Earth Mover's Distance, is a metric from optimal transport theory used to compare two probability distributions. It is a fundamental tool for evaluating the statistical fidelity of synthetic tabular data.

• Intuition: Measures the minimum "cost" of transforming the synthetic data distribution into the real data distribution.
• Advantages: Overcomes limitations of metrics like Jensen-Shannon divergence by providing a meaningful distance even for distributions with non-overlapping support.
• Application in TSTR: A low Wasserstein distance between synthetic and real training set distributions is a strong indicator that TSTR performance will be high.

Differential Privacy (DP) is a rigorous mathematical framework that provides a strong, quantifiable privacy guarantee for data synthesis algorithms. When applied to tabular data generation, it ensures the inclusion or exclusion of any single individual's record has a negligible statistical effect on the synthetic output.

• Privacy-Utility Trade-off: A core challenge is balancing the ε (epsilon) privacy budget with the utility of the generated data, as measured by TSTR.
• Mechanisms: DP is typically enforced by injecting calibrated noise during the training of the generative model (e.g., in gradient updates) or into the learned parameters of a statistical model (e.g., a Bayesian network).
• Formal Guarantee: Provides resilience against membership inference and reconstruction attacks on the synthetic data.

PrivBayes is a canonical differentially private algorithm for synthesizing high-dimensional tabular data. It uses a Bayesian network to model the joint distribution of the dataset and injects calibrated Laplace noise into the learned conditional probability parameters before sampling synthetic records.

• Workflow: 1) Select a differentially private Bayesian network structure. 2) Learn noisy conditional distributions. 3) Sample synthetic data from the noisy model.
• Strength: Provides a provable DP guarantee while attempting to preserve complex attribute correlations.
• TSTR Context: Serves as a benchmark for evaluating the privacy-utility frontier; TSTR performance on PrivBayes-generated data demonstrates the cost of strong formal privacy.

Fairness-aware synthesis is the generation of tabular data with the explicit goal of mitigating biases present in the original dataset. It extends beyond statistical fidelity to enforce ethical constraints in the synthetic output.

• Objective: Generate data where protected attributes (e.g., race, gender) are statistically independent of other features or model outcomes, or where historical disparities are corrected.
• Techniques: Can involve pre-processing (debiasing the real data before synthesis), in-processing (adding fairness constraints to the generative model's loss), or post-processing (transforming the synthetic data).
• TSTR Role: The ultimate test of a fairness-aware generator is whether a model trained on its synthetic data (TSTR) makes fairer predictions on real-world holdout data than a model trained on the original, biased data.

Synthetic data validation is the comprehensive process of assessing the quality, fidelity, and privacy of a generated dataset. TSTR is a critical component of this process, focusing on downstream utility.

• Validation Pillars:
  • Fidelity: Do synthetic records look real? (e.g., visualization, summary statistics).
  • Utility: Can the data be used for its intended task? (TSTR is the gold standard here).
  • Privacy: Does the data leak information about real individuals? (e.g., membership inference tests).
  • Diversity: Does it cover the full distribution, including edge cases?
• Automation: In MLOps pipelines, this validation is often automated with predefined statistical tests and performance thresholds.