Glossary

Train-Synthetic-Test-Real (TSTR)

Train-Synthetic-Test-Real (TSTR) is an evaluation paradigm where a machine learning model is trained exclusively on synthetic data and tested on a held-out set of real data to quantify the synthetic data's utility for downstream predictive tasks.

Get in touch Learn more

Product manager reviewing autonomous task execution dashboard on laptop, completed tasks visible, casual work session.

SYNTHETIC DATA EVALUATION

What is Train-Synthetic-Test-Real (TSTR)?

A rigorous evaluation paradigm for measuring the utility of synthetic data by training a model entirely on generated samples and testing its performance on a held-out set of real data.

Train-Synthetic-Test-Real (TSTR) is an evaluation paradigm where a machine learning model is trained exclusively on artificially generated data and subsequently tested on a held-out set of real, authentic data. The core premise is that if the synthetic data captures the true underlying distribution, a model trained on it should generalize effectively to real-world samples. This metric directly quantifies the downstream utility of synthetic data for predictive tasks, bypassing purely statistical similarity measures.

TSTR is often compared against Train-Real-Test-Real (TRTR), which establishes an upper performance bound using real training data. The gap between TSTR and TRTR performance reveals the utility loss caused by substituting real data with synthetic proxies. A narrow gap indicates high-fidelity synthesis, while a wide gap signals that the generative model failed to preserve the discriminative patterns necessary for the target task, often due to mode collapse or privacy-preserving noise injection.

TSTR EVALUATION

Frequently Asked Questions

Clarifying the Train-Synthetic-Test-Real paradigm for assessing synthetic data utility in downstream machine learning tasks.

Train-Synthetic-Test-Real (TSTR) is an evaluation paradigm that measures the utility of synthetic data by training a machine learning model exclusively on artificially generated records and then evaluating its performance on a held-out set of real data. The core premise is that if the synthetic data faithfully captures the underlying statistical structure of the original dataset, a model trained on it should generalize effectively to real-world examples. The process involves three steps: first, a generative model (such as a CTGAN or Variational Autoencoder) produces a synthetic dataset; second, a downstream predictive model is trained solely on this synthetic data; third, the model's accuracy, F1-score, or RMSE is measured against a real test set. The resulting performance metric serves as a direct proxy for the synthetic data's utility. A TSTR score close to the Train-Real-Test-Real (TRTR) baseline indicates high-fidelity synthesis.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

EVALUATION & UTILITY

Related Terms

Core concepts for assessing the quality and privacy of synthetic data used in the Train-Synthetic-Test-Real paradigm.

Statistical Fidelity

The degree to which a synthetic dataset preserves the univariate distributions, multivariate correlations, and aggregate statistics of the original real data. High fidelity means the synthetic data is a realistic proxy for the real data.

Measured by comparing column shapes and pair trends
Essential for TSTR to produce a model that generalizes to real data
Low fidelity leads to a model learning non-existent patterns

Column Shapes

KSComplement Score

Pair Trends

Correlation Similarity

Privacy-Utility Trade-off

The fundamental balancing act between the strength of a privacy guarantee and the statistical utility of the resulting synthetic data. In TSTR, a model trained on data with overly aggressive privacy protections will perform poorly on real test data.

Tight privacy budgets introduce noise that degrades signal
The optimal point is use-case specific
TSTR is the direct empirical measure of utility in this trade-off

Membership Inference Attack

An adversarial technique that determines whether a specific data point was included in the training set of a generative model. This is a critical privacy evaluation for synthetic data used in TSTR.

A successful attack indicates the synthetic data has memorized real records
Defenses include differential privacy during training
TSTR models trained on memorized data offer a false sense of utility

SDMetrics

An open-source Python library for evaluating synthetic data quality by computing statistical, privacy, and efficacy metrics. It provides the standardized framework for quantifying if a TSTR workflow is valid.

Statistical Metrics: Column shapes, pair trends
Privacy Metrics: Nearest neighbor distance ratio, membership inference risk
Efficacy Metrics: TSTR performance compared to TRTR (Train-Real-Test-Real)

EXPLORE

Mode Collapse

A failure condition in GAN training where the generator learns to produce only a limited variety of outputs, failing to capture the full diversity of the real data distribution. This is catastrophic for TSTR.

A model trained on collapsed data will fail on rare real-world edge cases
Detected by low coverage metrics in quality reports
Often addressed with Wasserstein loss or conditional generation

Synthcity

A comprehensive open-source library providing a unified interface for training, evaluating, and benchmarking a wide range of synthetic data generation models. It is a key tool for systematically comparing TSTR performance across different generators.

Supports GANs, VAEs, and diffusion models
Includes built-in TSTR evaluation benchmarks
Enables apples-to-apples comparisons of utility

EXPLORE

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.