Downstream Task Performance

The task of assigning a single label from a predefined set to an input image. It is a foundational computer vision task where synthetic data fidelity is critical for model generalization.

Key Performance Metrics:

• Accuracy: The proportion of total predictions that are correct. Simple but can be misleading on imbalanced datasets.
• Top-k Accuracy: The proportion of times the correct label appears in the model's top k predicted probabilities. Common for large label spaces (e.g., ImageNet).
• Precision, Recall, and F1-Score: Provide a more nuanced view, especially for per-class performance. Precision measures correctness when the model predicts a class. Recall measures the model's ability to find all instances of a class.
• Confusion Matrix: A table showing correct predictions and error types, essential for diagnosing systematic failures introduced by synthetic data artifacts.

Tasks that involve localizing and identifying objects within an image. Object Detection draws bounding boxes, while Semantic Segmentation classifies each pixel. These require synthetic data to preserve precise spatial relationships and object geometries.

Key Performance Metrics:

• For Detection (mAP): Mean Average Precision is the standard. It calculates the average precision (area under the precision-recall curve) for each object class and then averages across classes. IoU (Intersection over Union) threshold (e.g., 0.5) defines a "correct" detection.
• For Segmentation:
  • mIoU (Mean Intersection over Union): The standard metric, averaging the ratio of intersection to union for each class between predicted and ground truth pixel masks.
  • Dice Coefficient (F1-Score): Measures the overlap between predictions and ground truth, commonly used in medical imaging.
• Per-Class Performance: Critical for safety; synthetic data must not degrade performance for rare but important classes (e.g., "pedestrian").

The task of categorizing text documents into predefined classes (e.g., topic, spam/not spam) or estimating sentiment polarity (positive, negative, neutral). Synthetic text must preserve semantic meaning and stylistic nuance.

Key Performance Metrics:

• Accuracy: Useful for balanced datasets.
• F1-Score (Macro/Micro): The harmonic mean of precision and recall. Macro-F1 computes the metric independently for each class and then averages, treating all classes equally. Micro-F1 aggregates contributions of all classes to compute the average, favoring larger classes.
• AUC-ROC (Area Under the ROC Curve): Measures the model's ability to distinguish between classes across all classification thresholds, robust to class imbalance.
• Confusion Matrix Analysis: Reveals if synthetic data causes bias, such as a model consistently misclassifying nuanced negative sentiment as neutral.

A sequence labeling task that identifies and classifies key entities (e.g., persons, organizations, locations) in text into predefined categories. Synthetic data must generate coherent entities with correct contextual relationships.

Key Performance Metrics:

• Span-Based F1-Score: The standard metric. A prediction is correct only if both the entity boundary (start and end token) and the entity type match the ground truth.
• Precision & Recall: Reported at the token or entity level. Low recall may indicate synthetic data lacks entity diversity; low precision may indicate poor contextual grounding.
• Per-Entity Performance: Breakdown of F1 for each entity type (e.g., PERSON, DATE) is essential to ensure synthetic data fidelity for all required categories.

The task of automatically translating text from a source language to a target language. Synthetic parallel corpora must preserve semantic equivalence and grammatical fluency.

Key Performance Metrics:

• BLEU (Bilingual Evaluation Understudy): The most common metric, based on the precision of n-gram matches between the model's output and human reference translations. It correlates well with human judgment at the corpus level.
• METEOR: Considers synonymy and stemming via WordNet alignment, often correlating better with human judgment at the sentence level than BLEU.
• TER (Translation Edit Rate): The number of edits (insertions, deletions, substitutions, shifts) required to change the output into the reference, normalized by reference length. Measures post-editing effort.
• Human Evaluation: Ultimately, synthetic data quality is judged by human-rated Adequacy (meaning preservation) and Fluency (grammaticality).

The task of predicting future values based on previously observed values over time. Synthetic time-series data must preserve temporal dynamics, seasonality, and noise characteristics.

Key Performance Metrics:

• MAE (Mean Absolute Error): The average absolute difference between predictions and actuals. Easy to interpret and robust to outliers.
• RMSE (Root Mean Square Error): The square root of the average squared differences. Penalizes larger errors more heavily than MAE.
• MAPE (Mean Absolute Percentage Error): The average absolute percentage error. Useful for understanding error relative to scale but problematic near zero values.
• sMAPE (Symmetric MAPE): A variant of MAPE that is symmetric and bounded.
• Critical: Metrics should be evaluated over multiple forecast horizons (e.g., next step, 10 steps ahead) to test if synthetic data preserves long-term dependencies.

The synthetic-to-real gap is the measurable performance degradation observed when a model trained exclusively on synthetic data is evaluated on real-world data. This gap quantifies the failure of synthetic data to fully capture the complexity and nuance of the target domain.

• Primary Cause: Imperfections in the generative model, leading to a distributional shift between synthetic and real data.
• Measurement: Directly observed as a drop in accuracy, F1 score, or other task-specific metrics when moving from a synthetic validation set to a real-world test set.
• Mitigation: A core goal of high-fidelity synthetic data generation is to minimize this gap, making downstream task performance on real data the definitive benchmark.

Distributional shift refers to any change in the joint probability distribution of input features and target labels between the data a model was trained on and the data it encounters during deployment. It is the fundamental statistical cause of degraded downstream performance.

• Types: Includes covariate shift (input distribution changes), concept drift (input-output relationship changes), and label shift (output distribution changes).
• Impact on Synthetic Data: If synthetic data does not perfectly match the real data distribution, a shift is introduced at training time, guaranteeing poor generalization.
• Detection: Methods like Domain Classifier Tests (Adversarial Validation) or two-sample tests (e.g., Kolmogorov-Smirnov) are used to quantify the shift between synthetic and real datasets.

Feature space alignment is the process of minimizing the discrepancy between the latent or learned representations of data from two domains—such as real and synthetic—within a model's embedding space. Good alignment is a prerequisite for strong downstream performance.

• Objective: To make the feature distributions of synthetic and real data statistically indistinguishable, so a model cannot tell which domain a sample came from.
• Techniques: Includes domain adaptation methods, gradient reversal layers, and loss functions based on Maximum Mean Discrepancy (MMD) or Wasserstein Distance.
• Proxy Metric: High alignment in a feature space (e.g., from a pre-trained model) often correlates with better downstream task performance, as the model learns more transferable representations.

Precision and Recall for Distributions is a framework that decomposes generative model evaluation into two separate metrics assessing the quality and coverage of synthetic data, providing finer-grained insight than a single fidelity score.

• Precision (Quality): Measures what fraction of the synthetic distribution is contained within the real data manifold. High precision means generated samples are highly realistic.
• Recall (Coverage): Measures what fraction of the real data distribution is covered by the synthetic manifold. High recall means the synthetic data captures the full diversity of the real data.
• Relation to Downstream Tasks: A synthetic dataset with high precision but low recall may train a model that is accurate on common cases but fails on rare but important edge cases, harming real-world performance.

Data plausibility is a per-sample assessment of whether a synthetic data point is realistic and could feasibly exist according to the rules and constraints of the target domain. It is a necessary but not sufficient condition for high downstream task performance.

• Assessment Methods:
  • Rule-based validation: Checking for physical or logical constraints (e.g., a person's age cannot be negative).
  • Anomaly detection: Using models trained on real data to flag synthetic samples that appear as outliers.
  • Expert review: Human-in-the-loop evaluation for complex domains like medical imagery.
• Failure Mode: Implausible data acts as noise during training, teaching the model incorrect feature relationships and directly harming generalization to real, plausible data.

The fidelity-privacy trade-off describes the inherent tension between creating synthetic data that is highly faithful to the original dataset and ensuring that data provides formal privacy guarantees for the individuals in the source data.

• Core Conflict: Techniques that increase fidelity (e.g., memorizing rare, detailed examples) often increase the risk of privacy leaks via membership inference attacks. Techniques that enhance privacy (e.g., differential privacy) typically add noise that reduces fidelity.
• Impact on Downstream Tasks: Excessively privatized data may have degraded statistical properties, leading to poorer model performance. The engineering challenge is to optimize synthetic data at the Pareto frontier of this trade-off for a given application's risk tolerance.