Synthetic Data Quality Score

Measures how closely the synthetic data preserves the joint probability distribution of the original dataset. High fidelity ensures that column distributions, pairwise correlations, and multivariate interactions are indistinguishable from real data.

• Column Shapes: Kolmogorov-Smirnov test for continuous variables; Chi-squared test for categoricals.
• Correlation Preservation: Pearson and Spearman coefficients must remain within acceptable bounds.
• Multivariate Structure: Captured via metrics like Frechet Inception Distance (FID) or propensity score divergence.

Low fidelity introduces distributional shift, causing models trained on synthetic data to fail when deployed on real-world inputs.

Quantifies whether synthetic data can replace real data for a specific machine learning task. Evaluated using the Train-Synthetic-Test-Real (TSTR) paradigm, where a model is trained exclusively on synthetic data and evaluated on a held-out real dataset.

• Classification/Regression: Compare TSTR performance to Train-Real-Test-Real (TRTR) baseline.
• Feature Importance: SHAP or permutation importance rankings should be consistent between real and synthetic training sets.
• Decision Boundary Similarity: The model's learned function should generalize identically.

A high utility score indicates the synthetic data captures the signal necessary for prediction, not just the noise.

Assesses the risk of re-identification or attribute inference from the synthetic dataset. A high privacy score means an adversary cannot determine if a specific individual was in the training data or infer sensitive attributes about them.

• Membership Inference Attack (MIA): AUC of an attacker classifier distinguishing real training records from hold-out records using only the synthetic data.
• Nearest Neighbor Adversarial Accuracy (NNAA): Measures the ratio of synthetic records whose closest neighbor in the real data is a training record vs. a hold-out record.
• Distance to Closest Record (DCR): The Euclidean distance between a synthetic record and its nearest real counterpart; larger distances imply lower identifiability.

Formal guarantees like (ε, δ)-Differential Privacy can be integrated into the generator's training loop.

A fundamental tension exists: increasing privacy protection inevitably degrades utility and fidelity. Differential Privacy adds calibrated noise to gradients during training, which obscures individual contributions but can destroy rare categories or subtle correlations.

• ε (Epsilon): The privacy budget. Lower ε means stronger privacy but worse data quality.
• Catastrophic Collapse: At very low ε, the generator may fail to learn minority classes entirely.
• Pareto Frontier Analysis: Practitioners must plot utility vs. privacy to select an optimal operating point for their specific regulatory and business requirements.

There is no universal best score; the acceptable trade-off is context-dependent.

Measures how closely the synthetic data preserves the joint probability distribution of the original dataset.

• Marginal Distributions: Compares univariate histograms and summary statistics (mean, variance, skewness) between real and synthetic data.
• Bivariate Correlations: Evaluates pairwise relationships using Pearson, Spearman, or mutual information matrices.
• Multivariate Structure: Assesses higher-order interactions through dimensionality reduction techniques like PCA or t-SNE, comparing the overlap of real and synthetic point clouds.
• Domain-Specific Constraints: Verifies adherence to logical rules (e.g., age_at_diagnosis < age_at_death) and medical ontologies like SNOMED CT.

Quantifies the synthetic data's fitness for its intended analytical purpose using the Train-Synthetic-Test-Real (TSTR) paradigm.

• Classification/Regression Performance: A model trained on synthetic data is evaluated on a held-out real test set. The performance gap relative to a model trained on real data defines the utility score.
• Feature Importance Concordance: Compares the ranking of predictive features derived from synthetic-trained models against those from real-trained models using metrics like Kendall's tau.
• Clustering Preservation: Evaluates whether patient subgroups discovered in synthetic data match those in real data using Adjusted Rand Index (ARI) or Normalized Mutual Information (NMI).
• Statistical Test Replication: Verifies that hypothesis tests (e.g., differential gene expression) yield consistent p-values and effect sizes when run on synthetic versus real data.

Assesses the risk of re-identification and attribute disclosure using adversarial auditing techniques.

• Nearest Neighbor Adversarial Accuracy (NNAA): Measures how well an attacker can distinguish real records from synthetic ones by comparing distances to nearest neighbors. An NNAA score near 0.5 indicates strong privacy (no better than random guessing).
• Membership Inference Attack Resistance: Trains a shadow classifier to determine if a specific individual's record was in the original training set. The attack's AUC quantifies leakage risk.
• Distance to Closest Record (DCR): Computes the minimum Euclidean or Hamming distance between each synthetic record and the nearest real record. A large median DCR indicates low identifiability.
• K-Anonymity & L-Diversity: Verifies that synthetic records satisfy formal privacy models, ensuring each record is indistinguishable from at least k-1 others and sensitive attributes have sufficient diversity.

Standardized methodologies for aggregating fidelity, utility, and privacy metrics into a single interpretable score.

• Synthetic Data Vault (SDV) Quality Report: Generates a holistic score from 0-100% by combining column shapes, pair trends, and privacy metrics with configurable weights.
• Weighted Multi-Objective Optimization: Allows domain experts to assign priority weights to each dimension (e.g., privacy > utility for public release) to compute a task-specific quality index.
• Radar/Spider Chart Visualization: Plots the three dimensions on orthogonal axes to visually identify trade-offs, such as high fidelity but poor privacy, enabling rapid diagnostic assessment.
• Threshold-Based Certification: Defines minimum acceptable scores for each dimension (e.g., fidelity > 0.85, NNAA < 0.55) that synthetic data must meet before being approved for clinical research or regulatory submission.

A domain-specific quality dimension ensuring synthetic medical data respects established physiological and clinical constraints.

• Ontology Conformance: Checks that diagnosis codes map to valid SNOMED CT or ICD-10 hierarchies and that drug prescriptions align with formulary databases.
• Temporal Consistency: Validates that clinical event sequences follow realistic disease progression pathways (e.g., lab test precedes diagnosis, not the reverse).
• Physiological Bounds: Ensures vital signs and lab values fall within biologically possible ranges (e.g., systolic blood pressure between 40-300 mmHg) and respect known correlations (e.g., creatinine and eGFR).
• Contraindication Avoidance: Verifies that synthetic patients are not prescribed medications contraindicated with their documented allergies or comorbidities.

A technique for detecting whether synthetic data faithfully captures the real data distribution by training a discriminator to distinguish between them.

• ROC-AUC Metric: A classifier is trained to predict whether a sample is real or synthetic. An AUC near 0.5 indicates the distributions are indistinguishable; an AUC near 1.0 signals detectable shift.
• Feature-Level Drift Detection: Identifies which specific variables contribute most to the discriminator's success, pinpointing columns where the synthetic generator failed to model the true distribution.
• Iterative Refinement Loop: Adversarial validation results are fed back to tune the generative model (e.g., adjusting GAN architecture or VAE latent dimensions) until the discriminator AUC approaches 0.5.
• Cross-Validation Stability: Performs adversarial validation across multiple train/test splits to ensure distribution matching is robust and not an artifact of a particular sampling seed.