Synthetic Data Quality Report

Evaluates univariate distributional similarity between real and synthetic data for each column.

• Kolmogorov-Smirnov (KS) Complement: Measures the maximum distance between cumulative distribution functions. A score near 1.0 indicates high shape fidelity.
• Total Variation Distance (TVD): Quantifies the average probability mass difference across discrete categories.
• Continuous vs. Discrete: Applies appropriate metrics based on data type—KS for numerical columns, TVD for categorical.

Example: A KS complement of 0.92 for an income column means the synthetic distribution overlaps 92% with the real distribution.

Measures how well the synthetic data preserves multivariate relationships between column pairs.

• Correlation Similarity: Compares the correlation matrix of synthetic data against the real data matrix using metrics like Pearson, Spearman, or mutual information.
• Contingency Table Similarity: For categorical pairs, evaluates the normalized similarity of joint frequency tables.
• Detection Risk: Low pair trend fidelity makes synthetic data vulnerable to attribute inference attacks where adversaries exploit broken correlations.

A report typically flags column pairs with similarity scores below 0.70 for investigation.

Validates that synthetic records respect the domain constraints and logical boundaries of the real data.

• Min/Max Violations: Detects synthetic values falling outside the observed real data range (e.g., negative ages).
• Category Adherence: Ensures synthetic categorical columns contain only valid classes present in the real data.
• Referential Integrity: In multi-table synthesis, verifies that foreign keys resolve to existing primary keys.

A boundary adherence score of 1.0 indicates zero constraint violations across all columns.

Quantifies the re-identification risk and membership disclosure exposure of the synthetic dataset.

• Nearest Neighbor Distance Ratio (NNDR): Measures how closely synthetic records mimic individual real records. A high NNDR indicates strong privacy.
• Identical Match Rate: The percentage of synthetic records that are exact copies of real records—should be 0%.
• Membership Inference AUC: Evaluates how easily an attacker can determine if a real record was in the training set.

A privacy-safe report typically requires an NNDR above 0.85 and zero exact matches.

Employs the Train-Synthetic-Test-Real (TSTR) paradigm to measure downstream task utility.

• A machine learning model is trained exclusively on synthetic data and evaluated on a held-out real test set.
• Utility Score: The ratio of synthetic-trained model performance to real-trained model performance (e.g., F1-score or RMSE).
• Task-Specific: Utility is measured for the intended use case—classification, regression, or clustering.

Example: If a real-trained classifier achieves 0.95 AUC and a synthetic-trained classifier achieves 0.90 AUC, the utility score is 0.947.

Leverages the SDMetrics library to generate a comprehensive, reproducible quality report in a standardized format.

• Quality Report Object: Programmatically generates a multi-metric diagnostic with column-level and table-level scores.
• Visualization Suite: Includes distribution plots, correlation heatmaps, and privacy risk histograms.
• Threshold Flagging: Automatically highlights metrics falling below configurable quality thresholds (e.g., KS complement < 0.80).

SDMetrics integrates directly with the Synthetic Data Vault (SDV) ecosystem for end-to-end evaluation pipelines.

The cornerstone of any quality report, measuring how well the synthetic data replicates the original. It analyzes:

• Univariate distributions: Column shapes, means, and standard deviations.
• Bivariate correlations: Pairwise trends and relationships between columns.
• Boundary adherence: Ensuring synthetic values respect the min/max ranges of real data. High fidelity means the synthetic data is a statistically indistinguishable mirror of the source.

A critical privacy metric that quantifies the probability an adversary can link a synthetic record back to a real individual. The report evaluates this by:

• Measuring the distance between synthetic records and their nearest real neighbors.
• Calculating the proportion of synthetic records that are exact copies of real data (a failure condition).
• Assessing k-anonymity thresholds within the synthetic dataset to ensure each record is sufficiently hidden in a crowd.

A diagnostic test used in a quality report to gauge privacy leakage. An attacker model is trained to distinguish real records from synthetic ones. The report then measures:

• Attack AUC: The area under the curve of the attacker's classifier. A score near 0.5 indicates the synthetic data doesn't leak membership.
• Precision/Recall: How accurately the attacker can pinpoint which specific records were in the original training set. This provides a direct, empirical measure of privacy risk.

A critical fidelity failure that a quality report must explicitly check for. Mode collapse occurs when the generator fails to capture the full diversity of the real data, producing only a few 'safe' examples. The report detects this by:

• Coverage analysis: Measuring the proportion of real data modes represented in the synthetic data.
• Diversity scores: Quantifying the variety of synthetic samples compared to the real distribution. A high-fidelity report confirms that the synthetic data captures the long tail, not just the common patterns.