Propensity Score Matching

PSM evaluates synthetic data quality by comparing the distribution of propensity scores between real and generated datasets. A well-synthesized dataset should produce propensity scores that are indistinguishable from the original, indicating that the statistical relationships between covariates have been faithfully preserved. This method is particularly valuable for healthcare data where maintaining clinical correlations is critical.

• Measures similarity of conditional distributions across all variables
• Detects systematic biases introduced during generation
• Provides a single scalar metric for overall distributional similarity

The propensity score is defined as the conditional probability of a record belonging to the synthetic dataset given its observed covariates: e(X) = P(T=1|X). A logistic regression or other binary classifier is trained to distinguish real from synthetic records. If the synthetic data is high-quality, the classifier should perform no better than random chance (AUC ≈ 0.5), indicating the distributions are statistically indistinguishable.

• Models the assignment mechanism between real and synthetic groups
• Reduces multidimensional comparison to a single balancing score
• AUC values near 0.5 indicate strong distributional overlap

After propensity score estimation, the Standardized Mean Difference quantifies the balance achieved for each covariate. SMD is calculated as the difference in means between real and synthetic groups divided by the pooled standard deviation. An SMD less than 0.1 is generally considered negligible imbalance, confirming that the synthetic data adequately replicates the original variable distributions.

• Threshold of |SMD| < 0.1 indicates acceptable balance
• Complements propensity score distributions with per-variable diagnostics
• Identifies which specific features may require generation improvement

PSM serves as a critical diagnostic at the intersection of data utility and privacy preservation. A synthetic dataset that perfectly matches propensity score distributions may risk overfitting to the original data, potentially exposing individual records. Conversely, excessive privacy noise degrades covariate balance. PSM helps identify the optimal generation threshold where statistical fidelity is maintained without compromising patient confidentiality.

• Detects overfitting when propensity scores are too perfectly matched
• Guides differential privacy budget allocation in generative models
• Balances clinical plausibility against re-identification risk

Beyond aggregate metrics, PSM enables stratified analysis where records are grouped into propensity score quintiles. Within each stratum, the distribution of covariates should be balanced between real and synthetic data. This granular approach reveals whether generation quality degrades in specific regions of the feature space, such as rare disease phenotypes or extreme laboratory values that are challenging to synthesize accurately.

• Quintile stratification reveals localized generation failures
• Identifies tail distribution representation quality
• Supports targeted refinement of generative model training

PSM is often combined with the Train-Synthetic-Test-Real (TSTR) paradigm for comprehensive validation. While TSTR measures whether models trained on synthetic data generalize to real data, PSM confirms that the underlying data structure is preserved. Together, they provide both predictive utility and distributional fidelity assessments, forming a complete evaluation framework required for regulatory submissions in healthcare AI applications.

• PSM validates distributional similarity
• TSTR validates downstream task performance
• Combined framework supports FDA submission readiness

The primary diagnostic for successful matching. Covariate balance measures the similarity of the distribution of confounding variables between the treatment and control groups after matching.

• Standardized Mean Difference (SMD): A common metric; a value below 0.1 typically indicates negligible imbalance.
• Love Plot: A visual tool for assessing balance across all covariates before and after matching.
• Variance Ratio: Checks if the spread of a covariate is similar between groups, not just the mean.

The most common estimand in PSM. ATT quantifies the effect of an intervention specifically for the units that actually received it.

• Formula: E[Y(1) - Y(0) | T=1]
• Interpretation: The average difference in outcomes for treated subjects compared to what would have happened if they had not been treated.
• Contrast with ATE: The Average Treatment Effect measures the effect if the entire population were treated, which is often not the policy-relevant question.

An alternative to matching that uses the propensity score to create a pseudo-population where treatment assignment is independent of covariates.

• Mechanism: Subjects are weighted by the inverse of their probability of receiving the treatment they actually received.
• Stabilized Weights: A variant that reduces variance in extreme weights by multiplying by the marginal probability of treatment.
• Advantage: Retains all subjects in the analysis, unlike matching which may discard unmatched units.

A method that divides subjects into blocks based on quantiles of the propensity score distribution. Treatment effects are estimated within each stratum and then aggregated.

• Quintiles: A common approach is to create 5 strata, which has been shown to remove approximately 90% of bias from each confounder.
• Within-Stratum Balance: Covariate balance must be checked within each stratum, not just overall.
• Limitation: Residual confounding can persist within strata if the score is not finely stratified.

Advanced generative models that incorporate structural causal models (SCMs) and do-calculus to generate synthetic data that preserves causal mechanisms, not just statistical correlations.

• Counterfactual Generation: Can simulate 'what if' scenarios by intervening on variables in the learned causal graph.
• Utility for PSM: These models can generate synthetic control arms for clinical trials, allowing direct validation of propensity score methods against known ground-truth effects.
• Debiasing: Explicitly models confounding structure to prevent the synthetic generator from learning spurious associations.

A technique for evaluating synthetic data fidelity by training a classifier to distinguish between real and generated samples. It directly tests the core assumption of PSM utility.

• ROC AUC Metric: An AUC near 0.5 indicates the synthetic data is indistinguishable from real data; a higher score reveals distributional gaps.
• Feature Importance: Analyzing which features drive the discriminator's success pinpoints where the synthetic generator fails to capture the true covariate distribution.
• Iterative Refinement: Used in a feedback loop with GANs to improve the realism of generated control groups.