Propensity Score Matching

The propensity score is the predicted probability that a given record belongs to the synthetic dataset rather than the real dataset. A classifier—typically logistic regression or a gradient-boosted tree—is trained on a combined dataset where real records are labeled 0 and synthetic records are labeled 1. The model outputs a score between 0 and 1 for each record.

• Score near 0.5: The classifier cannot distinguish the record's origin, indicating high fidelity.
• Score near 0 or 1: The classifier easily identifies the record as real or synthetic, revealing distributional gaps.
• The distribution of propensity scores across all records forms the basis for the utility metric.

The pMSE is the primary summary statistic derived from propensity scores. It is calculated as the mean squared difference between each record's propensity score and the expected proportion of synthetic records in the combined dataset (typically 0.5 for balanced datasets).

• Formula: pMSE = (1/N) * Σ (p̂ᵢ - c)², where c is the proportion of synthetic records.
• Lower pMSE values indicate better utility, with 0 representing perfect indistinguishability.
• A null baseline pMSE can be calculated by repeatedly splitting the real data in half and computing the metric, establishing an expected value for identical distributions.
• The ratio of the observed pMSE to the null baseline provides a standardized utility score.

PSM evaluates whether the synthetic data covers the same regions of the feature space as the real data. A well-synthesized dataset should have propensity scores concentrated around 0.5, forming a unimodal distribution centered at the midpoint.

• Bimodal distributions with peaks near 0 and 1 indicate the generator failed to capture certain modes of the real data or introduced artifacts.
• Density plots of propensity scores for real and synthetic records should overlap substantially.
• This assessment catches mode collapse—a common failure in GANs where the generator produces only a subset of the real data's diversity.
• Unlike univariate metrics, PSM captures multivariate distributional fidelity in a single scalar value.

Beyond the aggregate pMSE, PSM enables feature-level utility analysis by examining which variables most influence the classifier's decisions. Feature importance scores from the propensity model reveal where the synthetic data deviates from the real distribution.

• High-importance features indicate columns where the generator struggled to replicate marginal distributions or inter-attribute correlations.
• This diagnostic guides iterative model improvement: data scientists can focus on specific columns for preprocessing adjustments or generator architecture changes.
• Partial dependence plots of propensity scores against individual features show exactly where in the feature space the synthetic data diverges.
• This granularity makes PSM more actionable than single-number utility metrics like the Hellinger distance.

PSM complements but does not replace other synthetic data evaluation methods. Each metric captures a different dimension of fidelity:

• Univariate statistics (mean, variance, KS test): Compare marginal distributions column-by-column but miss multivariate relationships.
• Pairwise correlation matrices: Assess bivariate relationships but scale poorly to high-dimensional data.
• Discriminative metrics like PSM: Capture joint distributional fidelity across all features simultaneously.
• Domain-specific utility: Measures whether synthetic data produces the same downstream model performance as real data.
• Best practice is to report multiple utility dimensions—PSM for overall distributional fidelity, univariate tests for column-level accuracy, and task-specific benchmarks for practical utility.

In sovereign AI infrastructure, PSM is executed entirely within the on-premises synthetic data vault, ensuring that real sensitive data never leaves the controlled environment during quality assessment.

• The combined real-synthetic dataset is constructed in-memory within the secure enclave.
• The propensity classifier is trained using differentially private algorithms if the scores themselves could leak information.
• Automated CI/CD pipelines run PSM as a quality gate before synthetic datasets are released to downstream consumers.
• Thresholds for acceptable pMSE are defined in the disclosure control framework and enforced programmatically.
• Results are logged to tamper-proof audit trails for regulatory compliance and model governance.

The degree to which a synthetic dataset accurately reproduces the statistical properties, joint distributions, and complex inter-attribute relationships of the original real-world data. Propensity score matching is a direct measure of this fidelity.

• Measures preservation of multivariate relationships
• A propensity score of 0.5 indicates perfect indistinguishability
• Complements univariate metrics like histogram similarity

A privacy attack where an adversary determines whether a specific data record was included in the training set of a machine learning model by analyzing its output behavior. Propensity score matching formalizes this risk by quantifying how easily a classifier can distinguish real from synthetic records.

• High discriminability signals potential training data leakage
• Used to audit generative models before release
• Related to differential privacy guarantees

A neural network architecture where a generator creates synthetic data samples and a discriminator evaluates their authenticity, iteratively competing until the artificial data is statistically indistinguishable from the real training distribution.

• The discriminator's loss curve mirrors the propensity score concept
• CTGAN adapts this architecture specifically for tabular data
• Convergence occurs when the discriminator performs no better than random chance

The probability that an attacker can successfully link anonymized or synthetic records back to the specific real-world individual they describe by using auxiliary information. Propensity score matching helps bound this risk.

• Low propensity scores correlate with higher singling-out risk
• Used alongside k-anonymity and l-diversity checks
• Essential for regulatory compliance under GDPR and HIPAA

A metric measuring the minimum cost of transforming one probability distribution into another, often used as a stable training objective for generative models. While propensity score matching evaluates discriminability, Wasserstein distance evaluates distributional similarity.

• Provides more stable gradients than Jensen-Shannon divergence
• Used in WGAN architectures for improved convergence
• Complements propensity scores in comprehensive quality evaluation frameworks

A mathematical framework that injects calibrated statistical noise into datasets or queries to guarantee that the presence or absence of any single individual's record is indistinguishable. Propensity score matching can validate that differential privacy mechanisms haven't destroyed utility.

• Parameterized by epsilon (ε) — the privacy budget
• DP-SGD applies this during model training
• Trade-off: lower epsilon improves privacy but degrades propensity scores