Out-of-Distribution (OOD) Detection

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test that determines if two samples (real vs. synthetic) are drawn from the same distribution. It computes the distance between the mean embeddings of the datasets in a Reproducing Kernel Hilbert Space (RKHS).

• A low MMD score suggests the synthetic data distribution closely matches the real one.
• It is a non-parametric test, making no assumptions about the underlying distribution form.
• Commonly used kernels include the Gaussian (RBF) kernel and polynomial kernel.
• Kernel Inception Distance (KID) is a related, unbiased estimator often used for image data.

This method trains a discriminative classifier (the domain classifier) to distinguish between real and synthetic data samples. The core premise is that high-quality synthetic data should be indistinguishable from real data.

• A classifier achieving near random (50%) accuracy indicates successful distribution matching.
• This approach is a practical implementation of the two-sample test.
• Adversarial validation extends this concept to detect any distribution shift, not just between real and synthetic, but also between training and validation sets.
• The classifier's performance (e.g., AUC-ROC) serves as a direct fidelity score.

These methods leverage the underlying generative model's estimation of data probability. They assume in-distribution (ID) data has a higher likelihood or density under the model than OOD data.

• For models like Variational Autoencoders (VAEs), the Evidence Lower Bound (ELBO) or reconstruction error can be thresholds.
• For Normalizing Flows and Autoregressive Models, the exact log-likelihood can be used directly.
• A key limitation is the "confidence gap", where some models assign higher likelihood to OOD data, necessitating calibrated thresholds or outlier exposure techniques.

These techniques identify OOD samples based on their geometric proximity to known in-distribution data in a feature space.

• Compute distances (e.g., Euclidean, cosine) between a synthetic sample's embedding and its k-nearest neighbors in the real data embedding set.
• Samples with anomalously large distances are flagged as OOD.
• The feature space is often the penultimate layer of a pre-trained neural network (e.g., an Inception network for images).
• This method directly relates to evaluating support coverage of the synthetic data.

This approach uses the predictive uncertainty of a model (or ensemble of models) trained on real data as an OOD signal. OOD samples typically induce higher uncertainty.

• Deep Ensembles: Measure variance in predictions across multiple models.
• Monte Carlo Dropout: Treats dropout at inference as approximate Bayesian inference, measuring prediction variance.
• Deterministic Uncertainty Quantification: Uses specialized architectures to produce both a prediction and an uncertainty estimate.
• High entropy in the softmax distribution of a classifier can also indicate OOD inputs, though it requires careful calibration.

These methods explicitly model the boundary or support of the in-distribution data, flagging any sample falling outside this learned region as OOD.

• Support Vector Data Description (SVDD): Learns a minimal hypersphere that encloses most ID data.
• One-Class SVM: Learns a decision boundary to separate ID data from the origin in a kernel space.
• Deep Support Vector Data Description: Uses neural networks to learn a transformed feature space where a compact hypersphere can be fitted.
• These are particularly useful when only ID data is available for training the detector, a common scenario in synthetic data validation.

A domain classifier is a discriminative model (e.g., a neural network) trained to distinguish between samples from the real data distribution and samples from the synthetic data distribution. Its performance is a direct proxy for distributional similarity.

• High accuracy indicates the synthetic data is easily distinguishable, signaling poor fidelity.
• Near-random accuracy (~50%) suggests the distributions are well-aligned, making them hard to tell apart.

This method is a core component of adversarial validation frameworks for synthetic data.

Adversarial validation is a general technique for detecting distribution shift between any two datasets. In synthetic data validation, it involves:

1. Combining & Labeling: Merging real and synthetic datasets, labeling their origin (real=0, synthetic=1).
2. Training a Classifier: Training a model (e.g., XGBoost, simple neural network) to predict this label.
3. Evaluating Performance: Measuring the classifier's AUC-ROC or accuracy.

A high-performing classifier indicates significant, detectable differences between the datasets. This method underpins the domain classifier approach and is a practical, model-agnostic test for distribution alignment.

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test used to determine if two samples (e.g., real vs. synthetic) are drawn from the same distribution. It works by comparing the mean embeddings of the datasets in a high-dimensional Reproducing Kernel Hilbert Space (RKHS).

• A low MMD value suggests the distributions are similar.
• A high MMD value provides evidence they are different.

MMD is a foundational metric for two-sample tests and is directly used in the calculation of the Kernel Inception Distance (KID) for image data.

A two-sample test is a statistical hypothesis test designed to answer a core validation question: "Are these two sets of observations drawn from the same distribution?"

• Null Hypothesis (H0): The real and synthetic data come from the same distribution.
• Common Tests: Include Kolmogorov-Smirnov (for 1D data), MMD (kernel-based, high-dimensional), and Classifier Tests (like adversarial validation).

Failing to reject H0 (high p-value) provides statistical evidence for distributional similarity. This is the formal statistical framework behind metrics like MMD and adversarial validation.

Support coverage evaluates the extent to which a synthetic dataset spans the entire range of possible values (the support) present in the real data distribution. It is a critical complement to fidelity metrics.

• Poor coverage occurs when synthetic data only represents a subset of the real distribution's modes, leading to mode collapse.
• Good coverage means the synthetic data contains examples from all regions of the real data space, including rare events and edge cases.

Metrics like Precision and Recall for Distributions (P&R) explicitly decompose quality into fidelity (precision) and coverage (recall).

Mode collapse is a fundamental failure mode in generative modeling where the model learns to generate only a limited subset of the true data distribution's variety.

• Manifestation: The generator produces very similar or identical outputs, ignoring many modes (high-density regions) of the target data.
• Consequence: Synthetic data lacks diversity and has poor support coverage, making it unsuitable for training robust models.

Detection involves analyzing output diversity and is a key risk that out-of-distribution detection and support coverage metrics aim to identify.