Distributional Shift

The most common form of drift, where the distribution of the input features P(X) changes, but the conditional relationship P(Y|X) remains stable.

• Example: A facial recognition system trained on high-resolution studio images deployed on low-resolution security cameras. The mapping from face to identity is unchanged, but the input pixel distribution has shifted.
• Synthetic Data Impact: Generators trained on old covariate distributions will produce samples that no longer represent the current feature space, leading to downstream model degradation.

Occurs when the distribution of the target variable P(Y) changes, while the class-conditional distributions P(X|Y) remain fixed.

• Example: A disease prediction model deployed during an epidemic where disease prevalence suddenly spikes from 1% to 15%. The symptom presentation for each condition is unchanged, but the base rate has shifted dramatically.
• Key Distinction: Unlike covariate shift, label shift is often easier to correct through recalibration of output probabilities without retraining the entire generator.

The most destructive form of shift, where the fundamental relationship between inputs and outputs P(Y|X) itself changes over time.

• Example: A fraud detection model where fraudsters continuously adapt their techniques. The same transaction pattern that was legitimate last month is now fraudulent—the meaning of the features has changed.
• Synthetic Data Consequence: Concept drift requires continuous retraining of synthetic generators on fresh real-world data to capture the new underlying relationships. Static generators will produce samples reflecting obsolete patterns.

A specific subtype where data distributions evolve as a function of time due to seasonality, trends, or external events.

• Example: Retail purchase patterns shifting between holiday and non-holiday seasons, or traffic patterns changing permanently after a bridge closure.
• Mitigation: Requires time-aware synthetic generation that conditions on temporal features or maintains a sliding window of recent training data to prevent the synthetic data factory from producing samples frozen in a past temporal context.

A systematic difference between the source domain (training data) and target domain (deployment environment) that is not strictly temporal.

• Example: A medical imaging model trained on scans from Hospital A's Siemens scanner but deployed on Hospital B's GE scanner. The underlying anatomy is identical, but the pixel-level representations differ due to hardware variations.
• Synthetic Data Strategy: Domain adaptation techniques can be integrated into generators to produce samples that bridge the gap between source and target distributions.

Monitoring for distributional shift requires rigorous statistical comparison between the reference distribution (training data) and the current production distribution.

• Population Stability Index (PSI): Measures the divergence between expected and actual feature distributions across bins.
• Kolmogorov-Smirnov Test: A non-parametric test comparing the cumulative distribution functions of two samples.
• Maximum Mean Discrepancy (MMD): A kernel-based method for detecting subtle multivariate shifts that univariate tests may miss.
• Practical Rule: Trigger synthetic data regeneration when drift metrics exceed predefined thresholds.

A metric measuring the minimum cost of transforming one probability distribution into another. Unlike Kullback-Leibler divergence, it provides a meaningful distance even when distributions have non-overlapping support. In synthetic data contexts, it serves as a stable training objective for generative models and a sensitive detector of distributional shift, quantifying exactly how much the real-world data has drifted from the training set.

A specific type of distributional shift where the input feature distribution P(X) changes, but the conditional relationship P(Y|X) remains constant. This is common in sensor degradation or demographic changes. Synthetic data generators must be retrained to reflect the new input distribution, otherwise they will produce samples that no longer represent the current operational environment.

The most disruptive form of distributional shift, where the fundamental relationship between inputs and outputs P(Y|X) changes over time. For example, a fraud detection model's definition of 'suspicious behavior' evolves as attackers adapt. Synthetic data factories must implement online learning or periodic retraining pipelines to capture these new patterns before the generated data becomes dangerously misleading.

The degree to which a synthetic dataset accurately reproduces the joint distributions, correlations, and marginal properties of the original data. Distributional shift in the source data directly degrades fidelity if the generator is not updated. Monitoring fidelity metrics like propensity score matching serves as a proxy for detecting when the real-world distribution has drifted beyond acceptable bounds.

A training algorithm that clips per-sample gradients and adds calibrated Gaussian noise during optimization. When retraining synthetic generators on fresh data to combat distributional shift, DP-SGD ensures that the updated model does not inadvertently memorize and expose new sensitive records. It provides a formal privacy budget (epsilon) that accumulates over successive retraining cycles.

Architectures that allow models to iteratively adapt in production without suffering from catastrophic forgetting. To combat distributional shift, synthetic data factories integrate these systems to continuously fine-tune generators on streaming data. Techniques include experience replay buffers and elastic weight consolidation, ensuring the generator evolves with the data distribution while retaining knowledge of previously valid states.