Distributional Shift Detection

Identifies changes in the input feature distribution P(X) while the conditional label distribution P(Y|X) remains stable. Common detection methods include:

• Two-sample statistical tests: Kolmogorov-Smirnov and Maximum Mean Discrepancy (MMD) compare training and production feature distributions
• Domain classifier AUC: Train a classifier to distinguish training from production samples; an AUC significantly above 0.5 signals drift
• Dimensionality reduction projections: PCA or t-SNE visualizations of combined training and production embeddings reveal clustering separation

Covariate shift is the most common drift type in production and often indicates sensor degradation, demographic changes, or adversarial manipulation of input features.

Detects changes in the marginal label distribution P(Y) without corresponding changes in P(X|Y). This is critical for classification tasks where class priors drift over time. Detection approaches include:

• Black Box Shift Estimation (BBSE): Uses confusion matrix properties to estimate new label proportions without access to true production labels
• Confidence-based weighting: Compares model confidence distributions across classes between training and production windows
• Importance re-weighting: Computes density ratios between training and production label distributions

Label shift is particularly dangerous in security contexts because an attacker can poison the training set to shift class balance, causing systematic misclassification.

Detects changes in the underlying relationship P(Y|X) between inputs and outputs. Unlike covariate or label shift, concept drift means the same input now maps to a different correct output. Detection methods include:

• Prediction error monitoring: Sustained increases in model loss or error rate on labeled production samples
• Drift detection methods (DDM/EDDM): Sequential analysis algorithms that track error rate statistics and raise alarms when error distributions change significantly
• Adaptive windowing (ADWIN): Maintains a variable-length window of recent samples and compares sub-window means to detect distribution changes

Concept drift is the most severe form of shift because it invalidates the model's learned decision boundary and requires retraining.

Employs rigorous hypothesis tests to determine if training and production samples originate from the same distribution. Key techniques include:

• Maximum Mean Discrepancy (MMD): A kernel-based metric that measures the distance between mean embeddings of two distributions in a reproducing kernel Hilbert space; zero if and only if distributions are identical
• Energy distance: A non-parametric test based on Euclidean distances between samples that is consistent against all alternatives
• Fréchet Inception Distance (FID): Adapted from GAN evaluation, measures distributional distance in feature space for high-dimensional data like images

These tests provide p-values and statistical confidence rather than heuristic thresholds, making them suitable for regulated environments requiring auditable drift decisions.

Pinpoints which specific features are driving distributional shift rather than only detecting global drift. This enables targeted remediation. Methods include:

• Marginal distribution comparisons: Per-feature KS tests or Jensen-Shannon divergence between training and production histograms
• SHAP value drift: Tracks changes in feature importance distributions over time to identify features whose relationships with predictions have shifted
• Correlation structure monitoring: Detects changes in the feature covariance matrix or mutual information network

Feature-level attribution is essential for root cause analysis. A detected shift in a single sensor reading may indicate hardware failure, while coordinated shifts across multiple features may signal a poisoning attack.

Distributional shift detectors operate across different temporal granularities, each suited to specific threat models:

• Online streaming detection: Processes samples one at a time using incremental statistics; detects rapid poisoning attacks injected into live data pipelines
• Batch window comparison: Compares fixed-size windows (e.g., hourly or daily batches) to a reference distribution; suitable for detecting slow-burn poisoning campaigns
• Landmark windowing: Maintains multiple reference windows from known-clean periods and compares current data against all of them simultaneously

Online detectors minimize detection latency but are more susceptible to false alarms from natural variance. Batch detectors provide higher statistical power at the cost of delayed response.

A specific type of distributional shift where the input feature distribution P(X) changes between training and deployment, but the conditional label distribution P(Y|X) remains constant. Detection is critical because a model trained on one input region may be forced to extrapolate on unseen data.

• Example: A self-driving car trained in sunny California deployed in snowy Norway.
• Detection: Use two-sample tests like Maximum Mean Discrepancy (MMD) on feature embeddings.
• Impact: Causes silent degradation in model confidence and accuracy without obvious errors.

A shift where the underlying relationship between inputs and outputs P(Y|X) changes over time, making the model's learned mapping obsolete. This is distinct from covariate shift and often signals a fundamental change in the environment.

• Example: Fraud patterns evolving to bypass a static detection model; the same transaction features now map to different risk levels.
• Detection: Monitor model performance metrics (precision, recall) against a delayed ground truth stream.
• Mitigation: Requires online learning or periodic retraining with fresh labeled data.

A kernel-based statistical test that compares two distributions by measuring the distance between their mean embeddings in a Reproducing Kernel Hilbert Space (RKHS). MMD is a non-parametric method widely used for distributional shift detection.

• Mechanism: Computes the squared distance between empirical means of feature maps for the reference and test datasets.
• Advantage: Does not assume a parametric form for the data distribution.
• Application: Run MMD on the penultimate layer embeddings of a deployed model to detect input drift without labels.

A symmetric metric that quantifies how much a variable's distribution has shifted over time by comparing the proportion of values across bins. Originating in credit risk modeling, PSI is a staple for monitoring feature drift in production ML pipelines.

• Formula: PSI = Σ (Actual% - Expected%) * ln(Actual% / Expected%)
• Interpretation: PSI < 0.1 indicates no significant shift; 0.1-0.25 suggests moderate shift; > 0.25 signals a major distributional change.
• Use Case: Track PSI for each input feature to isolate which specific signals are drifting, potentially indicating targeted data poisoning.

A specialized monitoring approach that looks for distributional shifts consistent with clean-label poisoning or backdoor trigger injection. Unlike natural drift, adversarial shifts are often subtle and target specific latent subspaces.

• Technique: Apply Spectral Signatures to the covariance matrix of feature representations to identify anomalous directions of variance.
• Signal: A sudden, localized increase in the density of points near a decision boundary can indicate a poisoning attempt.
• Defense Integration: Couple drift detection with activation clustering to quarantine suspicious batches before they enter the training pipeline.

A non-parametric statistical test that measures the maximum distance between the empirical cumulative distribution functions (CDFs) of two samples. The KS statistic provides a scalar value for drift severity.

• Output: Returns a test statistic D and a p-value indicating the probability that the two samples are drawn from the same distribution.
• Sensitivity: Highly sensitive to shifts in location and shape, but less powerful in high-dimensional spaces without dimensionality reduction.
• Production Use: Apply per-feature KS tests in a data quality firewall to trigger alerts when individual feature distributions deviate beyond a set threshold.