Federated Model Drift Detection

Identifies when the statistical relationship between input features and the target variable changes over time. In healthcare, this occurs when disease presentation patterns shift due to new variants, demographic changes, or evolving clinical practices.

• Monitors for covariate shift (changes in input distribution)
• Detects prior probability shift (changes in class distribution)
• Tracks concept drift (changes in P(Y|X) relationship)
• Triggers alerts when model reliability degrades below threshold

A decentralized computation of Population Stability Index (PSI) that quantifies distribution shifts between a reference baseline and current monitoring window without pooling raw patient data across institutions.

• Each client computes local PSI on its own data
• Only aggregated statistics are shared with the central server
• Enables detection of data drift at individual hospital sites
• Common threshold: PSI > 0.25 indicates significant drift

Tracks key model performance metrics across distributed nodes using privacy-preserving aggregation of evaluation statistics. Each institution computes local metrics on its own patient data, and only aggregated results are shared.

• Monitors federated AUC for threshold-independent assessment
• Tracks federated F1-score for balanced precision-recall evaluation
• Computes federated confusion matrices via secure summation
• Maintains per-client performance dashboards for site-specific drift

Measures the epistemic uncertainty of model predictions across the federated network to identify when the model encounters unfamiliar data patterns. Rising uncertainty signals potential drift before accuracy metrics degrade.

• Implements Monte Carlo Dropout at inference time on each client
• Aggregates predictive entropy across the distributed network
• Computes Expected Calibration Error (ECE) in a federated manner
• Flags high-uncertainty predictions for clinical review

Identifies inference-time inputs that differ fundamentally from the federated training distribution. When a hospital encounters novel patient presentations, OOD detection prevents the model from making unreliable predictions.

• Uses density estimation techniques on each local node
• Employs distance-based methods in feature space
• Aggregates OOD scores without sharing patient data
• Triggers model retraining workflows when OOD rate exceeds threshold

Establishes threshold-based policies that automatically initiate federated model retraining or fine-tuning when drift metrics exceed acceptable bounds, ensuring the global model adapts to evolving clinical realities.

• Configurable drift severity levels (warning, critical, action-required)
• Integrates with federated learning pipelines for seamless retraining
• Supports A/B testing of updated models before full deployment
• Maintains audit trails of all drift events and remediation actions

A fundamental failure mode in production ML where the statistical relationship between input features and the target variable changes over time. In healthcare, this manifests when disease presentation patterns shift due to new variants, demographic changes, or evolving clinical practices. Unlike data drift, concept drift means the same input now maps to a different output—a model that once accurately predicted sepsis risk from vital signs may become unreliable as treatment protocols advance. Detection requires monitoring prediction distributions rather than just input features, making it particularly challenging in federated settings where raw predictions cannot be centralized.

A decentralized metric for quantifying distributional shifts in model inputs across monitoring periods without pooling raw data. Each institution computes local PSI by comparing the distribution of features in the current inference window against a reference baseline, then shares only aggregated bin counts. Key characteristics:

• PSI < 0.1: No significant drift
• PSI 0.1–0.25: Moderate drift requiring investigation
• PSI > 0.25: Severe drift demanding model retraining Federated PSI enables privacy-compliant early warning systems across hospital networks.

Techniques for estimating model confidence and epistemic uncertainty in decentralized settings, critical for drift detection. Methods include:

• Monte Carlo Dropout: Applying dropout at inference to generate prediction distributions
• Deep Ensembles: Training multiple models locally and aggregating variance
• Federated Conformal Prediction: Generating prediction sets with guaranteed coverage Rising uncertainty scores across clients often signal concept drift before accuracy metrics degrade, providing an early warning system for clinical decision support reliability.

A metric measuring the mismatch between predicted confidence and actual accuracy, computed in federated fashion. A well-calibrated model predicting 90% confidence should be correct 90% of the time. ECE degradation signals drift:

• Overconfidence: Model reports high confidence on incorrect predictions—dangerous in clinical settings
• Underconfidence: Model hesitates on correct predictions—reduces clinical utility Federated ECE computation requires secure aggregation of binned confidence-accuracy pairs across institutions without exposing individual prediction outcomes.

The task of identifying inference inputs that differ fundamentally from the federated training distribution. In healthcare, OOD inputs include rare disease presentations, new medical devices generating unfamiliar data formats, or patient demographics absent from training. Detection methods include:

• Mahalanobis distance in feature space
• Energy-based models scoring input typicality
• Gradient-based detection analyzing model activation patterns OOD detection triggers clinical review workflows, preventing silent failures when models encounter unfamiliar cases.

A technique for embedding verifiable ownership identifiers into model weights during federated training, enabling drift monitoring across deployment instances. Watermarks serve dual purposes:

• Intellectual property protection: Proving model provenance across institutions
• Version tracking: Identifying which training round produced a degrading model Watermark extraction can be performed locally at each institution, enabling drift detection without central coordination. Trigger-set watermarks embed specific input-output pairs that activate only for authorized verification queries.