Cross-Silo Validation

The gold-standard evaluation strategy where the global model is trained on all silos except one, which is held out entirely for testing. This process is repeated iteratively for every site.

• Mechanism: For N institutions, N independent training runs occur, each producing a performance metric on the held-out site.
• Purpose: Directly measures the model's ability to generalize to a completely unseen clinical environment, including its unique equipment, demographics, and labeling practices.
• Contrast: Unlike standard k-fold cross-validation, LOSO explicitly accounts for the non-IID nature of institutional data, where each site represents a distinct statistical domain.

A diagnostic metric that quantifies the consistency of model performance across the federation. High variance is a critical warning signal.

• Calculation: The standard deviation or interquartile range of AUC, F1-score, or calibration error across all LOSO folds.
• Interpretation: A low average error with high variance indicates the model performs excellently on some hospitals but dangerously poorly on others.
• Root Cause: Often triggered by a hidden confounder—a variable that correlates with the outcome in most sites but is absent or reversed in a minority site due to different clinical protocols.

Evaluates the probabilistic trustworthiness of model predictions across sites, ensuring that a predicted 10% risk of mortality actually corresponds to a 10% observed event rate.

• Expected Calibration Error (ECE): Computed locally per silo by binning predicted probabilities and comparing them to the actual fraction of positive cases.
• Federated Aggregation: Only the binned counts and ECE metrics are shared with the aggregation server, preserving patient privacy.
• Clinical Impact: A well-calibrated model is essential for clinical decision support; a miscalibrated model can lead to systematic over-treatment or under-treatment at specific institutions.

The systematic comparison of input feature distributions between the training federation and a new candidate site before deployment, preventing silent model failures.

• Covariate Shift: Detected by comparing the mean and variance of structured features (e.g., lab values, vital signs) using federated histograms or kernel density estimates.
• Label Shift: Detected by comparing the prevalence of target conditions (e.g., disease incidence rates) across sites without sharing individual labels.
• Action: If a significant shift is detected, the model may require federated domain adaptation or site-specific fine-tuning before clinical use.

A privacy-preserving method for constructing a global Receiver Operating Characteristic curve by aggregating only the confusion matrix elements at various thresholds from each silo.

• Vertical Aggregation: True positive and false positive counts are summed across sites at each threshold to compute a macro-averaged global ROC.
• Threshold Interpolation: Since sites may use different decision thresholds, secure interpolation protocols harmonize the operating points.
• AUC Calculation: The Area Under the Curve is computed on the server from the aggregated points, providing a single summary statistic of discriminative power without pooling raw prediction scores.

A fine-grained evaluation that slices LOSO performance by protected attributes and clinical subgroups to ensure algorithmic fairness across the entire federation.

• Federated Stratification: Performance metrics (e.g., sensitivity, specificity) are computed locally for subgroups defined by age, sex, ethnicity, or comorbidity status.
• Fairness Metrics: Equal opportunity difference and disparate impact ratios are calculated from the aggregated subgroup statistics.
• Regulatory Alignment: This analysis provides the quantitative evidence required by FDA and EMA submissions to demonstrate that a SaMD (Software as a Medical Device) performs equitably across diverse patient populations.

A rigorous external validation strategy where the global model is trained on data from all institutions except one, which is held out entirely for testing. This process is repeated iteratively, with each site serving as the test set once. It directly measures generalizability to completely unseen clinical environments, workflows, and patient demographics, exposing brittle models that overfit to dominant site characteristics.

The foundational challenge that necessitates cross-silo validation. Clinical data across hospitals is notoriously non-identically distributed due to varying patient demographics, equipment vendors, and clinical coding practices. Handling this statistical heterogeneity involves specialized algorithms like FedProx or SCAFFOLD, but validation is the ultimate test of whether these methods successfully produce a model that performs equitably across all local distributions.

The statistical quantification of performance variability across sites. This goes beyond a single global metric to analyze the distribution of local metrics using tools like forest plots and the I-squared statistic. A model with a high average AUC but extreme variance between sites is clinically unsafe; heterogeneity assessment identifies this risk, prompting investigation into the root causes of site-specific failures.

The process of ensuring a global model does not systematically disadvantage a specific patient subpopulation at a particular site. Cross-silo validation is the primary detection mechanism for this algorithmic bias. By disaggregating performance metrics by site and sensitive attributes (e.g., race, ethnicity) during validation, developers can identify and correct for demographic skew before clinical deployment.

Techniques for understanding why a federated model makes a specific prediction, validated across sites. This involves computing feature attribution methods like SHAP or LIME locally and aggregating the explanations to ensure the model uses clinically valid reasoning everywhere. Cross-silo validation of explanations confirms that a model is not relying on spurious, site-specific correlations (like scanner model) for its predictions.