Federated Conformal Prediction

A distribution-free framework that wraps any pre-trained model to produce prediction sets with a finite-sample coverage guarantee. Instead of a single point prediction, it outputs a set of plausible labels such that the true label is included with a user-specified probability (e.g., 90%).

• Key Mechanism: Uses a held-out calibration dataset to compute nonconformity scores that measure how unusual a new input-output pair is.
• Guarantee: The marginal coverage probability holds exactly, assuming only exchangeability of the calibration and test data.
• Clinical Relevance: Provides rigorous uncertainty bounds for high-stakes decisions where a single incorrect prediction is unacceptable.

The aggregation of nonconformity scores computed locally at each client institution without sharing raw patient data. Each client calculates scores on its local calibration split, and these scores are securely transmitted to a central server.

• Privacy Preservation: Raw features and labels never leave the local institution; only abstract nonconformity scores are shared.
• Heterogeneity Challenge: Differing data distributions across sites can bias the global quantile if not properly weighted.
• Weighted Aggregation: Client scores are often weighted by local calibration set size to ensure the global coverage guarantee holds across the entire federated population.

The core statistical requirement for conformal prediction to provide valid coverage guarantees. A sequence of data points is exchangeable if their joint distribution is invariant under any permutation.

• Federated Implication: The global calibration set formed by pooling local scores must be exchangeable with future test points.
• Violation Risks: Non-IID data distributions across hospitals can break exchangeability, leading to overconfident or underconfident prediction sets at specific sites.
• Mitigation: Federated conformal methods often employ client-specific calibration or domain adaptation to restore approximate exchangeability.

The privacy-preserving calculation of the empirical quantile of aggregated nonconformity scores, which serves as the threshold for constructing prediction sets. For a target coverage of 1-α, the (1-α) quantile is computed.

• Secure Aggregation: Cryptographic protocols like Secure Aggregation (SecAgg) ensure the server learns only the aggregated quantile, not individual client score distributions.
• Differential Privacy: Noise can be added to the quantile computation to provide formal DP guarantees, trading off coverage precision for privacy.
• Finite-Sample Validity: The coverage guarantee holds exactly for finite calibration sets, unlike asymptotic methods.

An extension of standard marginal conformal prediction that aims to provide valid coverage within specific subgroups or clinical contexts, not just on average across the entire population.

• Clinical Necessity: Marginal guarantees may mask poor coverage for rare diseases or minority demographic groups.
• Federated Challenge: Subgroup definitions must be consistent across institutions, and sufficient calibration data per subgroup must exist in the distributed network.
• Mondrian Conformal Prediction: A federated adaptation that partitions the calibration data into categories and computes separate quantiles for each, ensuring coverage within each group.

The broader category of techniques for estimating the confidence and epistemic uncertainty of model predictions in decentralized settings, of which conformal prediction is a key distribution-free method.

• Epistemic vs. Aleatoric: Distinguishes between model uncertainty (reducible with more data) and inherent data noise (irreducible).
• Complementary Methods: Includes Federated Monte Carlo Dropout and Federated Deep Ensembles, which estimate uncertainty through stochastic forward passes or model diversity.
• Clinical Decision Support: Prediction sets from conformal methods provide actionable uncertainty—when the set is large, the model is uncertain, and a clinician should intervene.