Federated Confusion Matrix

The core mechanism relies on Secure Aggregation (SecAgg) protocols to sum the raw counts for each cell (TP, FP, TN, FN) across institutions. Each client computes its local confusion matrix on its private test set and encrypts the four integer values. The central server computes the element-wise sum of these encrypted vectors, producing the global confusion matrix without ever decrypting or inspecting any single institution's patient-level predictions.

Once the global confusion matrix is securely aggregated, all standard classification performance metrics can be derived centrally without further client communication:

• Precision: TP / (TP + FP)
• Recall (Sensitivity): TP / (TP + FN)
• Specificity: TN / (TN + FP)
• F1-Score: Harmonic mean of precision and recall
• Accuracy: (TP + TN) / Total Predictions This decouples metric computation from raw data access.

To prevent membership inference attacks that could be enabled by exact confusion matrix counts, Differential Privacy (DP) noise is often applied. Each client adds calibrated noise—typically from a Laplace or Gaussian distribution—to their local TP, FP, TN, and FN counts before encryption. This consumes a portion of the privacy budget (epsilon) but provides a mathematical guarantee that the final aggregated matrix does not leak information about any single patient's test record.

Federated confusion matrices can be computed for specific demographic subgroups to audit algorithmic fairness across institutions. By securely aggregating separate matrices for each protected group (e.g., by race, sex, or age bracket), a governance team can compute Equalized Odds and Demographic Parity metrics. This reveals whether a model exhibits disparate false positive or false negative rates against specific populations without exposing the underlying patient demographics of any single hospital.

A single confusion matrix is tied to a specific classification threshold. For a more complete picture, federated systems often aggregate confusion matrices at multiple thresholds to construct a Federated ROC Curve and compute the Federated AUC. This threshold-independent metric is critical in healthcare, where the clinical operating point may vary by institution based on local risk tolerance and resource constraints.

A malicious or malfunctioning client could submit arbitrarily large or negative confusion matrix counts to corrupt the global evaluation. Byzantine-resilient aggregation techniques, such as trimmed mean or median-based aggregation, can be applied to each cell independently. This ensures that a minority of adversarial nodes cannot skew the perceived global performance of a diagnostic model, maintaining the integrity of the federated audit trail.

The building blocks of the Federated Confusion Matrix. Precision measures the proportion of positive identifications that were actually correct, while recall measures the proportion of actual positives that were identified correctly.

• Federated Precision: TP / (TP + FP), aggregated across nodes
• Federated Recall: TP / (TP + FN), computed without sharing individual predictions
• These metrics are derived directly from the securely summed confusion matrix cells
• Critical for evaluating diagnostic models where false positives and false negatives carry different clinical costs

The harmonic mean of federated precision and federated recall, providing a single balanced metric when class distributions are imbalanced across institutions.

• Computed as 2 * (Precision * Recall) / (Precision + Recall)
• Harmonic mean penalizes extreme values more than arithmetic mean
• Particularly useful when one institution has rare disease cases
• Requires only the aggregated TP, FP, and FN counts from the federated confusion matrix
• Avoids the accuracy paradox where a model appears performant by simply predicting the majority class

The Area Under the Receiver Operating Characteristic Curve computed in a decentralized manner. Unlike the confusion matrix, AUC is threshold-independent and evaluates model performance across all possible classification thresholds.

• Requires aggregating prediction scores, not just hard classifications
• Secure aggregation protocols enable summing scores without exposing individual patient predictions
• More robust than accuracy for imbalanced medical datasets
• Often used alongside confusion matrix metrics for comprehensive evaluation
• ROC curve plots True Positive Rate vs. False Positive Rate at various thresholds

A fairness metric that extends the federated confusion matrix to audit for algorithmic bias across protected groups. Requires that true positive and false positive rates be equal across demographic subgroups.

• Evaluated by computing separate confusion matrices for each protected group
• Demographic parity is a related but less strict criterion
• Federated computation ensures patient demographics remain local
• Critical for regulatory compliance in healthcare AI under frameworks like the EU AI Act
• Detects when a model systematically misdiagnoses specific populations

The cryptographic protocol that makes federated confusion matrices possible. SecAgg allows a central server to compute the sum of model updates or metrics from multiple clients without being able to inspect any individual client's contribution.

• Uses secret sharing and pairwise masking to protect in-transit data
• Ensures that even the aggregator cannot see individual TP, FP, TN, FN counts
• Often combined with differential privacy for defense-in-depth
• Essential for HIPAA and GDPR compliance in multi-institutional studies
• Prevents inference attacks that could reconstruct patient-level predictions

A metric that measures how well a model's predicted probabilities align with its actual accuracy, computed in a federated fashion. A perfectly calibrated model with 80% confidence should be correct 80% of the time.

• Computed by binning predictions by confidence and comparing to observed accuracy
• Federated ECE requires aggregating bin counts across institutions
• Complements confusion matrix analysis for clinical decision support
• Miscalibrated models can lead to overconfident misdiagnoses
• Critical when probability estimates directly inform treatment decisions