Federated F1-Score

The Federated F1-Score is defined as the harmonic mean of federated precision and federated recall, giving equal weight to both metrics. Unlike a simple arithmetic mean, the harmonic mean penalizes extreme imbalances, ensuring a high F1-Score is only achieved when both precision and recall are high.

• Formula: 2 * (Federated Precision * Federated Recall) / (Federated Precision + Federated Recall)
• Penalizes Extremes: A model with perfect recall but near-zero precision will have a very low F1-Score, preventing misleadingly optimistic evaluations.
• Single Metric: It distills the trade-off between false positives and false negatives into one interpretable value for model selection.

The global F1-Score is not computed by averaging local F1-Scores, which is mathematically incorrect. Instead, it relies on the federated confusion matrix to securely aggregate the raw components: True Positives (TP), False Positives (FP), and False Negatives (FN) from each client.

• Privacy-Preserving Summation: Techniques like Secure Aggregation (SecAgg) are used to sum TP, FP, and FN counts across institutions without revealing any single institution's values.
• Global Calculation: The central server computes Global Precision = ΣTP / (ΣTP + ΣFP) and Global Recall = ΣTP / (ΣTP + ΣFN), then derives the F1-Score.
• Avoids Averaging Pitfalls: This method correctly accounts for varying dataset sizes and class distributions across silos.

In healthcare federated learning, disease prevalence often varies dramatically between institutions. The Federated F1-Score is particularly sensitive to this non-IID class distribution.

• Macro vs. Micro Averaging: A macro-averaged F1-Score calculates the metric for each class independently and then takes the unweighted mean, treating rare and common diseases equally. A micro-averaged F1-Score aggregates contributions from all classes globally, which can be dominated by high-prevalence conditions.
• Weighted Strategies: A weighted F1-Score can be computed by weighting each client's contribution by its total number of samples, ensuring larger hospitals don't skew the metric unfairly.
• Bias Detection: A significant discrepancy between macro and micro F1-Scores is a strong signal of class imbalance that requires investigation.

To prevent membership inference attacks from the aggregated counts, noise is often added to the TP, FP, and FN tallies using Differential Privacy (DP). This introduces a critical trade-off between privacy and metric accuracy.

• Privacy Budget (Epsilon): A lower epsilon value adds more noise, providing stronger privacy guarantees but potentially distorting the true F1-Score.
• Noise Calibration: The noise scale is calibrated to the sensitivity of the count queries. Since adding a single patient can change a count by at most 1, the sensitivity is 1.
• Confidence Intervals: Instead of reporting a single noisy F1-Score, a confidence interval should be provided to quantify the uncertainty introduced by the DP mechanism.

The F1-Score is a threshold-dependent metric, meaning its value is tied to a specific classification threshold (usually 0.5 for predicted probabilities). This is a critical limitation in federated settings where optimal thresholds may vary per site.

• Site-Specific Thresholds: A model might need a higher threshold at a community clinic (prioritizing precision to avoid false alarms) and a lower threshold at a research hospital (prioritizing recall for screening).
• Complement with Federated AUC: For a threshold-agnostic view of model performance, the Federated F1-Score should be reported alongside the Federated AUC, which evaluates performance across all possible thresholds.
• Federated Threshold Tuning: A separate federated process may be required to find the optimal global or personalized threshold that maximizes the F1-Score for a specific clinical objective.

The Federated F1-Score can be decomposed to audit for fairness across protected demographic groups. Equalized Odds requires that the True Positive Rate (Recall) and False Positive Rate are equal across groups.

• Group-Specific F1: Compute the Federated F1-Score independently for each demographic subgroup by securely aggregating TP, FP, and FN counts stratified by group membership.
• Fairness Gap: A significant difference in F1-Scores between groups indicates a violation of predictive equality, which is a critical regulatory risk in clinical AI.
• Intersectional Analysis: The same technique can be extended to intersectional subgroups (e.g., race and gender) to detect compounded biases that might be hidden in aggregate metrics.

The foundational building block for computing the Federated F1-Score. This technique securely aggregates True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN) counts from each institution without sharing individual patient predictions.

• Enables computation of federated precision and recall
• Uses secure summation protocols to protect per-patient outcomes
• Forms the basis for all federated classification metrics

The ratio of True Positives to total Predicted Positives (TP + FP), computed across a decentralized network. This metric answers: Of all patients the model flagged as positive, how many were actually positive?

• Critical for minimizing false alarms in clinical screening
• Aggregated from secure confusion matrix counts
• High precision is vital when follow-up procedures are invasive or costly

Also known as sensitivity, this is the ratio of True Positives to total Actual Positives (TP + FN). It answers: Of all patients who truly have the condition, how many did the model correctly identify?

• Essential for safety-critical diagnostic applications
• High recall minimizes missed diagnoses
• Computed without centralizing ground truth labels from any institution

The primary challenge that makes the Federated F1-Score essential. In healthcare, data across hospitals is not independent and identically distributed (non-IID) due to varying patient demographics, equipment, and clinical practices.

• A global accuracy metric can mask catastrophic local failure
• The F1-Score's harmonic mean penalizes models that sacrifice recall for precision on minority subgroups
• Client-level F1 tracking reveals performance disparities across sites

The Area Under the ROC Curve computed in a decentralized manner. While the F1-Score evaluates a model at a specific classification threshold, the Federated AUC provides a threshold-independent view of overall discriminative ability.

• Complements the F1-Score for comprehensive evaluation
• Aggregates prediction scores using privacy-preserving protocols
• Useful for comparing models before selecting an operational threshold

The systematic audit of a federated model for unfair performance disparities across protected demographic groups. The Federated F1-Score is computed per subgroup to detect imbalances.

• Evaluates demographic parity and equalized odds across distributed data
• Identifies if high global F1 masks poor performance on minority populations
• Essential for healthcare equity compliance under regulations like the EU AI Act