Global Model Accuracy

Global model accuracy is the primary performance metric for the aggregated federated model, measured on a held-out test set that is statistically representative of the overall target population. Its core purpose is to answer the fundamental question: How well does the collaboratively trained model generalize to new, unseen data from the same overall distribution? This metric is distinct from local accuracy, which measures performance on individual clients' private data, and serves as the ultimate benchmark for the success of the federated training process.

The value and interpretation of global accuracy are intrinsically linked to the degree of statistical heterogeneity (non-IID data) across clients.

• In an ideal IID setting, local and global accuracy converge, and the metric is straightforward.
• Under high heterogeneity, a high global accuracy indicates the model has learned a robust, general representation that transcends local data biases. A significant gap between high global accuracy and low local accuracy for specific clients signals a generalization-performance trade-off, where the global model may not be optimal for all participants, prompting the need for personalized federated learning techniques.

Measuring global accuracy requires a federated evaluation protocol, as raw test data cannot be centralized.

Common methods include:

• Central Test Set: The server maintains a small, curated, and globally representative dataset (the federated test set) for final evaluation. This is the most direct method but requires upfront data collection.
• Cross-Client Validation: The model is validated on the held-out data from a randomly selected subset of clients not used in a given training round. The aggregated results estimate generalization to new client distributions.
• Secure Accuracy Aggregation: Clients compute accuracy metrics locally on their private test splits and send only the encrypted results (e.g., correct/total counts) to the server for secure summation, preserving data privacy during evaluation.

Global accuracy exists in a direct tension with privacy guarantees, formalized as the utility-privacy trade-off.

• Applying differential privacy (DP) mechanisms by adding calibrated noise to client updates provides formal privacy guarantees but typically reduces final global model accuracy. The privacy budget (epsilon, ε) controls this trade-off: a smaller ε (stronger privacy) usually results in lower accuracy.
• Similarly, secure aggregation protocols protect individual contributions but add computational overhead without directly reducing accuracy, unlike DP noise.
• System designers must explicitly balance the required accuracy threshold against the mandated level of data privacy for the application domain (e.g., healthcare vs. next-word prediction).

Global accuracy is the outcome metric for two critical federated learning properties: convergence and robustness.

• Convergence Rate: The speed at which global accuracy improves per communication round. Slow convergence increases communication costs and delays deployment.
• Model Convergence: The state where global accuracy plateaus, indicating the federated averaging process has stabilized.
• Byzantine Robustness: A robust global model (achieved via robust aggregation rules like Krum or median) maintains high accuracy even when a fraction of clients are malicious and send corrupted updates. A sudden drop in global accuracy can signal an active poisoning attack.

For practitioners, global accuracy is not a standalone number but a benchmark for comparative analysis.

Key comparisons include:

• Vs. Centralized Baseline: The accuracy achieved if all data were pooled centrally (often the performance ceiling).
• Vs. Local-Only Models: Demonstrates the collaborative benefit of federated learning.
• Across Aggregation Algorithms: Used to evaluate the effectiveness of different federated averaging algorithms (e.g., FedAvg, FedProx) under varying conditions.
• Across Communication Rounds: Plotted to visualize convergence and identify the point of diminishing returns for training. A high, stable global accuracy validates the entire federated learning pipeline—from client selection and local training to secure aggregation and privacy mechanisms—as effective for the task.

Local model accuracy measures a model's performance on the private data held by an individual client device. In federated learning, this metric often diverges from the global model's accuracy due to statistical heterogeneity (non-IID data). A significant gap between high local accuracy and low global accuracy indicates the model has overfit to the client's specific data distribution and fails to generalize.

• Key Insight: High local accuracy does not guarantee a useful global model.
• Evaluation Use: Used to assess personalization performance and to detect client drift during training.

Model convergence in federated learning refers to the state where the global model's parameters stabilize and its performance on a federated test set ceases to improve significantly with further training rounds. Measuring convergence is complex due to decentralized, asynchronous updates.

• Convergence Rate: The speed at which accuracy improves per communication round, heavily impacted by client selection strategies and data heterogeneity.
• Monitoring: Tracked via the global loss function and accuracy across rounds. Slow or unstable convergence often signals issues like high client drift or poor robust aggregation.

Federated evaluation is the decentralized process of assessing model performance without centralizing private data. Metrics like accuracy, loss, and fairness metrics are computed locally on each client's data and then aggregated (e.g., averaged) on the server.

• Core Challenge: Ensuring the aggregated metric is representative of the global population, not biased by which clients report.
• Cross-Client Validation: A technique where a model trained on one client subset is validated on held-out data from other clients to estimate generalization gap.
• Secure Protocols: Often paired with secure aggregation to prevent the server from learning individual client metrics.

Fairness metrics quantitatively assess whether a federated model's predictions exhibit unwanted bias against subgroups defined by sensitive attributes (e.g., race, gender). Evaluating fairness is critical but challenging due to decentralized, potentially unlabeled sensitive data.

• Common Metrics: Demographic Parity (equal prediction rates across groups), Equalized Odds (equal true positive and false positive rates).
• Federated Calculation: Requires careful, privacy-preserving protocols to compute these metrics across clients without exposing group membership.
• Trade-off: Often exists between global accuracy and fairness, requiring explicit optimization.

The generalization gap in federated learning is the performance difference between a model on its local training data and its performance on unseen, globally representative data. This gap is typically larger than in centralized learning due to non-IID data distributions.

• Primary Cause: Local models overfit to their client's unique data distribution.
• Measurement: Estimated via federated evaluation on a held-out federated test set or through cross-client validation.
• Mitigation: Addressed by techniques like Federated Averaging (FedAvg) with proper tuning, personalized federated learning, and robust aggregation.

The utility-privacy trade-off describes the inverse relationship where strengthening formal privacy guarantees in federated learning typically reduces the final model's accuracy or utility. This is a fundamental design consideration.

• Privacy Mechanisms: Adding differential privacy (DP) noise to client updates or using secure aggregation protects data but can harm convergence and final global model accuracy.
• Quantification: Managed by the epsilon (ε) privacy budget in DP; a smaller ε means stronger privacy but lower utility.
• Engineering Goal: To develop algorithms that maximize model performance for a given privacy constraint.