Federated Test Set

A core characteristic is that the test set must be statistically representative of the global target population the model is intended to serve. This is critical because client data is often non-IID (Non-Independent and Identically Distributed). The test set should approximate the aggregate distribution of all client data to provide a valid measure of global model accuracy and generalization.

• Purpose: To prevent over-optimistic performance metrics that might arise from testing only on a subset of client distributions.
• Challenge: Constructing this set without centralizing raw client data, often requiring synthetic data generation or secure, privacy-preserving sampling protocols.

The test set can be held and managed in two primary ways:

• Centralized Test Set: Held by the coordinating server, constructed from a separate, publicly available dataset or a small, consented subset of data that mirrors the target domain. This allows for consistent, repeatable evaluation but may not perfectly match the private client data distributions.
• Federated Test Set: Constructed via a secure aggregation protocol where test metrics (e.g., loss, accuracy) are computed locally on each client's private held-out data and then aggregated on the server. This better reflects true performance across heterogeneous data but requires careful design to ensure statistical validity and prevent information leakage.

The choice impacts evaluation fairness, privacy, and operational overhead.

The federated test set is the primary tool for quantifying the generalization gap—the difference between a model's performance on local training data and its performance on unseen, global data. In federated learning, this gap can be severe due to statistical heterogeneity.

• Process: After each communication round, the global model is evaluated on the federated test set. This provides a signal of how well the aggregated model generalizes beyond the specific, potentially biased data of any single client.
• Outcome: A persistently large gap indicates the model is overfitting to local client data patterns and failing to learn a robust, globally useful representation, guiding adjustments to training algorithms like Federated Averaging.

Accessing or constructing the test set must adhere to the same privacy-preserving principles as the training process. Direct centralization of client data for testing is often prohibited.

• Secure Federated Evaluation: Clients compute metrics (e.g., accuracy, F1-score) on their local test splits. These scalar metrics are then sent to the server, often protected by differential privacy (DP) noise or via secure multi-party computation protocols to prevent reverse engineering.
• Threat Mitigation: Protects against membership inference attacks that could determine if a specific data point was in a client's test (or training) set based on the model's output or aggregated metrics.

Beyond aggregate accuracy, a well-constructed federated test set enables the evaluation of model fairness and robustness across diverse subpopulations.

• Fairness Metrics: The test set should contain metadata (e.g., for sensitive attributes) to compute metrics like demographic parity or equalized odds across groups. This assesses if the global model performs equitably for all clients.
• Robustness Evaluation: The set can include adversarial examples or data from known edge-case distributions to test the model's Byzantine robustness and stability against out-of-distribution inputs, ensuring reliability in production.

The federated test set (or a separate validation set) is essential for federated hyperparameter tuning. Since the server cannot directly experiment on client data, performance on the central or aggregated test set serves as the objective function for tuning parameters like learning rate, number of local epochs, and client selection fraction.

• Process: The server proposes hyperparameter configurations, trains models across rounds, and uses the test set performance to guide the search (e.g., via federated versions of random search or Bayesian optimization).
• Caveat: To avoid overfitting, a strict hold-out test set used only for final evaluation is critical; a separate validation split should be used for tuning.

The decentralized process of assessing a model's performance across multiple clients without centralizing their private data. This involves:

• Clients computing metrics (e.g., accuracy, loss) on their local data.
• Securely transmitting these metrics to the central server.
• The server aggregating the metrics (e.g., via weighted averaging) to produce a global performance estimate. This method preserves data privacy but can be skewed by statistical heterogeneity across clients.

A model evaluation technique where a model trained on a subset of clients is validated on the held-out data from other, unseen clients. This is crucial for estimating generalization to new data distributions. Key approaches include:

• Leave-One-Client-Out: Train on N-1 clients, validate on the excluded client's data.
• K-Fold Cross-Client: Partition clients into K folds for iterative training and validation. It provides a more realistic estimate of performance on novel edge devices than a simple centralized test set.

The primary performance metric for the aggregated federated model, measured on a globally representative test set. This set may be:

• Held centrally by the server (a federated test set).
• Constructed via secure protocols from client-held validation data. It answers the core question: "How well does the final, unified model perform for the overall target population?" It must be distinguished from local model accuracy, which can vary significantly.

The quantitative evaluation of an individual client's impact on the final global model's performance. Measuring this is complex but essential for incentive design and data valuation. Common methods include:

• Shapley Value: A game-theoretic approach that fairly attributes the model's overall utility to each participant.
• Influence Functions: Estimate how the model's predictions would change if a specific client's data were removed from training.
• Update Magnitude/Quality: Simpler heuristics based on the norm or cosine similarity of a client's submitted gradient updates.

Quantitative measures used to evaluate whether a federated model's predictions exhibit unwanted bias against specific subgroups. These metrics must be computed in a privacy-preserving, federated manner. Common metrics include:

• Demographic Parity: Equal prediction rates across groups.
• Equalized Odds: Equal true positive and false positive rates across groups.
• Disparate Impact: A ratio of outcomes between groups. Challenges arise because sensitive attributes (e.g., race, gender) are often private client data that cannot be centralized.

The difference in performance between a model on its local training data and its performance on unseen global data. In federated learning, this gap can be exacerbated by non-IID data distributions. A large gap indicates overfitting to local client data, which harms the global model's utility. Mitigation strategies include:

• Regularization techniques applied during local training.
• Using proximal terms in the local objective (as in FedProx) to penalize deviation from the global model.
• Careful tuning of the number of local epochs.