Federated Evaluation

The core mechanism of federated evaluation is the execution of model inference on test data that remains physically on each client device. The server coordinates the process but never sees the raw test data. Clients compute performance metrics (e.g., accuracy, F1-score, loss) locally and typically send only the aggregated results back to the server. This preserves the data privacy principle of federated learning throughout the model lifecycle, from training to final assessment.

Federated evaluation is explicitly designed to measure model performance in the presence of Non-IID data. Instead of reporting a single global accuracy figure, it reveals a distribution of performance across clients. Key outputs include:

• Client-specific metrics: Performance for each participating device.
• Performance variance: The standard deviation of accuracy across clients, indicating fairness and robustness.
• Worst-case performance: The minimum accuracy observed, critical for high-stakes applications. This analysis identifies if the global model has overfit to dominant data patterns or fails on underrepresented client distributions.

After local evaluation, the server must aggregate results to form a global view. Common aggregation strategies include:

• Simple Averaging: The mean of all client metrics. This is common but can be skewed by clients with vastly more test data.
• Weighted Averaging: Metrics are weighted by the size of each client's local test set, providing a more representative global figure.
• Robust Aggregation: Using median or trimmed mean to reduce the influence of outliers from clients with extremely poor or atypical data. The choice of aggregation method directly impacts the reported global performance and must align with the application's goals.

It is crucial to distinguish federated evaluation from federated validation and training:

• Training: Clients compute gradients on local training data to update model weights.
• Validation: Used during training for hyperparameter tuning or early stopping. Requires a held-out set, but data may still be Non-IID.
• Evaluation (Testing): The final, one-time assessment on a separate, unseen test set to estimate real-world performance. Data must not be used in any prior training or validation steps. In federated learning, ensuring clean, unseen test data on each client is a significant systems challenge.

While more communication-efficient than sharing raw data, federated evaluation still incurs overhead. Clients must receive the latest model and send back results. For large-scale deployments, this requires careful orchestration. From a privacy perspective, even sharing performance metrics can leak information. Techniques like Differential Privacy (DP) can be applied to the metrics before aggregation, or Secure Multi-Party Computation (MPC) can be used to compute global statistics without revealing individual client results, providing formal privacy guarantees.

Non-Independent and Identically Distributed (Non-IID) data is the statistical property where data samples across different clients are not drawn from the same underlying probability distribution. This violates the standard assumption of centralized machine learning and is the primary driver of challenges in federated learning, including client drift and performance disparities.

• Key Challenge: Causes local models to diverge from the global objective.
• Example: Medical imaging data from a rural clinic (mostly cases of disease X) vs. an urban hospital (a balanced mix of diseases).
• Synthetic Generation: Often created for research using Dirichlet distribution sampling to partition a dataset with a controlled concentration parameter α.

Statistical heterogeneity is the umbrella term for the variations in data distributions across participating clients in a federated system. It is the practical manifestation of Non-IID data and encompasses differences in:

• Feature Distribution (Covariate Shift): The distribution of input features P(X) varies.
• Label Distribution (Prior Probability Shift): The distribution of output labels P(Y) varies.
• Concept Drift (Conditional Shift): The relationship between features and labels P(Y|X) varies.
• Quantity Skew: The amount of data per client varies significantly.

This heterogeneity is quantified in theory by assumptions like Bounded Gradient Dissimilarity.

Personalized Federated Learning (PFL) is a family of techniques designed to produce models tailored to individual clients' local data distributions, rather than relying on a single global model. This is a direct solution to performance degradation caused by Non-IID data.

• Goal: Achieve high accuracy on each client's unique data distribution.
• Methods: Include training local fine-tuned models, learning client-specific parameters, or using meta-learning.
• Example Algorithm: Ditto, which trains personalized models regularized towards a global model to balance personalization and collaboration.
• Evaluation: PFL requires federated evaluation across all clients to measure both personalized and global model performance.

Client drift is the phenomenon where local models, trained on their unique Non-IID data, diverge from the global optimization objective. This causes instability in federated averaging and can degrade the final global model's performance.

• Cause: Local SGD steps on heterogeneous data pull models in different directions.
• Impact: Slows convergence and reduces final test accuracy.
• Mitigation Algorithms: FedProx adds a proximal term to local loss to limit drift. SCAFFOLD uses control variates to correct for client update variance.
• Evaluation: Detecting and measuring client drift is a core part of federated evaluation, often by tracking loss/accuracy variance across clients.

Federated Learning with Fairness Constraints involves optimizing not just for average global accuracy but for equitable performance across all participating clients. This is critical when Non-IID data naturally leads to performance disparities.

• Objective: Minimize the worst-case client performance or reduce accuracy variance across a demographic group.
• Challenge: Requires evaluation metrics beyond simple averages, such as the minimum client accuracy or the variance of accuracy.
• Connection to Evaluation: Federated evaluation is the essential process for measuring these fairness metrics, as it assesses model performance on each client's local test set.
• Methods: Include re-weighting client updates or adding fairness regularization terms to the global objective.