Federated Evaluation

The core mechanism where performance metrics (e.g., accuracy, F1-score, loss) are computed locally on each client's private dataset. Only the aggregated results (e.g., mean, weighted average, confidence intervals) are shared with the central server, never the raw data or individual data points. This is the fundamental privacy-preserving step.

• Example: A hospital client computes the AUC for a diagnostic model on its local patient records and sends only the scalar value 0.92 to the server.

Directly addresses statistical heterogeneity, where data distributions vary significantly across clients (e.g., different user demographics, regional patterns, device types). Federated evaluation must accurately measure global model performance despite this variance, which can cause significant client drift and divergent local accuracies.

• Techniques: Use weighted averaging based on dataset size, reporting performance percentiles across clients, and employing cross-client validation to estimate generalization.

Employs cryptographic and statistical techniques to prevent the server from inferring sensitive information from individual client metrics. While simpler than securing gradient updates, basic averaging of metrics can still leak data characteristics.

• Secure Aggregation Protocols: Extend beyond model updates to securely sum metrics.
• Differential Privacy (DP): Adds calibrated noise to locally computed metrics before aggregation, providing a formal privacy budget (ε) guarantee.

Must account for system heterogeneity—clients have varying computational power, memory, network connectivity, and availability. Evaluation rounds must be resilient to stragglers (slow devices) and device dropouts.

• Adaptive Client Selection: The server may select a representative but reliable subset of clients for evaluation rounds.
• Asynchronous Protocols: Allow clients to report metrics as they complete, avoiding bottlenecks.

Evaluates beyond basic accuracy to ensure model reliability and fairness in production. Key dimensions include:

• Global vs. Local Model Accuracy: Measures the aggregated model's performance versus its performance on individual client distributions.
• Fairness Metrics: Evaluates demographic parity, equalized odds across subgroups defined on-device.
• Model Robustness & Calibration: Assesses resilience to adversarial inputs and whether predicted probabilities reflect true likelihoods (Expected Calibration Error).

Encapsulates the fundamental tension in federated systems. Increasing privacy guarantees (e.g., lowering the epsilon (ε) in differential privacy) typically increases noise in aggregated metrics, reducing their statistical utility and precision for model selection.

• Quantifiable Trade-off: Engineers explicitly balance the privacy budget against the required confidence intervals for performance metrics. This is a key architectural decision.

A cryptographic protocol that allows a central server to compute the sum of client model updates without inspecting any individual contribution. This is a foundational privacy mechanism that complements federated evaluation by ensuring the aggregated evaluation metrics themselves do not leak information about a single client's data.

• Core Function: Enables privacy-preserving computation of averages (e.g., for accuracy, loss).
• Protocols: Often based on Secure Multi-Party Computation (MPC) or homomorphic encryption.
• Evaluation Use: Allows the server to learn the global mean of a metric (like average accuracy) without knowing which client contributed which value.

A rigorous mathematical framework that provides formal, quantifiable privacy guarantees. In federated evaluation, DP mechanisms can be applied to the metrics reported by clients (e.g., accuracy, loss) to prevent the server from inferring details about the underlying private data.

• Privacy Budget (ε): Controls the trade-off between privacy strength and metric utility.
• Mechanisms: Gaussian or Laplace noise is added to locally computed metrics before aggregation.
• Guarantee: Ensures the output of the evaluation (e.g., "global accuracy is 85%") does not reveal if any specific data point was included.

A model evaluation technique where a model trained on a subset of clients is validated on the held-out data from other, unseen clients. This is critical for estimating a federated model's generalization to new data distributions, which is a core challenge in non-IID settings.

• Purpose: Simulates how the model will perform for a new participant joining the system.
• Method: Clients are split into training cohorts and validation cohorts; the server aggregates validation metrics from the latter.
• Outcome: Provides a more realistic performance estimate than a single, centralized test set.

The process of quantifying the impact of an individual client's data and updates on the final global model's performance. This is important for incentive design, resource allocation, and detecting malicious participants.

• Measurement Techniques: Uses methods like Shapley values, TMR (Total Model Reward), or influence functions adapted for federated learning.
• Evaluation Link: A client's contribution can be assessed based on how its local evaluation metrics (e.g., improvement in loss) correlate with global improvement.
• Challenge: Must be computed in a privacy-preserving manner, often leveraging secure aggregation.

A class of aggregation algorithms designed to produce a reliable global model even when a fraction of clients are malicious or submit corrupted updates. These techniques are directly relevant to ensuring the integrity of federated evaluation metrics.

• Goal: Achieve Byzantine robustness.
• Algorithms: Include Krum, coordinate-wise median, and trimmed mean.
• Application to Evaluation: Robust aggregation rules can be applied to aggregated evaluation metrics (e.g., median accuracy) to filter out outliers from malicious clients attempting to skew reported performance.

Measures how well a federated learning model, after global training, can be adapted or fine-tuned to perform accurately on the specific data distribution of an individual client. This is a key evaluation target in heterogeneous environments.

• Evaluation Metric: The performance gain (e.g., accuracy delta) of a personalized model over the global model on a client's local test set.
• Methods: Includes Local Fine-Tuning, Meta-Learning (Per-FedAvg), and Multi-Task Learning.
• Federated Evaluation Role: Requires a two-stage assessment: first evaluating the global model, then evaluating the personalized variants on each client's private data.