Federated Learning Benchmark

The cornerstone of a realistic benchmark. Data is partitioned across simulated clients to reflect real-world statistical heterogeneity, moving beyond simple random splits. Common partition strategies include:

• Label distribution skew: Different clients have different class proportions (e.g., Dirichlet distribution).
• Feature distribution skew: The same label manifests with different features per client (e.g., different writing styles for digit '2').
• Quantity skew: Vastly different amounts of data per client.
• Concept shift: The relationship between features and labels changes across clients. Examples: LEAF benchmark, FedScale, Federated EMNIST with Dirichlet(α=0.1) partitioning.

Models real-world variability in client hardware and availability. A benchmark must specify constraints for each simulated client, including:

• Compute capability: Varying CPU/GPU power and memory limits.
• Network bandwidth: Heterogeneous and potentially slow upload/download speeds.
• Participation patterns: Clients are available sporadically (partial participation); some may drop out mid-round.
• Battery/power constraints: Limits on local training time for edge devices. This tests an algorithm's resilience and efficiency under the system constraints typical of cross-device federated learning.

Defines the adversarial capabilities against which algorithms are evaluated for robustness. A comprehensive benchmark includes threat models for:

• Privacy attacks: Model inversion, membership inference, and property inference attacks on aggregated updates.
• Byzantine robustness: Tolerance against a fraction of malicious clients performing poisoning attacks (e.g., label-flipping, backdoor insertion).
• Differential privacy guarantees: Benchmarks measure the trade-off between the privacy budget (ε, δ) and final model utility.
• Secure aggregation: Evaluates communication overhead and correctness of cryptographic protocols that hide individual updates.

Standardized metrics beyond final accuracy are crucial for holistic comparison. Key metrics include:

• Communication efficiency: Total bytes transmitted vs. target accuracy.
• Convergence speed: Number of training rounds to reach a threshold accuracy.
• Fairness: Performance variance across client groups (worst-case client accuracy).
• Personalization utility: Performance of personalized models vs. the global model on local client data.
• Robustness scores: Accuracy under attack or with noisy clients. Established baselines like Federated Averaging (FedAvg), FedProx, and SCAFFOLD must be included for relative performance comparison.

A diverse set of tasks and datasets to prevent overfitting to a single domain. A strong benchmark suite includes:

• Vision: Image classification (CIFAR-10/100, Federated EMNIST), object detection.
• Natural Language Processing: Next-word prediction (Reddit dataset), sentiment analysis.
• Speech: Keyword spotting (Google's Speech Commands).
• Recommendation: Collaborative filtering on decentralized user interaction data.
• Healthcare simulation: Synthetic medical imaging data with realistic institutional splits. Datasets should be scalable and come with pre-defined, reproducible client partitions.

The foundational aggregation algorithm for federated learning. A central server selects a subset of clients, each of which performs local Stochastic Gradient Descent (SGD) on its private data. The server then computes a weighted average of the returned model updates to form a new global model. Benchmarks rigorously test FedAvg's performance under non-IID data and partial client participation.

A mathematical framework providing a quantifiable privacy guarantee. In federated learning, DP-SGD is often applied on clients before updates are sent. A benchmark must evaluate the privacy-utility trade-off: how much model accuracy degrades as stronger noise (epsilon) is added to guarantee that any single data point cannot be identified in the training process.

Non-Independent and Identically Distributed data is the statistical norm, not the exception, in federated settings. This heterogeneity arises because client data distributions differ significantly (e.g., different writing styles per smartphone user). Benchmarks must simulate realistic non-IID splits (e.g., by label distribution or feature skew) to stress-test algorithms against client drift.

Advanced algorithms designed to overcome the limitations of basic FedAvg in heterogeneous environments. Key examples benchmarked include:

• FedProx: Adds a proximal term to the local loss function to limit client drift.
• SCAFFOLD: Uses control variates (variance reduction) to correct for client update bias.
• FedAvgM: Incorporates server-side momentum to stabilize convergence.

The two primary operational scales for federated learning, which dictate benchmark design:

• Cross-Device: Involves millions of unreliable, resource-constrained devices (smartphones). Benchmarks focus on partial participation, communication efficiency, and robustness to device dropouts.
• Cross-Silo: Involves tens of reliable, data-rich organizations (hospitals). Benchmarks focus on vertical FL, complex model architectures, and inter-organizational trust mechanisms.

A cryptographic protocol that allows a federated learning server to compute the sum of client model updates without being able to inspect any individual client's contribution. This protects privacy against a curious server. Benchmarks measure the computational and communication overhead of secure aggregation protocols and their impact on overall training time.