Federated Learning Benchmarks (LEAF)

A core contribution of LEAF is its systematic method for generating statistically heterogeneous data splits that mirror real-world federated scenarios. It uses a Dirichlet distribution with a concentration parameter (α) to partition datasets across simulated clients.

• Key Mechanism: A smaller α value (e.g., α=0.1) creates highly skewed, Non-IID label distributions, while a larger α (e.g., α=100) approximates IID conditions.
• Purpose: This allows researchers to stress-test algorithms across a spectrum of heterogeneity, from mild to extreme, which is critical for evaluating robustness to client drift.
• Example: In the FEMNIST dataset (handwritten character recognition), a low α might assign one client almost exclusively the digit '7', while another gets mostly '3'.

LEAF curates and preprocesses several canonical datasets tailored for federated evaluation, spanning text, image, and synthetic data domains. Each dataset includes metadata on client counts and data points per client.

• FEMNIST: Extended MNIST for handwritten character/letter recognition, with data partitioned by writer (a natural source of Non-IID).
• Sentiment140: Twitter data for sentiment analysis, partitioned by user.
• Shakespeare: Next-character prediction on the works of Shakespeare, partitioned by speaking character.
• Synthetic Dataset: A configurable, algorithmically generated dataset for controlled experiments on feature and label skew.

These datasets provide a common ground for comparing algorithm performance on tasks with varying complexity and data modalities.

LEAF provides a Python-based simulation environment that abstracts the complexities of distributed systems, allowing researchers to focus on algorithmic development. It models key federated constraints.

• Client Sampling: Supports various client selection strategies (random, stratified, power-of-choice).
• System Heterogeneity: Can simulate variations in client availability, compute speed, and connectivity.
• Communication Modeling: Tracks the volume of data (model parameters/gradients) exchanged between clients and the server.
• Extensibility: Researchers can easily plug in new datasets, partitioning schemes, and aggregation algorithms (FedAvg, FedProx, SCAFFOLD) for comparison.

Beyond simple accuracy, LEAF emphasizes comprehensive evaluation metrics critical for understanding federated learning dynamics under Non-IID data.

• Core Metrics: Tracks global model accuracy, personalized/local model accuracy, and convergence speed (communication rounds to target accuracy).
• Fairness & Variance: Measures performance disparity across clients (e.g., standard deviation of accuracy) to assess algorithmic fairness.
• Communication Efficiency: Reports total bytes transmitted, a key cost factor in real deployments.
• Standardized Reporting: Facilitates the generation of consistent plots and tables for comparing algorithms across different levels of statistical heterogeneity.

To establish a performance baseline, LEAF includes reference implementations of foundational federated learning algorithms. This allows new methods to be benchmarked against established techniques.

• Federated Averaging (FedAvg): The canonical algorithm, serving as the primary baseline.
• FedProx: An algorithm incorporating a proximal term to handle system and statistical heterogeneity.
• Role: These implementations ensure the benchmark is not just a dataset repository but a complete experimental suite. They demonstrate how to correctly handle client-side training, server-side aggregation, and evaluation within the framework.

Non-Independent and Identically Distributed (Non-IID) data is the core statistical challenge that federated learning benchmarks like LEAF are designed to model. It describes a scenario where data samples across different clients are not drawn from the same underlying probability distribution. This violates the standard assumption of centralized machine learning and leads to:

• Label distribution skew: Some clients have many examples of one class, while others have few.
• Feature distribution skew: The same feature (e.g., pixel brightness) has different statistical properties across clients.
• Quantity skew: Vastly different amounts of data per client. LEAF's data partitioning tools, such as Dirichlet sampling, create realistic, reproducible Non-IID splits for controlled experimentation.

Statistical heterogeneity is the umbrella term for the variations in data distributions across federated clients, of which Non-IID is a key property. Benchmarks must accurately capture this to be useful. It encompasses:

• Concept shift: The relationship between features and labels (P(Y|X)) differs.
• Covariate shift: The distribution of input features (P(X)) differs.
• Prior probability shift: The distribution of labels (P(Y)) differs. LEAF provides datasets where this heterogeneity is inherent (e.g., Reddit comments per user) or synthetically induced, allowing researchers to test algorithm robustness against these specific distributional shifts.

Dirichlet Distribution Sampling is the canonical method used by LEAF and other benchmarks to synthetically generate controlled, reproducible Non-IID data partitions. A concentration parameter α (alpha) controls the degree of heterogeneity:

• α → ∞: Data approaches an IID, uniform split across clients.
• α → 0: Data becomes extremely heterogeneous, with each client potentially holding data from only one or a few classes. This method provides a quantitative knob for researchers to systematically stress-test algorithms. For example, testing an algorithm at α=0.1 simulates a highly skewed, challenging real-world scenario, while α=100 simulates a near-homogeneous environment.

Federated Evaluation is the critical process of assessing a trained model's performance across the diverse test sets held by each client in a benchmark. LEAF facilitates this by providing client-specific test data. Key evaluation paradigms include:

• Global Model Test: Evaluating the single aggregated model on each client's local test set, revealing performance disparities.
• Personalized Model Test: Evaluating client-fine-tuned or personalized models on their local data.
• Cross-Client Generalization: Testing a model trained on one set of clients on the data from a held-out set of clients. This moves beyond a single accuracy number to a distribution of accuracies, highlighting fairness and robustness issues caused by Non-IID data.

Personalized Federated Learning is a primary solution direction for the performance degradation caused by Non-IID data, which LEAF benchmarks are designed to expose. Instead of a one-size-fits-all global model, PFL techniques produce models tailored to individual clients. LEAF's user-partitioned datasets (like FEMNIST) are ideal for evaluating PFL methods such as:

• Local Fine-Tuning: Taking the global model and adapting it locally.
• Multi-Task Learning: Framing each client's problem as a related task.
• Model Interpolation: Learning a weighted combination of global and local models (e.g., Ditto). Benchmark performance is measured by the average accuracy of each client's personalized model on its own data, a metric directly supported by LEAF's structure.