Local Epochs

A local epoch is one complete pass by a federated learning client over its entire local training dataset. The hyperparameter E specifies the number of such passes performed locally before the client sends its model update (or gradient) back to the server. This creates a fundamental trade-off:

• High E (e.g., E=10): More local computation per round, fewer communication rounds needed, but risk of client drift.
• Low E (e.g., E=1): Less local computation, more frequent communication, closer alignment with centralized SGD but higher communication cost. In the canonical Federated Averaging (FedAvg) algorithm, E > 1 is what differentiates it from the baseline FedSGD, where E=1.

The primary role of E is to manage the system's efficiency trade-off between on-device compute and network bandwidth.

• Increasing E amortizes the cost of each communication round over more local work. This is crucial in cross-device FL where millions of slow, intermittent mobile devices participate. It makes training feasible where frequent communication is prohibitive.
• Decreasing E reduces the client drift caused by overfitting to local, non-IID data. It keeps client updates more aligned with the global objective, often improving final model accuracy at the expense of more rounds. Engineers tune E based on the specific federated network's constraints: device compute power, network latency, and data heterogeneity.

The setting of E has profound effects when client data is non-independent and identically distributed (non-IID).

• High E on non-IID data: Clients perform many steps on their unique data distribution, causing their local models to diverge from the global optimum. This is the client drift problem, which can slow convergence or lead to a poor final global model.
• Mitigation Strategies: Algorithms like FedProx add a proximal term to the local loss function, penalizing updates that stray too far from the global model. SCAFFOLD uses control variates to correct local update directions. These methods allow for higher E values despite data heterogeneity.

The total number of local optimization steps a client takes per round is determined by both E and the **local batch size (B)`.

• Local Steps per Round = E * (Local Dataset Size / B) Therefore, increasing E linearly increases local computation. For a fixed dataset size, you can achieve a similar number of local steps by either increasing E or decreasing B. However, they have different effects:
• A larger B provides a more stable, lower-variance gradient estimate per step.
• A larger E means more steps, increasing the risk of overfitting to the local data. Tuning requires considering both hyperparameters jointly.

The value of E directly influences theoretical convergence guarantees and practical algorithm design.

• Convergence Rate: Under convex assumptions, increasing E can lead to faster convergence in terms of communication rounds, but may require more total local steps to reach a given accuracy.
• Algorithm Variants: FedNova explicitly normalizes client updates by their local number of steps (a function of E) to correct for the bias introduced when clients perform different amounts of work. Without this, a client with higher E would have an disproportionately large influence on the global update.
• Adaptive Tuning: Advanced FL orchestrators may dynamically adjust E per client or per round based on observed convergence metrics or client capabilities.

Selecting E is an empirical decision based on the FL deployment scenario:

• Cross-Silo FL (10-100 reliable institutional clients): Use moderate E (e.g., 2-5). Communication is less costly, so focus on mitigating drift.
• Cross-Device FL (1000s+ mobile/IoT devices): Use higher E (e.g., 5-20) to tolerate partial participation and high communication latency.
• Highly Non-IID Data: Start with a low E (1-2) and consider algorithms like FedProx if you need to increase it.
• Homogeneous (IID) Data: Higher E values are generally safe and improve communication efficiency. Benchmarking involves monitoring global validation accuracy vs. total communication rounds and vs. cumulative client compute time.

A communication round is the fundamental cycle of the federated averaging loop. It consists of three phases:

• Broadcast: The server sends the current global model to a selected subset of clients.
• Local Training: Each client trains the model on its local data for a specified number of local epochs.
• Aggregation: The server collects the client updates (model deltas) and aggregates them, typically via weighted averaging, to produce a new global model. The frequency of communication rounds is a primary determinant of total training time and bandwidth cost.

Client drift is a convergence-hindering phenomenon where local client models diverge from the global objective. It is primarily caused by:

• Statistical Heterogeneity (Non-IID Data): Clients optimize for their unique local data distributions.
• Excessive Local Epochs: Too many local epochs can cause clients to overfit to their local data, amplifying divergence. Algorithms like FedProx and SCAFFOLD are explicitly designed to mitigate client drift by adding constraints or correction terms to the local optimization objective.

Federated Stochastic Gradient Descent (FedSGD) is a foundational baseline algorithm. It is mathematically equivalent to Federated Averaging (FedAvg) configured with:

• Local Epochs = 1
• Local Batch Size = Entire Local Dataset In this configuration, each client performs exactly one gradient descent step per communication round. Comparing FedAvg (with E>1) to FedSGD reveals the communication-computation trade-off: more local computation reduces the required number of communication rounds.

Statistical heterogeneity, or non-IID data, is the defining characteristic of federated learning where data distributions vary significantly across clients. This variation directly influences the optimal setting for local epochs.

• High Heterogeneity: Fewer local epochs are often beneficial to prevent severe client drift.
• Low Heterogeneity: More local epochs can be used to efficiently reduce communication rounds. Managing this tension is central to federated optimization algorithm design.

The local model is the instance of the global neural network downloaded by a client at the start of a communication round. The lifecycle of the local model during a round is:

1. Initialization: Parameters are set from the current global model.
2. Training: The model is updated via stochastic gradient descent for E local epochs over the client's private dataset.
3. Update Calculation: The difference (delta) between the final local parameters and the initial parameters is computed. This local model is ephemeral; only the computed update is sent to the server.

FedProx is a federated optimization algorithm designed to handle statistical heterogeneity. It modifies the local client objective function by adding a proximal term: F_k(w) + (μ/2) * ||w - w^t||^2 where w^t is the global model and μ is a hyperparameter. This term:

• Constrains Local Updates: Acts as a regularizer, penalizing the local model for drifting too far from the global model.
• Mitigates Heterogeneity Impact: Allows for the use of more local epochs even on highly non-IID data by explicitly controlling divergence. It provides a more robust alternative to standard FedAvg under challenging data distributions.