Client Drift

The most fundamental driver. In federated learning, each client's local dataset is drawn from a unique, non-identically and independently distributed (non-IID) data source. This creates heterogeneous local objectives that conflict with the single global objective the server aims to learn.

• Example: Smartphone keyboards where user A writes technical reports and user B sends casual messages. Their local data distributions over vocabulary and style are vastly different.
• Consequence: Local Stochastic Gradient Descent steps on client A's data pull the model in a different direction than steps on client B's data, causing their models to drift apart.

A core technique for communication efficiency that directly exacerbates drift. To reduce communication frequency, clients perform multiple passes (epochs) over their local data before sending an update.

• Mechanism: Each additional local epoch allows the client model to overfit or 'specialize' further to its local non-IID data distribution, moving it farther from the global starting point.
• Trade-off: While increasing local epochs (E) reduces communication rounds, it linearly increases the magnitude of client drift, creating a direct tension between efficiency and model consistency.

The local optimizer's behavior on a small, skewed dataset introduces bias. Stochastic Gradient Descent (SGD) on a non-representative sample does not produce an unbiased estimate of the global gradient.

• Gradient Variance: The variance of the client's stochastic gradients is high relative to the global batch gradient. Multiple local steps amplify this variance as bias.
• Objective Mismatch: The client effectively solves min F_k(w) (its local loss) rather than min F(w) (the global loss). The minimizers of these two functions are different under non-IID data, creating inherent drift.

In each communication round, only a subset of all clients is selected for training. This stochastic sampling means the global model is updated based on a biased sample of the total data distribution at each step.

• Compounding Effect: The server aggregates updates only from the participating clients, whose drift may be correlated (e.g., if selection is based on geography or device type). The global model then drifts toward the aggregate of this subset.
• Delayed Correction: Non-participating clients receive a global model that has drifted away from their data distribution, making their next local training step start from a suboptimal point, which can increase their subsequent drift magnitude.

Variations in client hardware and connectivity create operational drift. Clients have different computational capabilities, training times, and likelihoods of dropping out.

• Stragglers & Dropouts: Slow clients may perform many more local steps than intended if the server uses a timeout-based aggregation. Dropped clients fail to communicate their drift, leaving the global model uninformed by their data distribution.
• Resource-Constrained Optimization: Clients with limited memory may use smaller batch sizes, increasing gradient variance and local optimization bias, thereby increasing drift.

In standard Federated Averaging (FedAvg), clients train in isolation with no real-time anchor to the global state. The only corrective signal is the infrequent download of the global model.

• Problem: During local training, the client has no information about the updates being computed by other clients concurrently. It cannot correct its trajectory to align with the emerging global direction.
• Contrast with Centralized SGD: In centralized training, every gradient step is immediately averaged into the model, providing a continuous, synchronized reference point that prevents this type of decentralized divergence.

Non-IID (Non-Independent and Identically Distributed) data across clients is the root statistical cause of client drift, making local objectives diverge from the global objective.

• Types of Heterogeneity:
  • Label Distribution Skew: Different class frequencies per client (e.g., one user's photos are mostly cats, another's are mostly dogs).
  • Feature Distribution Skew: Same label, but different features (e.g., cats photographed indoors vs. outdoors).
  • Quantity Skew: Vastly different amounts of data per client.
• Impact on Training: Under non-IID data, the local gradient ∇F_k(w) on client k is a biased estimator of the true global gradient ∇f(w). Multiple local SGD steps amplify this bias, causing drift.
• System Challenge: Communication-efficient schemes that reduce aggregation frequency (e.g., many local epochs) directly exacerbate the negative effects of non-IID data.

Partial participation is a system-level constraint where only a subset of clients is active in each federated round. It is a primary reason for using many local steps, which in turn induces client drift.

• System Drivers: Client availability, limited server bandwidth, and energy constraints prevent all clients from communicating every round.
• Relation to Drift: To make progress with only a subset of clients, algorithms typically have each participant perform multiple local SGD steps. This deep local exploration is necessary but is the direct technical action that causes models to drift from the global state.
• Algorithmic Trade-off: Communication-efficient methods must balance the selection frequency (how often a client participates) with the local computation depth (how many steps it takes) to manage the drift-communication trade-off.