Federated Synchronous Training

The central parameter server acts as a strict coordinator. In each federated communication round, it selects a cohort of clients, dispatches the current global model, and then blocks all computation until every selected client has returned its local gradient update. Only after the final update arrives does the server compute the weighted average—typically via Federated Averaging (FedAvg)—and broadcast the new global model. This barrier guarantees that all contributing updates are derived from the same initial model state, preventing stale gradient contamination.

The synchronous protocol is fundamentally bottlenecked by the slowest participant, known as the straggler. A single client with limited compute, poor network connectivity, or a large local dataset can delay the entire round. To mitigate this, implementations often employ federated straggler mitigation techniques:

• Timeout-based exclusion: Drop clients that fail to respond within a predefined deadline.
• Over-selection: Select more clients than strictly needed, accepting the first k responses.
• Federated client dropout is treated as a non-event; the round proceeds with the remaining clients, though this can introduce bias if dropouts are systematic.

Synchronous aggregation provides strong theoretical convergence properties under standard optimization assumptions. Because the global update is a linear combination of gradients computed from an identical starting point, the process closely approximates centralized mini-batch SGD. This mathematical tractability makes synchronous training the default choice for cross-silo federated learning in healthcare, where a small number of reliable hospital nodes collaborate. The predictable convergence behavior simplifies hyperparameter tuning and model debugging compared to asynchronous alternatives.

While computationally deterministic, synchronous rounds incur significant wall-clock inefficiency. Fast clients remain idle while waiting for stragglers, wasting local compute resources. The protocol also generates synchronized traffic bursts: all clients receive the model simultaneously and all return updates near the round deadline, creating network congestion. Communication-efficient federated learning techniques—such as gradient compression and quantization—are often layered on top of synchronous protocols to reduce this bandwidth pressure without breaking the barrier semantics.

The synchronous barrier creates a natural aggregation point that simplifies the integration of cryptographic privacy and security protocols. Federated secure aggregation—where the server computes only the sum of client updates without inspecting individual contributions—is designed for synchronous rounds where all inputs are collected before computation. Similarly, Byzantine fault tolerance mechanisms can analyze the distribution of updates at the barrier to detect and exclude malicious or corrupted contributions before they poison the global model.

In federated asynchronous training, the server updates the global model immediately upon receiving any single client's update, eliminating the straggler bottleneck entirely. However, this introduces staleness: a slow client's update may be computed from an outdated model version, potentially degrading convergence or causing instability. Synchronous training trades raw throughput for mathematical cleanliness. The choice between the two is a fundamental architectural decision driven by client reliability, network heterogeneity, and the tolerance for non-deterministic convergence behavior.

The atomic unit of synchronous training. Each round consists of four distinct phases:

1. Client Selection: The server chooses a subset of available nodes
2. Broadcast: The current global model is distributed to selected clients
3. Local Training: Each client performs E epochs of SGD on its private data
4. Aggregation: The server collects and averages all returned updates Round latency is dominated by the slowest client's computation and communication time.

A failure mode where selected clients fail to return updates within the round deadline due to connectivity loss, resource preemption, or device unavailability. In synchronous training, excessive dropout rates stall convergence. Production systems implement graceful degradation—the server proceeds with a quorum of responses rather than waiting indefinitely. Typical healthcare deployments target a 90%+ response rate per round to maintain statistical validity.