Communication Round

The round begins with the central server broadcasting the current global model parameters to a selected cohort of clients. This phase establishes the common starting point for decentralized training. The selection can be random, stratified, or based on system criteria like device capability and connectivity. The broadcast must be efficient, often leveraging model compression techniques to minimize communication overhead, especially in bandwidth-constrained environments.

Each participating client downloads the global model and performs local stochastic gradient descent (SGD) on its private dataset. This is defined by key hyperparameters:

• Local Epochs (E): The number of complete passes over the local data.
• Local Batch Size (B): The size of mini-batches used for SGD.
• Learning Rate (η): The step size for local optimization. The client computes a model delta—the difference between its final local parameters and the initial global parameters. Training occurs entirely on-device, ensuring raw data never leaves the client.

Clients send their computed model updates (deltas) back to the server. This is often the primary communication bottleneck. To mitigate this, update compression techniques are frequently applied:

• Quantization: Reducing the numerical precision of the update tensors.
• Sparsification: Sending only the most significant gradient values.
• Gradient Clipping: Bounding the update norm to control magnitude and improve stability. This phase must also handle stragglers—slow clients that can delay the round—through mechanisms like deadline enforcement.

The server collects the client updates and aggregates them to form a new global model. The standard method is weighted averaging, where each client's update is weighted by the size of its local dataset. For enhanced privacy, secure aggregation protocols use cryptographic multi-party computation to combine updates without the server ever inspecting individual contributions. This step is crucial for maintaining the collaborative learning objective while providing a privacy safeguard.

The aggregated update is applied to the global model. While Federated Averaging (FedAvg) uses a simple weighted sum, advanced algorithms within the FedOpt framework apply adaptive server optimizers:

• FedAdam: Uses the Adam optimizer on the server.
• FedAvgM: Incorporates server-side momentum. This update produces the new global model version, which is then ready for broadcast in the next communication round. The cycle repeats until a convergence criterion is met.

After the global model is updated, the round is considered complete. The server typically evaluates the new model's performance on a held-out validation set to monitor convergence. Key metrics tracked include global accuracy, loss, and, in personalized settings, per-client performance. This evaluation phase informs decisions about continuing training, adjusting hyperparameters (like client participation rate), or deploying the model. It closes the loop on one discrete step of the federated training loop.

The aggregation protocol defines the server's waiting policy, creating a fundamental trade-off between consistency and latency.

• Synchronous Aggregation (e.g., classic FedAvg): The server waits for updates from all selected clients in a round. This ensures a consistent, stable global update but is vulnerable to stragglers—slow or offline clients that bottleneck the entire round.
• Asynchronous Aggregation: The server updates the global model immediately upon receiving any client's update. This eliminates waiting time and improves system throughput but can introduce staleness and convergence instability, as updates are applied to an older global state.

Determining which and how many clients train each round is a critical resource allocation problem. The client participation rate is the fraction of available devices selected.

• Random Sampling: The most common strategy, providing unbiased estimates of the population gradient. It must account for device availability and connectivity.
• Resource-Aware Selection: Prioritizes clients with sufficient battery, strong network connections, and high computational capability to complete local epochs reliably.
• Strategic Sampling: Advanced methods may select clients based on data diversity or update significance to improve learning efficiency, moving beyond simple randomness.

Real-world client data is rarely identically distributed, leading to client drift—where local models diverge from the global objective. Round design must mitigate this.

• Algorithmic Corrections: Algorithms like FedProx (adds a proximal term to local loss) and SCAFFOLD (uses control variates) are designed to correct local updates within a round, reducing variance and drift.
• Local Batch Normalization (FedBN): Keeping batch norm statistics local per client, rather than aggregating them, accounts for feature distribution shifts within the round.
• Personalized Layers: Methods like FedPer keep final model layers local, so the round only aggregates updates from the shared base layers, insulating the global model from extreme local variation.

Transmitting full model updates each round is often the system bottleneck. Efficiency techniques reduce bandwidth and cost.

• Update Compression: Techniques like quantization (reducing numerical precision), sparsification (sending only the largest gradient values), and subsampling dramatically shrink payload size.
• Local Epochs: Performing multiple local epochs of training before communicating amortizes the round's communication cost over more computation, a core principle of FedAvg.
• Gradient Clipping: Bounding the norm of updates before transmission stabilizes training, provides a form of differential privacy, and makes compression more effective.

System designers must plan the total number of rounds (the round budget) required for convergence, which depends on multiple interacting factors.

• Convergence Guarantees: Theoretical proofs for algorithms like FedAvg show convergence requires hundreds to thousands of rounds, depending on client participation rate, data heterogeneity, and learning rate.
• Local Epoch Trade-off: More local epochs per round reduce total rounds needed but increase the risk of client drift. This is a key hyperparameter to tune.
• Adaptive Server Optimizers: Using frameworks like FedOpt (e.g., FedAdam) allows the server to apply adaptive optimizers to aggregated updates, often reducing the total rounds to convergence compared to simple averaging.

Each round must be engineered to prevent data leakage and protect against malicious participants, even as updates are shared.

• Secure Aggregation: Cryptographic protocols ensure the server can compute the weighted average of client updates (the model delta) without inspecting any individual update, providing information-theoretic privacy within the round.
• Differential Privacy: Adding calibrated noise to client updates before transmission provides a rigorous, mathematical privacy guarantee for each participant's contribution to the round.
• Byzantine Robustness: Aggregation rules must be robust to adversarial clients sending poisoned updates. Techniques include median-based aggregation or anomaly detection on received updates.

The foundational algorithm that defines the standard communication round. In FedAvg, the server:

• Broadcasts the global model to selected clients.
• Clients perform local stochastic gradient descent for multiple local epochs.
• Clients send back their updated model parameters (or model deltas).
• The server performs a weighted average of these updates to form a new global model. This iterative loop is the canonical implementation of a communication round.

The number of complete passes a client makes over its local dataset within a single communication round. This is a key hyperparameter:

• Low values (e.g., E=1): Clients behave more like a distributed SGD node, sending frequent, noisy updates. This is sometimes called FedSGD.
• High values (e.g., E=5-10): Clients perform more local computation, reducing communication frequency but increasing the risk of client drift as models overfit to local, potentially non-IID data. Tuning this balances communication cost against convergence stability.

The fraction of the total client pool selected to participate in a given round. Due to partial client participation (a practical constraint), not all devices are available each round. Strategies include:

• Random uniform sampling: The standard approach.
• Resource-aware selection: Prioritizing clients with sufficient battery, compute, and bandwidth.
• Data-informed selection: Selecting clients whose data distribution may most benefit the global model. A low rate slows convergence but is often a system necessity.

Two fundamental protocols governing the timing of a communication round.

Synchronous Aggregation (the standard):

• The server waits for updates from all selected clients in the round.
• Prone to stragglers (slow clients) which bottleneck the entire round.

Asynchronous Aggregation:

• The server updates the global model immediately upon receiving any client update.
• Eliminates straggler problems but introduces gradient staleness and can destabilize convergence. Most production systems use synchronous rounds with straggler mitigation techniques like deadlines.

A major challenge arising within communication rounds on non-IID data. Client drift occurs when local client updates diverge from the global objective. Specialized algorithms modify the local training objective to counteract this:

• FedProx: Adds a proximal term to the local loss, penalizing updates that stray too far from the global model.
• SCAFFOLD: Uses control variates (correction terms) to reduce variance between client updates.
• FedNova: Normalizes client updates based on their local step counts before aggregation. These algorithms are embedded within the local training phase of the round.

The server-side computation that concludes a round. While weighted averaging (by dataset size) is standard, advanced methods exist:

• FedAvgM: Applies server-side momentum to the aggregated update for stabilization.
• FedOpt Framework: Uses adaptive optimizers (e.g., FedAdam) on the server to process aggregated gradients, rather than simple averaging.
• Secure Aggregation: Cryptographic protocols that sum client updates so the server only sees the aggregate, never individual contributions, enhancing privacy. The choice of aggregation directly impacts convergence speed and final model quality.