Communication Efficiency

A technique that transmits only the most significant gradient components during a model update, discarding near-zero values. Top-k sparsification retains only the largest-magnitude gradients, reducing communication payloads by 100x to 1000x without severely degrading model accuracy.

• Mechanism: A threshold or fixed ratio selects gradients for transmission; residuals are accumulated locally for future rounds.
• Benefit: Converts dense, high-dimensional updates into sparse vectors, directly lowering uplink bandwidth consumption.
• Trade-off: Aggressive sparsification can introduce variance, requiring warm-up phases or error feedback mechanisms.

The process of reducing the bit-precision of gradient values from standard 32-bit floating point (FP32) to low-bitwidth representations like 8-bit integers (INT8) or even 1-bit signs. This drastically shrinks the size of model updates transmitted from edge clients to the aggregation server.

• Stochastic Quantization: Probabilistic rounding preserves statistical properties of the original gradient distribution.
• QSGD (Quantized SGD): A proven scheme that provides convergence guarantees while achieving significant bandwidth savings.
• Extreme Compression: 1-bit quantization (signSGD) transmits only the sign of each gradient, enabling communication-efficient majority voting.

A hyperparameter controlling the number of local stochastic gradient descent (SGD) steps a client performs on its private dataset before communicating with the central server. Increasing local computation reduces the frequency of communication rounds.

• FedAvg Foundation: The core insight of Federated Averaging—more local epochs mean fewer global synchronization rounds.
• Communication Rounds: Reducing rounds from thousands to hundreds directly cuts total network traffic.
• Convergence Risk: Excessive local updates on non-IID data can cause client drift, where local models diverge from the global optimum.

Techniques applied to the downlink broadcast of the global model from the server to clients. While often less constrained than uplink, compressing the broadcast model reduces overall spectrum usage in wireless deployments.

• Weight Pruning: Removing redundant connections from the neural network before transmission.
• Knowledge Distillation: Transmitting a compact student model instead of the full teacher model.
• Federated Dropout: Broadcasting only a subnetwork of the global model, allowing heterogeneous clients with varying compute capabilities to participate.

A communication paradigm where the central server updates the global model immediately upon receiving an update from any single client, rather than waiting for a full cohort to respond. This eliminates straggler-induced latency and keeps the wireless channel continuously utilized.

• Staleness Control: Weighting updates inversely to their delay prevents outdated gradients from corrupting the model.
• Non-Blocking: Fast clients are never idle; slow clients contribute when ready.
• Challenge: Managing the temporal inconsistency of gradients in highly asynchronous settings requires robust optimization algorithms.

A physical-layer technique that exploits the superposition property of wireless multiple-access channels to compute the sum of simultaneously transmitted analog signals. Instead of decoding individual updates, the receiver directly obtains the aggregated gradient.

• AirComp: Aligns uncoded gradient transmission with the natural waveform summation of the channel.
• Latency: Aggregation latency becomes independent of the number of participating devices.
• Requirement: Demands strict time and frequency synchronization, as well as power control, across all transmitting clients.

Techniques that prevent slow or computationally constrained clients from delaying the entire training round. In wireless networks, channel fading and device heterogeneity create unpredictable latency. Strategies include:

• Timeout-based discarding: Ignore updates arriving after a deadline
• Asynchronous SGD: Update the global model immediately upon receiving any client's gradient
• Coded computation: Use error-correcting codes to reconstruct missing updates from faster clients

The scheduling strategy that determines which subset of available devices participate in each training round. Intelligent selection directly impacts communication efficiency by prioritizing clients with:

• Good channel conditions to minimize retransmissions
• Higher local data quality for faster convergence
• Sufficient battery and compute to complete training on time

Random selection remains the baseline, but utility-aware scheduling can reduce total rounds by 40%.

A distributed training paradigm where the neural network is partitioned between client and server. The client processes initial layers on raw private data and transmits only intermediate activations (smashed data) rather than model gradients. This dramatically reduces uplink communication when the cut layer is chosen appropriately.

• Activation size can be smaller than gradient size for deep networks
• Eliminates need for clients to compute full backward pass
• Particularly effective for cross-silo deployments with reliable links