Communication Efficiency

A family of lossy compression techniques applied to model updates before transmission. By reducing the precision or volume of gradient tensors, gradient compression trades a marginal amount of statistical accuracy for significant reductions in network I/O.

• Quantization: Maps 32-bit floating-point gradients to lower bit-width integers (e.g., 8-bit or 1-bit).
• Sparsification: Transmits only the top-k gradient elements with the largest magnitudes, zeroing out the rest.
• Error Feedback: Accumulates compression residuals locally to prevent staleness and maintain convergence.

The process of constraining a continuous set of values to a discrete set, drastically shrinking the message size of federated updates. Stochastic quantization maps high-precision weights to low-precision buckets with probabilistic rounding to maintain unbiased estimates.

• QAT (Quantization-Aware Training): Simulates quantization noise during local training to improve robustness.
• Binary/Ternary Quantization: Reduces gradients to 1-bit or 2-bit representations for extreme compression ratios.

A communication strategy where only a small fraction of the gradient vector is transmitted per round. Top-k sparsification selects the largest magnitude values, while random sparsification provides unbiased estimates.

• Gradient Dropping: Zeroes out low-magnitude updates that contribute minimally to convergence.
• Warm-up Scheduling: Gradually increases sparsity ratios during early training to stabilize optimization.
• Deep Gradient Compression: Combines momentum correction, local gradient clipping, and momentum factor masking.

A model compression paradigm where a compact student model is trained to mimic the soft output distributions of a larger teacher model. In federated settings, this enables heterogeneous model architectures across clients.

• Co-Distillation: Clients exchange logits rather than model weights, decoupling communication from model size.
• Ensemble Distillation: A central server distills an ensemble of client teacher models into a global student.
• Federated Distillation: Clients share class scores on a public proxy dataset instead of private gradients.

The foundational communication-efficient algorithm that reduces the frequency of synchronization. Clients perform multiple steps of local stochastic gradient descent (SGD) before sending updates to the server for averaging.

• Communication Rounds: Drastically reduced compared to vanilla FedSGD by increasing local computation per round.
• Hyperparameters: Local epochs (E) and batch size (B) control the computation-to-communication ratio.
• Convergence Trade-off: Excessive local training on non-IID data can cause client drift, degrading global accuracy.

Techniques that impose mathematical structure on gradient updates to enable compact representation. Low-rank approximation factorizes weight matrices, while count sketches compress high-dimensional vectors into fixed-size hash tables.

• SVD Compression: Decomposes weight updates into singular vectors and values for efficient transmission.
• Randomized Sketching: Uses Johnson-Lindenstrauss transforms to project gradients into lower dimensions.
• FetchSGD: Applies Count Sketch data structures to compress gradients with provable error bounds.

A model compression technique that reduces the numerical precision of weights or gradients to decrease communication overhead. Common schemes map 32-bit floating-point values to lower bit-width representations.

• Stochastic Quantization: Probabilistic rounding provides unbiased estimates of original values.
• Ternary Gradients: Restricts updates to {-1, 0, +1} for extreme compression.
• QSGD (Quantized SGD): Trades variance for communication savings with configurable bit-widths. Quantization is often paired with error feedback mechanisms to prevent accuracy degradation from accumulating compression errors across rounds.

A gradient compression method that transmits only a subset of significant gradient elements while zeroing out the rest. This exploits the empirical observation that many gradient components are near-zero and contribute minimally to convergence.

• Top-k Sparsification: Sends only the k largest gradient elements by absolute magnitude.
• Random Sparsification: Randomly selects a fraction of elements, preserving unbiasedness.
• Gradient Dropping: Drops entire layers or blocks below a threshold. Warm-up and gradient residual accumulation are critical techniques that carry forward the error from dropped elements to subsequent rounds, ensuring eventual convergence.

A multi-tier architecture that introduces intermediate edge aggregators between clients and the central cloud server. This topology reduces wide-area network traffic by performing partial model averaging closer to the data sources.

• Edge Aggregation: Base stations or regional nodes aggregate local updates before forwarding a single consolidated update upstream.
• Cloud Synchronization: The central server only communicates with a handful of edge nodes, not millions of devices. This dramatically reduces latency and backbone bandwidth consumption in large-scale cross-device deployments, such as smartphone keyboard prediction.

A model compression technique where a smaller student model is trained to replicate the soft output distributions of a larger teacher model. In federated settings, this serves as a communication-efficient alternative to weight averaging.

• Co-Distillation: Clients exchange logits on a public proxy dataset instead of model weights.
• Ensemble Distillation: A central student learns from the aggregated predictions of multiple teacher clients. This approach decouples model architecture from communication payload size, as only prediction vectors—not full weight tensors—are transmitted, enabling heterogeneous client models.