Communication Efficiency

The primary key performance indicator for communication-efficient methods, defined as the ratio of the original size of the gradient or model update tensor to its size after applying a compression algorithm.

• Formula: Original Size / Compressed Size
• Deep Gradient Compression (DGC) can achieve ratios exceeding 600x without significant accuracy loss.
• A higher ratio indicates greater bandwidth savings but must be balanced against the reconstruction error introduced.

A hard constraint on the total number of bits or bytes that can be transmitted per client per round or over the entire training run. This budget serves as the fundamental design parameter for benchmarking communication-efficient algorithms.

• Fixed Budget: A strict cap, e.g., 10 MB per round.
• Variable Budget: Adjusts based on network conditions or convergence phase.
• Algorithms like SignSGD operate under an extreme 1-bit budget, transmitting only the sign of each gradient coordinate.

The fundamental optimization curve plotting model accuracy (or loss) against the total volume of data transmitted. The goal is to shift this curve upward and to the left.

• Pareto Efficiency: A protocol is optimal if no other protocol achieves higher accuracy for the same communication cost.
• SCAFFOLD and FedProx improve this trade-off by reducing the number of communication rounds required to reach a target accuracy, directly lowering total bytes transmitted.

The mathematical difference between the original uncompressed gradient and the version reconstructed after decompression on the receiver side. This error is the direct consequence of lossy compression.

• Error Feedback mechanisms accumulate this error and add it back to the gradient before the next compression step, preserving convergence.
• PowerSGD bounds this error by using a low-rank matrix approximation with proven error guarantees.

The end-to-end latency of a single federated training round, encompassing local computation, gradient compression, network transmission, server aggregation, and model broadcast.

• Straggler Mitigation techniques like deadline-based aggregation directly reduce this metric.
• Overlap Communication hides transmission latency by executing gradient exchange concurrently with the backward pass of subsequent layers.

A measure of how effectively the available network throughput is saturated during the communication phase, often expressed as a percentage of theoretical maximum bandwidth.

• Gradient Bucketing maximizes this metric by grouping gradients into large buffers, reducing the overhead of many small network calls.
• Ring All-Reduce achieves optimal bandwidth scaling by ensuring each node sends and receives only the data necessary for its segment of the reduction.

A family of techniques that reduce communication overhead by applying lossy transformations to gradient vectors before transmission. The goal is to trade a controlled amount of information fidelity for significant bandwidth savings.

• Gradient Sparsification: Transmits only the largest gradient elements
• Gradient Quantization: Reduces 32-bit floats to 8-bit integers or 1-bit signs
• Low-Rank Approximation: Factorizes gradients into compact matrices

A critical mechanism that preserves model convergence under aggressive gradient compression. The compression error from the current iteration is accumulated locally and added back to the gradient before the next compression step.

• Prevents information loss from compounding over time
• Essential for Deep Gradient Compression (DGC) to match uncompressed accuracy
• Maintains the correct optimization trajectory despite sparsification

The foundational communication-efficient algorithm where clients perform multiple local SGD steps before sending model updates to the server. This reduces communication rounds by increasing local computation.

• Server aggregates client models via weighted averaging
• Local epochs determine the computation-to-communication ratio
• Serves as the baseline for advanced algorithms like SCAFFOLD and FedProx

A systems-level optimization that hides gradient exchange latency by executing communication concurrently with computation. While the backward pass computes gradients for one layer, the gradients of a previous layer are transmitted.

• Requires gradient bucketing to batch small tensors
• Maximizes bandwidth utilization and reduces idle time
• Critical for scaling distributed training to hundreds of nodes

A training paradigm where the server updates the global model immediately upon receiving an update from any single client, eliminating the synchronization barrier.

• Mitigates stragglers: Slow clients do not block the round
• Introduces gradient staleness as a trade-off
• Well-suited for cross-device FL with heterogeneous hardware

A communication-efficient alternative to weight sharing where clients exchange only the soft labels or logits produced by their local models on a public or synthetically generated dataset.

• Payload size depends on output dimensions, not model size
• Preserves privacy by never transmitting model parameters
• Enables heterogeneous model architectures across clients