Communication Budget

The communication budget defines an absolute upper limit on data volume exchanged between clients and the server. This constraint is typically specified in bytes per round per client or total bytes over the training run. Unlike soft targets, a hard budget forces algorithm designers to make explicit trade-offs between gradient fidelity and bandwidth consumption. For example, a budget of 10 MB per round on a 100 MB model forces at least 10:1 compression.

Communication budgets provide a standardized evaluation framework for comparing compression algorithms. Researchers report accuracy achieved under identical byte constraints rather than abstract compression ratios. This enables apples-to-apples comparisons between methods like gradient sparsification, quantization, and low-rank approximation. A typical benchmark might compare final model accuracy when all methods are limited to 1 GB total upload per client over the full training duration.

Intelligent allocation of the communication budget across training rounds and model layers can significantly improve convergence. Adaptive compression dynamically adjusts the compression ratio based on gradient signal-to-noise ratio. Layer-wise compression allocates more bytes to layers with higher gradient variance. Early training rounds often receive a larger share of the budget to establish a good initialization, while later rounds operate under tighter constraints.

The communication budget is enforced at the client node before transmission. The local training process must apply compression algorithms—such as gradient sparsification or quantization—to reduce the model delta to fit within the allocated byte limit. This client-side enforcement ensures that heterogeneous devices with varying uplink capacities can all participate without exceeding network constraints. Error feedback mechanisms preserve convergence despite aggressive compression.

Communication budgets directly interact with straggler mitigation strategies. A tight per-round budget reduces transmission time, which can help slow clients complete their uploads before the aggregation deadline. Conversely, asynchronous federated learning can relax the synchronization barrier, allowing clients with larger available bandwidth to transmit higher-fidelity updates while constrained clients send compressed versions. The budget ensures fairness across heterogeneous network conditions.

Information theory establishes fundamental limits on how much gradient information can be compressed without destroying convergence. The communication budget must respect these theoretical lower bounds. For convex optimization, the required bitrate scales with the dimension of the parameter space and the desired optimization accuracy. Practical budgets are designed to operate above these theoretical minima while achieving acceptable wall-clock training time.

A family of techniques that reduce communication overhead by applying lossy transformations to gradient vectors before transmission. The core trade-off: trading a controlled amount of information fidelity for significant bandwidth savings.

• Sparsification: Transmits only the top-k gradient elements by magnitude
• Quantization: Maps 32-bit floats to 8-bit integers or even 1-bit signs
• Low-rank approximation: Factorizes gradients into compact matrices (e.g., PowerSGD)

Compression ratios of 100x to 1000x are achievable without catastrophic accuracy loss when combined with error feedback mechanisms.

A convergence-preserving mechanism that prevents information loss from aggressive compression. The compression error from the current iteration is accumulated in a residual buffer and added back to the gradient before the next compression step.

• Ensures no gradient information is permanently discarded
• Critical for Deep Gradient Compression (DGC) to exceed 99% sparsification
• Prevents the momentum-correction mismatch that would otherwise cause divergence

Without error feedback, aggressive sparsification leads to stochastic noise that degrades final model accuracy.

The primary KPI for communication-efficient methods, defined as the ratio of the original gradient tensor size to its size after compression. A ratio of 100x means only 1% of the original bytes are transmitted.

• SignSGD achieves 32x by transmitting only 1-bit per coordinate
• DGC reaches 600x+ through sparsification and error accumulation
• PowerSGD provides variable ratios based on the chosen low-rank dimension

Higher ratios reduce bandwidth consumption but increase the risk of convergence slowdown if error feedback is insufficient.

The strategic process of choosing which clients participate in a federated round based on resource constraints and communication budgets. Not all clients can transmit updates within a given budget window.

• Bandwidth-aware scheduling: Prioritizes clients with higher available throughput
• Deadline-based selection: Excludes clients that cannot complete transmission in time
• Data quality weighting: Selects clients whose updates provide the most information gain per bit

Effective client selection maximizes the utility of every byte transmitted under a fixed communication budget.

A dynamic strategy that adjusts the compression ratio or quantization level in real-time based on current network conditions, model convergence stage, or gradient signal-to-noise ratio.

• Early training: Use aggressive compression when gradients are large and noisy
• Late convergence: Reduce compression to preserve fine-grained update information
• Network fluctuation: Tighten compression during congestion, relax during idle periods

This approach ensures the communication budget is spent where it yields the highest marginal improvement in model quality.

A communication-efficient alternative to weight sharing where clients exchange only soft labels or logits produced by their local models on a public or synthetically generated dataset.

• Eliminates the need to transmit large weight tensors entirely
• Payload size depends on the number of output classes, not model size
• Particularly effective for large models where weight transfer would exceed any practical budget

Federated distillation decouples communication cost from model complexity, making it ideal for tight communication budgets in heterogeneous networks.