Communication Compression

Quantization reduces the precision of numerical values (e.g., gradients or model weights) from 32-bit floating-point to lower-bit representations like 8-bit integers. This directly shrinks the payload size for transmission.

• Uniform Quantization: Maps values to evenly spaced levels within a defined range.
• Non-Uniform Quantization: Uses levels with non-even spacing, often providing better accuracy for skewed value distributions.
• Post-Training Quantization (PTQ): Applied after training is complete.
• Quantization-Aware Training (QAT): Simulates quantization during training to improve final model accuracy.

A common technique is stochastic quantization, which adds controlled randomness to the rounding process to preserve the statistical expectation of the update, preventing bias.

Sparsification transmits only a subset of the most significant values from a model update (e.g., gradients), setting all others to zero. This exploits the inherent redundancy in neural network updates.

• Top-k Sparsification: Selects only the k elements with the largest absolute magnitude.
• Random Sparsification: Selects elements randomly with a fixed probability.
• Threshold-based Sparsification: Transmits only elements whose magnitude exceeds a defined threshold.

To ensure convergence, the expectation of the sparsified update must equal the original dense update. This is often achieved with error accumulation, where the omitted values from one round are added to the local gradient in the next round before sparsification.

Subsampling reduces communication volume by transmitting updates from only a strategically selected subset of clients in each federated learning round. This is distinct from compressing an individual client's update.

• Random Subsampling: Clients are selected uniformly at random.
• Importance Sampling: Clients are selected with probability weighted by metrics like dataset size or update magnitude to accelerate convergence.
• Resource-Aware Selection: Selects clients based on system criteria like available bandwidth, battery level, or computational power.

This technique directly reduces the total number of messages the server must aggregate per round, which is critical in cross-device federated learning with millions of potential clients.

Gradient compression is an umbrella term for techniques applied specifically to the gradients during the federated averaging process. It often combines sparsification and quantization.

• Deep Gradient Compression (DGC): Employs momentum correction, local gradient clipping, and warm-up training to achieve extreme compression ratios (up to 600x) without loss of accuracy.
• SignSGD: Transmits only the sign (+1 or -1) of each gradient element, reducing each value to a single bit. The server aggregates by majority vote.
• TernGrad: Quantizes each gradient element to one of three values: {-1, 0, +1}.

These methods must be paired with techniques like error feedback to maintain convergence, where the compression error is stored locally and added to the next gradient computation.

Instead of transmitting a full, dense model update, structured update methods restrict the update to have a pre-defined, low-rank or sparse structure that is inherently more compressible.

• Low-Rank Updates: Constrains the update matrix to be represented as the product of two smaller matrices, drastically reducing the number of transmitted parameters.
• Random Masking: Applies a fixed, random binary mask known to both client and server. Only the unmasked parameters are trained and transmitted.
• Delta Encoding: Transmits only the difference (delta) between the current model and a reference model, which is often more compressible than the absolute weights.

This approach is a form of model compression applied dynamically during the training process rather than as a post-processing step.

Communication compression in federated learning is predominantly lossy, meaning the original update cannot be perfectly reconstructed. The goal is to preserve the statistical properties necessary for convergence, not bit-for-bit fidelity.

Key Trade-offs:

• Compression Ratio vs. Model Utility: Higher compression (e.g., 1-bit quantization) increases bandwidth savings but can slow convergence or reduce final accuracy.
• Compression vs. Privacy: Some compression techniques, like top-k sparsification, can inadvertently leak information about the most salient features in a client's local dataset.
• Compute vs. Communicate: Compression adds local computational overhead (for sparsification, quantization) to save network bandwidth—a favorable trade-off in most edge scenarios.

Lossless compression (e.g., gzip) is sometimes applied as a final step after lossy techniques, but its gains are typically marginal compared to the primary lossy methods.

This scenario involves training a global model across millions of mobile or IoT devices, where bandwidth is severely limited and client availability is intermittent. Communication compression is non-negotiable here.

• Key Challenge: Sending full-precision model updates (often 100s of MB) over cellular or Wi-Fi is prohibitively expensive and slow.
• Solution: Aggressive quantization (e.g., to 1-8 bits) and gradient sparsification (sending only the top 1% of values by magnitude) reduce payload size by 100-1000x.
• Example: Training a next-word prediction model on smartphones without draining user data plans.

Secure Aggregation protocols, which sum client updates before the server can decrypt them, are computationally intensive. Compression reduces this overhead.

• Mechanism: When clients send compressed (e.g., quantized and sparsified) updates, the cryptographic operations performed by the aggregation server (like masking and unmasking) act on much smaller vectors.
• Benefit: This significantly lowers the communication and computation cost of the privacy layer, making differentially private federated learning scalable.
• Trade-off: The noise added for differential privacy must be calibrated in conjunction with compression error.

Communication compression principles extend beyond model training to Federated Analytics—computing aggregate statistics over decentralized data.

• Application: Estimating the histogram of app usage times across millions of devices or finding the most common software versions.
• Technique: Sketching algorithms (like Count-Min Sketch) and quantized aggregation allow devices to send a compact, lossy representation of their local statistics.
• Result: The central server obtains accurate global metrics while each device transmits only a few kilobytes, preserving user privacy and network resources.

The overarching decentralized training paradigm where communication compression is applied. A global model is trained collaboratively across many client devices (e.g., smartphones, IoT sensors) without exchanging raw data. Clients compute updates on local data and send only the compressed model updates (gradients or weights) to a central server for aggregation.

• Core Challenge: Communication bandwidth is the primary bottleneck.
• Key Scenarios: Cross-Device FL (millions of mobile devices) and Cross-Silo FL (a few organizations like hospitals).

A mathematical framework for quantifying and bounding privacy loss. In federated learning, it is often achieved by adding calibrated noise to model updates before they leave the device. Communication compression interacts with DP: techniques like gradient clipping (to bound sensitivity) are a prerequisite for DP and also reduce the range of values, which can benefit subsequent compression steps like quantization.

• Key Parameter: The privacy budget (epsilon ε) controls the trade-off between privacy and model utility.