Gradient Compression

Reduces the numerical precision of gradient values from high-precision floats to low-bit representations. 32-bit floating-point gradients are mapped to 8-bit integers or even 1-bit signs, achieving up to 32× communication reduction. Common schemes include stochastic rounding to preserve statistical unbiasedness and QSGD (Quantized SGD) which provides convergence guarantees. The trade-off is between compression ratio and gradient fidelity—aggressive quantization introduces noise that can slow convergence but rarely prevents it entirely.

Transmits only the most significant gradient elements while zeroing out the rest. Top-k sparsification retains the largest-magnitude gradients, often keeping just 0.1% to 1% of elements. Random sparsification samples elements probabilistically based on magnitude. The key insight: most gradient elements are near-zero and contribute negligibly to convergence. Sparsification achieves 100× to 1000× compression ratios but requires error feedback mechanisms—accumulating residual errors locally and adding them back in subsequent rounds—to maintain model accuracy.

A critical technique that prevents accuracy degradation from aggressive compression. When gradients are compressed, the compression error (difference between original and compressed gradient) is stored locally in a residual accumulator. This residual is added to the next round's gradient before compression, ensuring no information is permanently lost. Error feedback guarantees convergence to the same optimum as uncompressed training, even with 99.9% compression ratios. Without it, models trained with high sparsification rates exhibit significant accuracy drops.

An extreme form of compression where entire gradient tensors are selectively not transmitted based on staleness or low magnitude. AdaComp dynamically adjusts compression rates per-layer based on gradient variance. Layers with consistently small updates can be dropped entirely for multiple rounds. This technique is particularly effective in cross-device federated learning where upload bandwidth is severely constrained. Combined with stale gradient reuse, the server can approximate missing updates using cached historical values.

Decomposes gradient matrices into compact low-rank representations before transmission. Using Singular Value Decomposition (SVD), a gradient matrix is factorized into smaller matrices that capture the dominant update directions. PowerSGD applies this to distributed training, transmitting only the top singular vectors. Compression ratios of 10× to 50× are achievable while preserving the principal optimization directions. This method is particularly effective for large fully-connected layers and attention mechanisms where gradient matrices exhibit strong low-rank structure.

Dynamically adjusts compression aggressiveness based on training phase and network conditions. Early training rounds tolerate aggressive compression (high noise, low precision) while later fine-tuning phases require conservative compression to preserve subtle gradient signals. Bandwidth-aware schedulers monitor available throughput and adjust quantization bits or sparsification ratios in real-time. Techniques include warm-up compression (gradually increasing compression) and layer-wise adaptive rates that apply different compression levels to convolutional versus classification layers.

Reduces the numerical precision of gradient tensors from high-bit representations to low-bit integers. Common schemes include:

• 8-bit quantization: Maps 32-bit floats to 8-bit integers, achieving 4× compression with negligible accuracy loss.
• 1-bit SGD: Transmits only the sign of each gradient element, reducing bandwidth by 32× while relying on error feedback to maintain convergence.
• Stochastic rounding: Probabilistically rounds values to prevent systematic bias accumulation across training rounds.

Transmits only a subset of gradient elements with the largest magnitudes, zeroing out the remainder. Key approaches:

• Top-k sparsification: Selects the k largest gradient values by absolute magnitude, typically retaining only 0.1%–1% of elements.
• Random sparsification: Samples gradient elements proportionally to their magnitude, providing unbiased estimates.
• Gradient dropping: Prunes entire gradient tensors below a threshold, skipping transmission entirely for marginal updates.

A compensation mechanism that accumulates compression residuals locally and adds them back into subsequent gradient updates before compression. This prevents information loss from aggressive quantization or sparsification by ensuring that discarded gradient components are eventually transmitted. Critical for maintaining model convergence when compression ratios exceed 100×.

Decomposes gradient matrices into compact factorized forms using singular value decomposition (SVD) or random projections. Instead of transmitting a full d×d matrix, the client sends two smaller matrices whose product approximates the original. Particularly effective for large fully-connected layers where gradients exhibit low-rank structure.

A selective omission strategy where entire gradient tensors are not transmitted if their norm falls below an adaptive threshold. Unlike sparsification, which zeroes individual elements, gradient dropping skips whole layers or update vectors. Often combined with staleness bounds to ensure no client's contributions are ignored indefinitely.