Gradient Compression

Reduces the precision of gradient values from 32-bit floating point to low-bit representations like 8-bit integers or even 1-bit signs. QSGD (Quantized SGD) maps each gradient component to a discrete set of values, achieving up to 32× compression while preserving convergence guarantees through stochastic rounding. This trades a small amount of gradient fidelity for massive bandwidth savings, critical for wireless backhaul links in telecom RAN deployments.

Transmits only the most significant gradient elements, setting the rest to zero. Top-k sparsification selects the largest k% of gradient magnitudes per layer, while Random-k selects a random subset to preserve unbiased estimates. Combined with error feedback—accumulating residual gradients locally for future transmission—sparsification can achieve 100-1000× compression without degrading final model accuracy.

A critical mechanism that prevents model divergence when using aggressive compression. When gradients are sparsified or quantized, the compression error is not discarded but accumulated in a local residual buffer and added to the next round's gradients before compression. This ensures that even small but persistent gradient signals eventually get transmitted, preserving convergence guarantees that would otherwise be lost with naive compression.

An extreme compression scheme where clients transmit only the sign of each gradient element (±1), reducing communication to 1 bit per parameter. The server aggregates by taking the majority vote per coordinate. Despite its simplicity, SignSGD converges robustly in distributed settings and is inherently resilient to Byzantine faults, making it attractive for large-scale cross-device federated learning with unreliable participants.

Exploits the observation that gradient matrices often exhibit low-rank structure during training. Techniques like PowerSGD factorize the gradient tensor into two smaller matrices using power iteration, transmitting only the low-rank components. This achieves compression ratios proportional to the matrix dimensions, often outperforming sparsification for large fully-connected and convolutional layers in deep neural networks.

Dynamically adjusts the compression ratio based on training progress and network conditions. Early in training, when gradients are large and noisy, higher compression can be tolerated. Near convergence, when gradients are small and precise, compression is reduced. AdaQSGD and similar schemes monitor gradient variance and communication budgets to optimize the accuracy-bandwidth tradeoff in real time across heterogeneous federated clients.

The process of mapping continuous or high-precision gradient values to a smaller set of discrete levels, significantly reducing the bit-width required for transmission. Stochastic quantization adds random noise during mapping to maintain statistical unbiasedness.

• 8-bit to 1-bit: Common targets for deep learning gradients
• QSGD: A popular algorithm that trades precision for communication savings
• Error Feedback: Critical for compensating quantization noise over successive rounds

A technique that transmits only the most significant gradient elements, setting the majority to zero. Top-k sparsification retains only the largest-magnitude updates, while random sparsification selects elements probabilistically.

• Gradient Dropping: Sending only 0.1%–1% of non-zero values
• Warm-up Scheduling: Gradually increasing sparsity to stabilize early training
• Residual Accumulation: Storing non-transmitted gradients locally to prevent information loss

A memory mechanism that accumulates compression errors locally and re-injects them into subsequent optimization steps. This prevents the model from diverging when aggressive compression is applied.

• Error Accumulation: Stores the delta between true and compressed gradients
• Convergence Guarantee: Proves that compressed SGD converges at the same asymptotic rate as uncompressed
• Memory Overhead: Requires maintaining a residual tensor equal to the model size on each client

A bandwidth-saving strategy where a client selectively skips transmitting an entire gradient update if its magnitude falls below a threshold. This avoids wasting uplink capacity on negligible updates.

• Lazy Client Updates: Only significant model changes trigger communication
• Staleness Bounding: Limits how long a client can remain silent before forced sync
• Adaptive Thresholds: Dynamically adjusts the drop criterion based on training phase

A combined approach integrating momentum correction, local gradient clipping, momentum factor masking, and warm-up training to achieve over 600x compression on large models without accuracy degradation.

• Momentum Correction: Applies momentum on the uncompressed, accumulated gradient
• Layer-Adaptive Thresholds: Different sparsity ratios for convolutional vs. fully-connected layers
• Proven on ResNet & LSTM: Validated on large-scale benchmarks

A systems-level optimization that pipelines gradient compression with the backward pass computation. While layer N computes gradients, layer N-1's compressed gradients are already being transmitted.

• Wait-Free Backpropagation: Hides compression latency entirely
• CUDA Stream Parallelism: Uses separate GPU streams for compute and compression kernels
• Tensor Partitioning: Splits large tensors to balance compression workload across cores