Gradient Compression

Transmits only a subset of gradient elements with the largest magnitudes, setting the rest to zero. This exploits the empirical observation that most gradients are near-zero and contribute minimally to convergence.

• Top-k sparsification: Only the k largest gradients by absolute value are sent per layer
• Random sparsification: Gradients are sampled probabilistically based on magnitude
• Threshold-based: Gradients below a fixed or adaptive threshold are dropped
• Typical sparsity ratios range from 99% to 99.9%, reducing communication by orders of magnitude
• Requires error feedback mechanisms to accumulate residual gradients and preserve convergence guarantees

Reduces the numerical precision of gradient values from 32-bit floating point to low-bit representations such as 8-bit integers or even 1-bit signs. This directly shrinks message sizes without selecting which gradients to send.

• 8-bit quantization: Maps FP32 gradients to INT8 with minimal accuracy loss
• 1-bit SGD: Transmits only the sign of each gradient, achieving 32x compression
• Stochastic rounding: Probabilistic rounding preserves statistical unbiasedness
• Vector quantization: Groups gradients into vectors and maps them to a learned codebook
• Quantization can be combined with sparsification for compound compression ratios exceeding 1000x

A critical mechanism that accumulates compression residuals locally and adds them back into subsequent gradient updates before compression. Without error feedback, aggressive sparsification causes divergence.

• Maintains a local error accumulator for each parameter
• Adds accumulated error to the current gradient before sparsification
• Guarantees convergence for unbiased compression operators
• Memory-efficient variants compress the error accumulator itself
• Essential for deep networks where layer-wise compression errors compound

Applies different compression ratios per layer based on gradient statistics rather than a uniform policy. Layers with larger gradients or higher sensitivity receive less aggressive compression.

• Gradient magnitude analysis: Layers with larger average gradients get higher precision
• Sensitivity profiling: Measures impact of compression on each layer's output
• Dynamic budget allocation: Redistributes a fixed communication budget across layers
• Particularly effective for transformer architectures where attention layers have different gradient distributions than feed-forward layers
• Can reduce communication by an additional 2-3x compared to uniform compression

A seminal technique combining momentum correction, local gradient clipping, momentum factor masking, and warm-up training to enable extreme sparsification rates without accuracy degradation.

• Momentum correction: Applies momentum on top of error-compensated sparse updates
• Local gradient clipping: Normalizes gradients before sparsification to prevent instability
• Momentum factor masking: Masks momentum updates for parameters not receiving gradient updates
• Warm-up training: Gradually increases sparsity ratio over initial training epochs
• Demonstrated 99.9% sparsity on large-scale tasks with no accuracy loss compared to dense training

Replaces full gradient matrices with low-rank approximations using power iteration to compute dominant singular vectors. This provides structured compression that preserves gradient geometry better than element-wise methods.

• Decomposes gradient matrices into two low-rank factors plus a residual
• Uses power iteration to efficiently approximate top singular vectors
• Achieves compression ratios proportional to matrix dimensions
• Particularly effective for fully-connected and embedding layers with natural low-rank structure
• Combines with error feedback for convergence guarantees
• Outperforms element-wise sparsification on tasks with large weight matrices

A compression technique that transmits only the most significant gradient components during distributed training. By sending the top-k largest gradients by magnitude and accumulating the remaining small updates locally, bandwidth reduction of 99% or more is achievable.

• Top-k sparsification: Selects gradients with the largest absolute values
• Random sparsification: Probabilistically samples gradients proportional to magnitude
• Warm-up strategies: Gradually increase sparsity ratio to maintain early convergence
• Error feedback: Tracks accumulated residuals to prevent information loss from dropped gradients

Reduces the numerical precision of gradient values from 32-bit floating point to 8-bit or even 1-bit representations before transmission. This dramatically shrinks message sizes without fundamentally altering the optimization trajectory.

• 8-bit quantization: Maps gradients to 256 discrete levels with minimal accuracy loss
• 1-bit SGD: Communicates only the sign of each gradient component
• Stochastic rounding: Probabilistically rounds values to maintain unbiased estimates
• Adaptive quantization: Dynamically adjusts quantization levels based on gradient distribution

A critical mechanism that prevents convergence degradation in aggressive compression schemes. When gradients are sparsified or quantized, the residual error from compression is accumulated locally and added to the next iteration's update.

• Memory: Maintains a local error accumulator per parameter
• Convergence guarantee: Ensures compressed SGD matches full-precision convergence rates
• Momentum correction: Adjusts for momentum-based optimizers like Adam
• Scale sensitivity: Requires careful tuning of learning rate relative to compression ratio

A compound technique combining momentum correction, local gradient clipping, momentum factor masking, and warm-up training to achieve 600x compression ratios without accuracy loss. Originally demonstrated on large-scale speech recognition and image classification tasks.

• Momentum correction: Applies momentum to accumulated rather than instantaneous gradients
• Local gradient clipping: Normalizes gradients before sparsification to prevent explosion
• Momentum factor masking: Only updates momentum for transmitted gradients
• Warm-up training: Uses less aggressive compression during initial epochs

A systems-level optimization that hides gradient communication latency behind computation. While compressed gradients from layer N are being transmitted, the backward pass continues computing gradients for layer N-1.

• Pipeline parallelism: Staggers computation and communication across layers
• Tensor partitioning: Splits large tensors to begin transmission earlier
• Wait-free backpropagation: Eliminates synchronization barriers during gradient exchange
• Hardware-aware scheduling: Leverages GPU Direct RDMA for direct GPU-to-GPU transfers