PowerSGD

PowerSGD decomposes the gradient matrix G into a compact product G ≈ P Q^T, where P and Q are tall, thin matrices of rank r. Instead of transmitting the full dense gradient, only the factor matrices are exchanged. This reduces the communication payload from O(mn) to O(r(m+n)), where r is typically 1–4, yielding compression ratios exceeding 100x for large layers.

The algorithm uses a warm-started power iteration method to compute the dominant left and right singular vectors of the gradient matrix without performing a full SVD. Key properties:

• One or two power iterations suffice for accurate approximation
• Warm-starting with the previous round's subspace vectors accelerates convergence
• Computational overhead is O(rmn) per iteration, negligible compared to the forward-backward pass
• Avoids the O(min(m,n)mn) cost of exact truncated SVD

PowerSGD integrates error feedback (also called memory or gradient residual) to preserve model accuracy under aggressive compression. The compression error e_t = G_t - compress(G_t) is stored locally and added to the gradient in the next step before compression. This prevents the accumulation of compression bias and ensures convergence to a stationary point, matching the theoretical guarantees of uncompressed SGD under standard assumptions.

PowerSGD can be applied directly to the momentum term rather than the raw stochastic gradient. By compressing the smoother momentum signal instead of the noisy instantaneous gradient, the algorithm achieves:

• Higher effective compression ratios for the same accuracy
• Reduced variance in the compressed update direction
• Compatibility with standard optimizers like SGD with momentum and Adam
• Seamless integration with existing training pipelines without modifying the optimizer internals

In federated settings, PowerSGD compresses both the downlink broadcast (global model to clients) and the uplink aggregation (client updates to server). This bidirectional approach:

• Reduces server-to-client bandwidth by compressing the full model broadcast
• Reduces client-to-server bandwidth by compressing local model deltas
• Is particularly effective for cross-silo FL with large models where both directions are bandwidth-constrained
• Maintains compatibility with secure aggregation protocols like SecAgg

PowerSGD exploits temporal coherence in gradient subspaces across training iterations. The factorization subspace from round t-1 is used to initialize the power iteration at round t, dramatically reducing the number of iterations needed. Empirical studies show:

• 1 power iteration with warm-starting matches the accuracy of 2–4 iterations from scratch
• Subspace overlap between consecutive steps often exceeds 0.9 cosine similarity
• This temporal stability is a key enabler for practical deployment at scale

A family of techniques that reduce communication overhead by applying lossy transformations to gradient vectors. PowerSGD is a specific instance of this class, using low-rank matrix factorization rather than sparsification or quantization. The core trade-off is accepting a controlled amount of information loss in exchange for significant bandwidth savings, with error feedback mechanisms preserving convergence guarantees.

A critical convergence-preserving mechanism that prevents model divergence under aggressive compression. When PowerSGD approximates a gradient matrix, the compression error—the difference between the true gradient and its low-rank reconstruction—is stored locally and added to the gradient in the next iteration. This ensures no information is permanently discarded, only delayed, maintaining theoretical convergence guarantees.

An alternative compression paradigm that transmits only the top-k gradient elements with the largest absolute magnitudes, zeroing out the rest. Unlike PowerSGD's low-rank approach, sparsification preserves the original coordinate basis but discards small-magnitude updates. Deep Gradient Compression (DGC) combines this with momentum correction and error feedback to achieve over 99% compression.

Maps high-precision 32-bit floating-point gradient values to lower bit-width representations such as 8-bit integers or even binary values. SignSGD represents the extreme case, transmitting only the sign of each gradient coordinate. PowerSGD is orthogonal to quantization—the factorized matrices it produces can themselves be quantized for additional compression, creating a compound compression pipeline.

The foundational federated learning algorithm where clients perform multiple local SGD steps before sending model updates to the server. PowerSGD integrates directly into the FedAvg workflow by compressing the model delta—the difference between the locally trained model and the received global model—before transmission. This dramatically reduces the upload bandwidth required per communication round.

A federated optimization algorithm that corrects for client drift caused by heterogeneous data distributions using control variates. When combined with PowerSGD, the control variate updates themselves can be compressed using the same low-rank factorization, ensuring that both the model delta and the correction terms benefit from reduced communication overhead without sacrificing convergence speed.