Federated Gradient Compression

Transmits only a fraction of the most significant gradient elements during each communication round, discarding updates near zero.

• Top-k Sparsification: Sends only the largest k gradient values by magnitude, reducing data volume by 99%+ in practice.
• Random Sparsification: Randomly selects a subset of coordinates to transmit, preserving unbiased convergence properties.
• Warm-up Strategies: Gradually increases sparsity ratio during early training to avoid destabilizing convergence.

Clinical Example: A hospital training a diagnostic model sends only 0.1% of gradient updates per round, reducing a 500MB update to 500KB without degrading final accuracy.

Reduces the numerical precision of gradient values from 32-bit floating point to lower bit-width representations before transmission.

• 8-bit Quantization: Compresses each gradient element to a single byte, achieving 4x compression with negligible accuracy loss.
• 1-bit SignSGD: Transmits only the sign (positive/negative) of each gradient, achieving 32x compression.
• Stochastic Rounding: Probabilistically rounds values to quantization levels, preserving statistical properties and preventing systematic bias.

Key Insight: Modern federated quantization schemes can compress gradients to 2-4 bits per element while maintaining model convergence on medical imaging tasks.

Compensates for information loss caused by aggressive compression by accumulating local compression errors and re-injecting them into subsequent optimization steps.

• Error Accumulation: The difference between the true gradient and its compressed version is stored locally and added to the next round's gradient before compression.
• Convergence Guarantees: Error feedback restores the convergence rates of uncompressed SGD, even with biased compressors like Top-k.
• Memory-Efficient Variants: Recent algorithms minimize the memory overhead of storing error residuals on resource-constrained hospital edge nodes.

Why It Matters: Without error feedback, aggressive sparsification causes model divergence. With it, convergence matches uncompressed training.

Combines Low-Rank Adaptation (LoRA) with 4-bit quantization to enable fine-tuning of massive foundation models across hospitals using minimal communication.

• Pre-quantized Base Model: The frozen foundation model is quantized to 4-bit precision before distribution, drastically reducing local memory requirements.
• Low-Rank Adapter Transmission: Only the small, full-precision LoRA adapter matrices are communicated and aggregated, not the entire model.
• Double Quantization: Further compresses the quantization constants themselves, reducing the average memory footprint of a 65B model to ~48GB per GPU.

Clinical Impact: Enables community hospitals with a single GPU to participate in federated fine-tuning of large language models for clinical note summarization.

An alternative to direct gradient sharing where institutions exchange only model outputs (logits) on a public or synthetically generated dataset, not model parameters.

• Logit Aggregation: The central server averages the soft predictions from all local models to train a global student model.
• Zero Gradient Transmission: Completely eliminates the need to communicate high-dimensional gradient vectors.
• Heterogeneous Model Support: Each institution can use a different model architecture, as only the output layer dimensionality must match.

Privacy Advantage: Sharing logits instead of gradients provides an additional layer of protection against gradient inversion attacks that could reconstruct patient data.

Dynamically adjusts the compression ratio throughout training based on model convergence state and network conditions.

• Loss-Aware Scheduling: Increases compression when training loss is decreasing smoothly; reduces compression when loss plateaus or oscillates.
• Bandwidth-Aware Throttling: Adapts compression aggressiveness to real-time available bandwidth, critical for hospitals sharing network infrastructure with clinical systems.
• Layer-Wise Policies: Applies different compression rates to different layers—early layers (learning general features) can be compressed more aggressively than task-specific final layers.

Production Reality: Adaptive scheduling achieves 50-80% additional bandwidth savings compared to fixed compression rates in real-world hospital network deployments.

A technique that transmits only a subset of the most significant gradient elements during each communication round, setting the rest to zero. This exploits the fact that many gradients are near-zero and contribute little to model convergence.

• Top-k sparsification: Only the k largest gradients by magnitude are sent.
• Random sparsification: A random subset is selected, preserving unbiased estimates.
• Effect: Can reduce communication by 100x–1000x without significant accuracy loss.
• Challenge: Requires error feedback mechanisms to accumulate residual gradients and prevent staleness.

The process of reducing the numerical precision of gradient values from 32-bit floating point to lower bit-widths (e.g., 8-bit, 4-bit, or even 1-bit) before transmission.

• QSGD (Quantized SGD): A family of stochastic quantization schemes that map continuous gradients to a discrete set of values.
• 1-bit SGD: Reduces each gradient to only its sign, achieving a 32x compression ratio.
• Ternarization: Maps gradients to {-1, 0, +1}, balancing compression with directional fidelity.
• Benefit: Directly reduces the payload size of every update packet.

A memory mechanism that locally stores the compression residuals (the gradients not transmitted in the current round) and adds them to the next round's update. This prevents information loss and ensures eventual convergence.

• Memory: Each node maintains a local residual buffer.
• Process: residual = old_residual + current_gradient; transmit compress(residual); update residual = residual - decompress(transmitted).
• Critical for sparsification: Without error feedback, aggressive sparsification leads to divergence.
• Analogy: Similar to carrying over a remainder in long division.

The foundational aggregation algorithm that gradient compression techniques are designed to optimize. FedAvg performs multiple local SGD steps on each client before communicating updates to the server, reducing the total number of communication rounds.

• Workflow: Server distributes model → clients train locally for E epochs → clients send model updates → server averages them.
• Compression synergy: Combining FedAvg with gradient compression yields multiplicative bandwidth savings.
• Trade-off: More local epochs increase computation but decrease communication frequency.

The fundamental design tension in federated learning systems. Gradient compression reduces communication cost at the expense of increased local computation (for compression/decompression) and potentially slower convergence.

• Compression overhead: Encoding gradients into sparse or quantized formats requires CPU/GPU cycles.
• Convergence penalty: Aggressive compression can increase the total number of training rounds needed.
• Optimal point: Depends on the ratio of network bandwidth to local compute power.
• Real-world example: A hospital with a 10 Mbps uplink but idle GPU capacity benefits heavily from aggressive compression.

An alternative to gradient sharing where institutions exchange model outputs (logits) on a public or synthetic dataset rather than model parameters. This sidesteps gradient compression entirely by transmitting only prediction vectors.

• Co-distillation: Each client trains a local model and shares its predictions on an unlabeled public dataset.
• Ensemble distillation: A central student model learns from the aggregated predictions of all local teacher models.
• Privacy benefit: Raw gradients can leak data; logits on public data are less revealing.
• Communication cost: Proportional to the size of the public dataset × number of classes, not model size.