Gradient Compression

Reduces the bit-width of gradient values from 32-bit floating point to lower precision representations like 8-bit integers or even 1-bit signs. Stochastic quantization adds probabilistic rounding to maintain statistical unbiasedness. For example, QSGD compresses gradients by up to 32x while preserving convergence guarantees. Common schemes include INT8, FP16, and binary quantization where only the sign of each gradient element is transmitted.

Transmits only a subset of gradient elements with the largest magnitudes, setting the rest to zero. Top-k sparsification selects the k largest values by absolute magnitude. Random sparsification samples elements probabilistically. Gradient dropping skips updates below a threshold. Deep gradient compression combines sparsification with momentum correction and local gradient accumulation to achieve 270x to 600x compression on large models without accuracy degradation.

Decomposes the gradient matrix into two smaller matrices using techniques like Singular Value Decomposition (SVD) or random projection. Only the low-rank factors are transmitted. PowerSGD applies a power iteration method to approximate the top singular vectors, achieving compression without introducing bias. Particularly effective for large fully-connected and convolutional layers where gradients exhibit low-rank structure.

Compression introduces residual error that can cause model divergence. Error feedback stores the compression error locally and adds it to the next iteration's gradient before compression, ensuring no information is permanently lost. Momentum correction adjusts the momentum buffer to account for compression lag. These techniques are essential for maintaining convergence when using aggressive sparsification or quantization in Non-IID federated settings.

Dynamically adjusts compression ratio based on training progress or network conditions. Early training rounds use mild compression to establish a good basin; later rounds apply aggressive compression. Bandwidth-aware scheduling monitors available throughput and selects compression parameters to minimize wall-clock time per round. Critical for production federated learning where hospital network conditions fluctuate unpredictably.

Combining gradient compression with cryptographic secure aggregation presents unique challenges. Quantized gradients must be compatible with the aggregation protocol's integer arithmetic. Ternary gradients with values in {-1, 0, +1} integrate cleanly with SecAgg's modular addition. This co-design ensures that communication-efficient federated learning does not sacrifice the privacy guarantees required for multi-institutional medical imaging consortia.

The foundational aggregation algorithm that gradient compression is designed to accelerate. After clients perform local training on private medical imaging data, FedAvg computes a weighted average of the compressed model updates to produce a new global model. The server never sees raw patient scans—only the mathematical deltas that gradient compression has further reduced in size. This pairing of compression and averaging is what makes multi-hospital training feasible over constrained hospital network links.

A cryptographic protocol that operates on top of compressed gradients. SecAgg ensures the central server can only compute the sum of encrypted model updates without ever inspecting any individual hospital's contribution in plaintext. When combined with gradient compression, the reduced payload size dramatically lowers the computational overhead of the cryptographic operations, making secure, privacy-guaranteed aggregation practical for large diagnostic vision models with millions of parameters.

The divergence of locally trained models from the optimal global objective, caused by Non-IID data distributions across hospitals. Gradient compression can exacerbate client drift if aggressive sparsification discards updates that are critical for rare pathologies. Mitigation strategies include:

• Error feedback: accumulating compression residuals locally
• FedProx: adding a proximal term to constrain local updates
• Adaptive compression rates: adjusting sparsity based on data heterogeneity

A mathematical framework that injects calibrated noise into model updates to provide provable privacy guarantees. Gradient compression and DP are synergistic: sparsification and quantization naturally remove some fine-grained information, reducing the amount of noise needed to achieve a target epsilon privacy budget. This dual approach—compressing then perturbing—enables stronger privacy protection with less degradation to diagnostic model accuracy than either technique alone.

A single complete cycle in federated training: the server distributes the current global model, selected hospitals perform local training on their private DICOM archives, and the server aggregates the resulting compressed updates. Gradient compression directly reduces the bottleneck in this cycle by shrinking the upstream payload. Key metrics:

• Round latency: dominated by the slowest hospital's upload
• Compression ratio: typically 100-500x for gradient sparsification
• Convergence rounds: may increase slightly with aggressive compression

A class of aggregation rules—such as Krum, Trimmed Mean, and Median—designed to defend the global diagnostic model against Byzantine failures or malicious data poisoning. Gradient compression complicates robust aggregation because sparsified updates obscure the statistical patterns that these defenses rely on. Modern approaches combine compression-aware Byzantine detection with cryptographic proofs to verify that compressed updates are honest transformations of genuine gradients.