Update Compression

Quantization reduces the numerical precision of model parameters (weights, gradients) from high-bit formats like 32-bit floating-point to lower-bit formats (e.g., 8-bit integers). This directly shrinks the payload size.

• Uniform Quantization: Maps float values to integers using a linear scaling function. Simple and hardware-friendly.
• Non-Uniform Quantization: Uses non-linear mappings (e.g., logarithmic) for better accuracy preservation where parameter distributions are skewed.
• Quantization-Aware Training (QAT): Simulates quantization during local client training to make the model robust to the precision loss, often yielding better final accuracy than Post-Training Quantization (PTQ).

Sparsification transmits only a subset of the most significant values from a model update, setting all others to zero. This creates a sparse tensor that can be efficiently compressed.

• Top-k Sparsification: Selects only the k parameters (or gradient entries) with the largest absolute magnitudes for transmission.
• Random Sparsification: Selects parameters randomly according to a probability distribution, which can be unbiased but less efficient.
• Error Feedback: A critical companion technique. The compression error (the difference between the original and transmitted update) is accumulated locally and added to the next round's update, preventing convergence drift and ensuring the server eventually receives full information.

These techniques use dimensionality reduction to create a compressed representation (a sketch) of the high-dimensional model update.

• Random Projection: Projects the update vector onto a random lower-dimensional subspace (e.g., using the Johnson-Lindenstrauss lemma). The server can aggregate in the compressed space.
• Count Sketch: A streaming algorithm data structure that can approximate the heavy hitters (largest values) in a high-dimensional vector with minimal memory.
• Benefits: Provides strong theoretical guarantees on approximation error and is often combined with sparsification for compounded compression rates.

This technique approximates a weight update matrix (e.g., for a fully connected layer) as the product of two smaller, low-rank matrices. Instead of sending an m x n matrix, clients send two matrices of sizes m x r and r x n, where r is the chosen rank (r << m, n).

• Communication Savings: Transmits r*(m+n) parameters instead of m*n, achieving significant compression when the update matrix has inherent low-rank structure.
• Application: Particularly effective for compressing updates to large weight matrices in transformer layers or recommendation models. It assumes the important update information lies in a low-dimensional subspace.

These are extreme forms of quantization that push each parameter value to a very small, discrete set.

• TernGrad: Quantizes each gradient value to one of three states: -1, 0, or +1. The scaling factor is communicated separately.
• 1-Bit SGD: Quantizes each gradient to just -1 or +1 (or 0/1). SignSGD is a prominent algorithm that transmits only the sign of the gradient, not its magnitude.
• Trade-off: Achieves up to 32x compression versus 32-bit floats but introduces significant noise. Requires careful tuning of learning rates and is often used with error accumulation to maintain convergence.

Google's dedicated framework for federated learning research and production. It provides foundational building blocks for implementing compression within the federated computation graph.

• Core Support: Built-in tff.learning APIs allow wrapping model updates with compression functions before aggregation.
• Modular Design: Researchers can implement custom tff.aggregators that apply quantization, sparsification, or structured updates.
• Integration: Seamlessly works with TensorFlow's model serialization and its own quantization tools (e.g., TensorFlow Model Optimization Toolkit).
• Example: The framework's tutorials demonstrate applying compression to the FedAvg algorithm to reduce client-server communication overhead.

A Python library for secure, private deep learning, with explicit support for federated learning and compression in its protocols.

• Protocol-Level Compression: The FederatedDataLoader and communication hooks allow for compressed tensor serialization before transmission over the network.
• Privacy-Compression Integration: Designed to work in tandem with secure multi-party computation and differential privacy, where compression can be applied to encrypted or noised updates.
• WebSocket Optimization: Its networking layer is built to handle the transmission of sparse or quantized tensors efficiently between clients and a PyGrid server.

An agnostic framework compatible with any ML library (PyTorch, TensorFlow, JAX). Its strategy abstraction is the primary point for integrating update compression.

• Strategy Customization: Developers subclass the FedAvg strategy to override the aggregate_fit method, applying compression/decompression logic on the server side.
• Client-Side Callbacks: The client's fit method can be extended to compress the model parameters (parameters_to_ndarrays) before returning them to the server.
• Example Implementations: The Flower repository includes examples for federated learning with quantization, showcasing how to reduce payload size by over 4x with minimal accuracy loss.

A scalable platform for federated learning applications, offering enterprise-grade features with built-in support for compression techniques.

• Turnkey Compression: Provides configurable controllers and executors that apply lossy (quantization, top-k sparsification) and lossless (encoding) compression by default.
• Heterogeneity Handling: Its compression filters are designed to account for varying client hardware, applying appropriate techniques based on system capabilities.
• Real-World Focus: Includes benchmarks demonstrating communication reduction of 75-90% for computer vision and NLP tasks in cross-silo federated scenarios.

While not frameworks themselves, these benchmarking suites provide standardized implementations of compression algorithms, enabling fair comparison and integration into custom codebases.

• LEAF: Includes reference implementations of compression methods for its FEMNIST and Shakespeare datasets, serving as a baseline for research reproducibility.
• FedML: The open-source library offers FedAvg with communication compression as a core algorithm, allowing users to directly compare the convergence of compressed vs. uncompressed updates.
• Utility: Essential for validating that a custom compression implementation meets expected performance and bandwidth reduction targets.

Many production systems build compression directly into their training loop using PyTorch's tensor operations and serialization.

• Core Techniques:
  • Quantization: Use torch.quantization modules or custom functions with torch.round() to reduce precision.
  • Sparsification: Apply torch.topk() or thresholding to create sparse tensors, then use torch.sparse compressed formats (COO, CSR).
  • Encoding: Use libraries like blosc or zlib via Python's pickle protocol for lossless compression of the final update byte stream.
• Integration Pattern: The local training function returns a compressed state dictionary; the server's aggregation function first decompresses all updates before averaging.