Federated Quantization

A technique that simulates the effects of low-precision arithmetic during the federated training loop. Instead of applying quantization after training, QAT allows the model to learn parameters that are inherently robust to precision loss.

• Fake Quantization Nodes: Inserts simulated quantization operations into the model graph during the forward pass.
• Straight-Through Estimator (STE): Approximates gradients through non-differentiable quantization functions during the backward pass.
• Benefit: Prevents the significant accuracy degradation often seen when a high-precision model is abruptly converted to INT8 or INT4 for edge deployment.

The primary bottleneck in federated learning is often the upload bandwidth. Quantizing gradients from 32-bit floating point (FP32) to 8-bit integers (INT8) or lower before transmission can reduce communication costs by up to 4x.

• Stochastic Rounding: A probabilistic rounding method that preserves the expected value of the gradient, preventing systematic bias that deterministic rounding introduces.
• Error Feedback: A mechanism that stores local quantization errors and adds them back into the next iteration's gradient, maintaining convergence speed.
• QSGD (Quantized SGD): A foundational algorithm that formalizes the trade-off between communication savings and variance introduced by gradient compression.

A calibration-based method applied to a fully trained federated model before deployment to edge devices. PTQ requires only a small, unlabeled representative dataset to determine optimal clipping ranges and scaling factors.

• Dynamic Quantization: Weights are quantized ahead of time, but activations are quantized on-the-fly during inference. Ideal for models with highly variable activation ranges.
• Static Quantization: Both weights and activations are quantized using pre-computed scales derived from calibration data, offering maximum inference speedup.
• Use Case: Converting a global chest X-ray model trained in FP32 to an INT8 model for deployment on a portable ultrasound device with limited RAM.

A federated aggregation strategy that accommodates clients with varying hardware capabilities. A central server can aggregate model updates submitted in different bit-widths (e.g., INT4, INT8, FP16) from different hospitals.

• Mixed-Precision Training: Allows computationally rich clients to train locally in FP16 for higher accuracy, while resource-constrained clients train in INT8.
• Weighted Aggregation: The server applies a dequantization step to align all updates to a common precision before averaging, often weighting contributions by the precision level.
• Hardware Awareness: Enables participation from a diverse fleet of devices, from powerful GPU clusters to legacy CPU-only servers, without sacrificing global model integrity.

Pushing precision below INT8 to ternary ({-1, 0, 1}) or binary ({-1, 1}) representations to achieve drastic model size reductions for medical wearables.

• Binary Neural Networks (BNNs): Replace arithmetic operations with bitwise XNOR and popcount operations, enabling massive speedups on custom hardware.
• Ternary Weight Networks (TWNs): Introduce a zero state to improve representational capacity over binary networks while maintaining significant compression.
• Federated Binarization: A challenging research area where local training must navigate a highly discrete loss landscape, often requiring specialized optimizers like Bop (Binary Optimizer) to converge.

A synergistic combination of Federated LoRA and 4-bit NormalFloat (NF4) quantization. This technique allows hospitals to collaboratively fine-tune massive language models on clinical text using minimal GPU memory and network transfer.

• Double Quantization: Further compresses the quantization constants themselves, reducing the memory footprint by an additional 0.4 bits per parameter.
• Paged Optimizers: Offloads optimizer states to CPU RAM to handle the memory spikes that occur during gradient checkpointing, enabling fine-tuning of a 65B parameter model on a single 48GB GPU.
• Federated Integration: Only the low-rank adapter weights (typically <1% of the model size) are transmitted and aggregated, while the 4-bit base model remains frozen and local.

A highly efficient combination of federated LoRA and 4-bit NormalFloat quantization that allows institutions to collaboratively fine-tune massive language models on clinical text using minimal GPU memory and network transfer.

• Freezes the quantized base model weights at 4-bit precision
• Only trains and transmits low-rank adapter matrices
• Reduces per-node GPU memory requirements by up to 65%
• Enables fine-tuning of 65B parameter models on a single consumer-grade GPU

A set of techniques applied to model updates before transmission from a hospital node to the central server, drastically reducing the bandwidth required for decentralized training.

• Gradient sparsification: Transmits only the top-k largest gradient values
• Stochastic quantization: Maps 32-bit gradients to 2-bit or 4-bit representations
• Error feedback: Accumulates quantization residuals locally to maintain convergence
• Critical for cross-silo deployments over hospital WAN links with limited bandwidth

A compression technique applied to a fully trained federated model before deployment to edge devices, converting 32-bit floating-point weights to 8-bit or 4-bit integers without retraining.

• Static quantization: Calibrates activation ranges using a representative dataset
• Dynamic quantization: Computes activation ranges on-the-fly during inference
• Enables deployment of federated diagnostic models on ultrasound machines, portable X-ray units, and wearable cardiac monitors
• Typically achieves 4x model size reduction with minimal accuracy loss

A federated training methodology that simulates the effects of low-precision arithmetic during the forward and backward passes, producing models that are robust to quantization noise from the start.

• Inserts fake quantization nodes into the model graph during local training
• Models learn to compensate for rounding errors during optimization
• Produces significantly higher accuracy than PTQ for aggressive bit-widths (≤4-bit)
• Essential for training models destined for neural processing units (NPUs) on medical edge devices

A complementary compression technique that removes redundant or low-magnitude weights from a model before or during federated training, reducing both computation and communication costs.

• Unstructured pruning: Zeroes out individual weights, creating sparse matrices
• Structured pruning: Removes entire channels or attention heads for hardware-friendly speedups
• Often combined with quantization for compound compression ratios
• Pruned masks can be learned locally and aggregated globally using majority voting or intersection strategies

A model compression technique where a large, high-precision global 'teacher' model's knowledge is transferred to a smaller, quantized 'student' model at each site by sharing only the teacher's output logits on a public or synthetic dataset.

• Avoids exchanging private model gradients entirely
• Student models can be aggressively quantized for edge deployment
• Logit aggregation at the server preserves privacy while transferring dark knowledge
• Enables a single federated teacher to spawn many site-specific, lightweight diagnostic models