Federated LoRA

Instead of updating the full weight matrix W, Federated LoRA injects trainable low-rank matrices A and B into each transformer layer, where the update is constrained to ΔW = BA. This reduces the number of trainable parameters by over 99% compared to full fine-tuning.

• Matrix dimensions: For a weight matrix of size d×k, LoRA uses matrices of size d×r and r×k where r << min(d,k)
• Typical rank values: r=8 or r=16 for clinical language models
• Frozen backbone: The original pre-trained weights remain untouched, preserving the model's general medical knowledge
• Inference efficiency: The low-rank matrices can be merged into the original weights post-training, adding zero latency overhead

Federated LoRA transmits only the compact low-rank adapter matrices between hospital nodes and the aggregation server, dramatically reducing network overhead compared to sharing full model gradients.

• Update size: A LoRA adapter for a 7B parameter model may be only 10-50MB versus several gigabytes for full fine-tuning
• Bandwidth savings: Enables participation from hospitals with limited internet connectivity
• Encryption-friendly: Smaller payloads make homomorphic encryption and secure aggregation computationally feasible
• Asynchronous updates: Lightweight adapters allow for more flexible aggregation schedules across time zones

Only the low-rank matrices A and B are shared and aggregated, never the raw patient data or the base model weights. This architecture aligns with HIPAA and GDPR requirements for clinical data sovereignty.

• Differential privacy integration: Gaussian noise can be added to LoRA updates before transmission, providing formal (ε, δ)-privacy guarantees
• Secure aggregation protocols: Updates can be combined using secure multi-party computation so the central server never sees individual hospital contributions
• Data remains on-premise: All training computation occurs within each institution's firewall
• Audit trail: The compact adapter format simplifies version control and compliance logging

Federated LoRA excels at adapting general-purpose foundation models to specialized medical subdomains—radiology reports, pathology notes, or discharge summaries—by training lightweight adapters on distributed clinical corpora.

• Multi-task adapters: Separate LoRA modules can be trained for different clinical tasks (ICD coding, NER, summarization) and swapped dynamically
• Cross-institutional terminology: The federated process harmonizes local clinical jargon into a shared representation without exposing proprietary documentation
• Continual learning: New adapters can be added for emerging conditions without retraining the entire network
• Personalization: Hospitals can further fine-tune the global adapter on their local patient demographics for site-specific accuracy

Because LoRA fine-tuning requires significantly less GPU memory than full fine-tuning, Federated LoRA enables hospitals with modest computational resources to participate in collaborative model development.

• Memory footprint: Fine-tuning a 7B model with LoRA requires only ~16GB VRAM versus ~60GB for full fine-tuning
• Edge deployment: Compatible with single-GPU workstations common in radiology and pathology departments
• Quantization synergy: Combines with QLoRA for 4-bit training, reducing memory requirements to ~6GB
• Democratized access: Community hospitals and rural clinics can contribute to and benefit from state-of-the-art medical AI

The constrained parameter space of LoRA adapters provides inherent regularization against malicious or faulty updates in the federated network, simplifying Byzantine fault tolerance.

• Bounded update norms: The low-rank structure naturally limits the magnitude of weight perturbations from any single node
• Anomaly detection: Statistical outliers in adapter matrices are easier to identify than in full-weight updates
• Robust aggregation: Median-based or trimmed-mean aggregation of LoRA parameters is computationally efficient
• Model watermarking: Compact adapters can embed institution-specific fingerprints for provenance tracking without degrading performance

A set of mathematical techniques to reduce the size of LoRA adapter updates before they are transmitted from a hospital node to the aggregation server. Methods include gradient sparsification (sending only the top-k largest magnitude weight updates) and probabilistic quantization (reducing 32-bit floats to 2- or 3-bit representations).

• Sparsification: Often retains 99% of update fidelity while transmitting only 1% of the data.
• Error Feedback: A critical mechanism that tracks compression residuals locally and re-injects them into the next round to prevent model divergence.
• Impact: Reduces per-round bandwidth from gigabytes to megabytes for large LoRA adapters.

The phenomenon where a global foundation model sequentially adapted via LoRA to different clinical tasks across institutions loses performance on previously learned tasks. When Hospital A fine-tunes for radiology reports and Hospital B for discharge summaries, the aggregated adapter may overwrite task-specific knowledge.

• Mitigation: Elastic Weight Consolidation (EWC) penalizes changes to parameters critical for prior tasks.
• Federated Replay: A privacy-safe approach where institutions generate synthetic samples from previous tasks to rehearse during new training rounds.
• Architectural Solution: Using task-specific LoRA modules that are switched at inference time rather than merged.

A post-training technique that combines independently fine-tuned LoRA adapters from different institutions into a single, more robust model without requiring further training or access to the original data. Spherical Linear Interpolation (SLERP) and Task Arithmetic are common merging algorithms.

• Task Arithmetic: Computes a task vector (fine-tuned weights minus base weights) for each hospital and merges them via weighted summation.
• TIES-Merging: Resolves interference between task vectors by trimming redundant parameters and resolving sign conflicts before merging.
• Use Case: Combining a cardiology-specialized adapter with a pulmonology adapter to create a general chest medicine model.

A technique that adds a penalty term to the local LoRA training objective to prevent the feature representations learned at one institution from drifting too far from the global consensus. This ensures that the semantic meaning of clinical concepts remains consistent across the network.

• Contrastive Regularization: Pulls local embeddings of the same concept closer to a shared global prototype.
• Knowledge Distillation Regularization: The local model is penalized if its output distribution diverges from a frozen copy of the previous round's global model on a public proxy dataset.
• Critical for: Ensuring that 'chest pain' has a similar vector representation across cardiology departments at different hospitals.

A two-stage method where multiple full models (or LoRA-adapted models) are trained independently at different sites. Instead of averaging their weights, their collective knowledge is compressed into a single, central student model by training it on the aggregated soft output logits of the local 'teacher' ensembles on a public or synthetic dataset.

• Privacy: Only logits are shared, never raw data or full model weights.
• Robustness: Naturally handles non-IID data distributions better than weight averaging because it captures the diversity of local decision boundaries.
• Output: A single, compact model that mimics the performance of a large, diverse committee of hospital-specific experts.