Federated Multi-Task Learning

FMTL learns a global feature extractor that captures common patterns across all clinical sites while allowing task-specific heads to diverge. This is distinct from standard Federated Averaging, which forces a single consensus model.

• Mechanism: A base neural network is trained collaboratively, but the final classification layers are personalized.
• Benefit: Prevents catastrophic interference where rare disease patterns from one hospital are averaged away by data from larger sites.
• Example: A shared CNN backbone for chest X-ray analysis, with Hospital A learning a head for pneumonia detection and Hospital B learning a head for tuberculosis screening.

FMTL imposes structural constraints that encourage client models to be similar but not identical. This is achieved by penalizing the divergence between local model parameters using a task relationship matrix.

• MOCHA Algorithm: A foundational FMTL framework that solves a joint optimization problem with a dual regularization term—minimizing both local empirical loss and the distance between client weight vectors.
• Statistical Foundation: Assumes that client tasks are drawn from a shared prior distribution, allowing the system to borrow statistical strength across sites.
• Practical Impact: Stabilizes training for clients with very small datasets by leveraging information from data-rich clients without copying their biases.

FMTL naturally accommodates label distribution skew—where different hospitals diagnose different disease prevalences—by treating each client's prediction task as a distinct but related problem.

• Architecture: The global model learns a universal feature space, while local models learn client-specific decision boundaries.
• Contrast with FedAvg: Standard federated averaging assumes a single global labeling function, which fails when Hospital A sees 40% cardiac cases and Hospital B sees 5%.
• Clinical Relevance: A rural clinic and an urban research hospital can collaboratively train without the urban data overwhelming the rural model's sensitivity to locally prevalent conditions.

While both address statistical heterogeneity, FMTL and Clustered Federated Learning differ fundamentally in their approach to personalization.

• FMTL: Every client gets a unique model. Task relationships are learned continuously through a similarity matrix, allowing for smooth interpolation between clients.
• Clustered FL: Clients are partitioned into discrete groups. A single model is trained per cluster, which can fail for clients at cluster boundaries.
• Computational Trade-off: FMTL requires solving a more complex optimization problem but provides finer-grained personalization than hard clustering approaches.

Standard FMTL requires transmitting full model updates, which can be bandwidth-intensive. Sparse FMTL and compressed FMTL variants reduce this overhead.

• Gradient Sparsification: Only the top-k gradient elements by magnitude are transmitted, exploiting the fact that many parameters converge quickly.
• Federated Distillation for MTL: Clients share soft label predictions on a public reference dataset instead of model weights, enabling heterogeneous local architectures.
• Practical Deployment: These techniques make FMTL viable for cross-device scenarios involving wearable health monitors with limited uplink bandwidth.

FMTL provides stronger theoretical convergence guarantees than standard FedAvg when data distributions are heterogeneous.

• Theorem: Under assumptions of smoothness and strong convexity, FMTL converges to a stationary point where each client's local model is optimal given the shared representation.
• Key Insight: The task relationship regularizer prevents the optimization from oscillating between incompatible local minima, a common failure mode in FedAvg on non-IID data.
• Practical Monitoring: Track the Frobenius norm of the difference between consecutive task relationship matrices to detect convergence.

The process of fine-tuning a globally aggregated federated model on local client data to improve performance on that specific site's unique data distribution. Unlike FMTL, which learns personalization during joint training, this is often a post-hoc step. Key techniques include:

• Local fine-tuning with a small learning rate
• Freezing shared layers and retraining only the head
• Balancing global knowledge retention with local adaptation

A strategy that partitions clients into groups with similar data distributions and trains a separate model for each cluster. This mitigates the negative impact of extreme statistical heterogeneity without requiring fully individualized models. Common clustering signals include:

• Model weight cosine similarity
• Local loss landscape comparisons
• Data distribution metadata (if shareable)
• Performance on a proxy validation set

An approach that trains a model initialization across clients such that it can rapidly adapt to a new client's local data distribution with only a few gradient steps. MAML (Model-Agnostic Meta-Learning) is the canonical algorithm adapted for federated settings. This is particularly valuable for onboarding new hospitals with minimal local compute and data.

A privacy-preserving technique where clients share soft label predictions on a public or synthetically generated dataset instead of model parameters. This enables heterogeneous model architectures across clients and naturally handles non-IID data. Advantages over FMTL include:

• No requirement for identical model architectures
• Reduced communication of large parameter vectors
• Inherent protection against gradient leakage attacks

An optimization framework that learns data representations which elicit the same optimal classifier across all training clients. The goal is to discover causal relationships robust to spurious correlations introduced by site-specific biases. Unlike FMTL, which embraces heterogeneity, FedIRM actively seeks to eliminate it from the learned representation.

A communication-efficient method where clients share compact class-representative vectors (prototypes) instead of full model updates. This naturally handles label distribution skew because each client contributes only its local class knowledge. Key properties:

• Prototypes are averaged globally per class
• Local classification uses distance to global prototypes
• Inherently robust to non-IID label distributions