Federated Multi-Task Aggregation (FedMT)

Unlike FedAvg, which forces a single global model, FedMT explicitly learns a task relationship matrix that captures pairwise similarities between client data distributions. This matrix is often parameterized as a low-rank embedding or a graph Laplacian, allowing the server to regularize local models toward statistically similar peers rather than a one-size-fits-all global mean. The result is a structured transfer of knowledge that respects the underlying clinical heterogeneity across sites.

FedMT maintains a shared base representation while allowing each client to learn a task-specific head or a set of personalized parameters. This architecture is typically formalized as:

• Shared layers: Capture universal features (e.g., low-level imaging patterns)
• Task-specific layers: Adapt to local population characteristics (e.g., demographic-specific diagnostic thresholds) This decoupling prevents the global model from averaging away clinically significant local variations.

The core optimization problem in FedMT extends the standard local loss with a multi-task regularization penalty. This penalty, often based on the trace norm or clustered task structure, encourages local models to be close to a weighted combination of other clients' models. The objective can be expressed as:

• Local loss: Task-specific empirical risk
• Coupling term: Penalizes deviation from the learned task relationship structure
• Complexity control: Prevents overfitting to small local datasets by borrowing strength from related tasks

FedMT natively supports architectural heterogeneity across clients, a critical requirement in real-world healthcare networks where different hospitals may use different model backbones. By operating at the representation level rather than the parameter level, FedMT can align features extracted by a CNN at one site with those from a Vision Transformer at another. This is achieved through representation matching losses or knowledge distillation between the task-specific and shared components.

FedMT provides theoretical convergence guarantees under statistically heterogeneous (non-IID) data partitions. By modeling client data as distinct tasks, the algorithm avoids the client drift problem that plagues FedAvg. The task relationship matrix acts as a structured prior that guides local optimization toward a consensus manifold, even when local data distributions are disjoint. Convergence analysis typically shows linear speedup with the number of clients under mild assumptions on task relatedness.

To reduce communication overhead, FedMT can be combined with periodic relationship updates and compressed gradient transmission. The task relationship matrix is typically updated less frequently than model parameters, as task similarities evolve slowly over time. Additionally, sparsification and quantization techniques can be applied to the gradient communication between server and clients, making FedMT practical for bandwidth-constrained hospital networks.

The foundational aggregation algorithm that constructs a global model by computing a weighted average of locally trained model updates. In FedMT contexts, FedAvg serves as the baseline against which multi-task personalization is measured.

• Weights updates proportionally to local dataset size
• Assumes all clients share a single underlying task
• Performance degrades under severe non-IID data distributions
• Communication rounds: typically 100-500 for convergence

A paradigm shift from single-model aggregation to client-specific model adaptation. FedMT is a core technique within this family, using multi-task learning to share statistical strength while preserving local specialization.

• Addresses the one-model-fits-all limitation of FedAvg
• Techniques include: model interpolation, meta-learning, and multi-task learning
• Critical for healthcare where patient populations differ across hospitals
• FedMT specifically models the task relatedness matrix between clients

The central challenge that FedMT addresses. Clinical data across institutions exhibits label distribution skew, feature distribution skew, and concept drift—violating the IID assumption of standard federated optimization.

• Covariate shift: Different hospitals image with different scanner vendors
• Label shift: Disease prevalence varies by demographic catchment area
• Concept shift: The same diagnosis may present differently across populations
• FedMT treats each client's distribution as a distinct but related task

An alternative to FedMT that partitions clients into discrete clusters based on data distribution similarity, maintaining separate global models per cluster. Contrasts with FedMT's continuous task-relationship modeling.

• Assigns clients to clusters via loss-based similarity
• Hard clustering vs. FedMT's soft task relatedness
• Effective when client populations form distinct groups
• FedMT excels when task relationships are gradual and overlapping

A complementary personalization technique where a global model serves as a pre-trained initialization for local fine-tuning. Unlike FedMT's joint multi-task optimization, transfer learning applies personalization as a post-aggregation step.

• Global model trained via standard FedAvg
• Local adaptation via few-shot fine-tuning on client data
• Simpler than FedMT but lacks inter-task knowledge sharing during training
• FedMT jointly optimizes all tasks, enabling mutual improvement

The broader framework that FedMT implements. FMTL applies multi-task learning principles to federated settings, learning a shared representation while allowing task-specific parameters. FedMT is a specific algorithmic instantiation.

• Learns a task covariance matrix to model relationships
• Uses regularization to encourage similar clients to have similar models
• Can be formulated with convex or non-convex objectives
• Enables knowledge transfer between clients with limited local data