Federated Transfer Learning

FTL addresses scenarios where source and target clients have non-overlapping feature spaces—a common occurrence in healthcare where different hospitals collect distinct biomarker panels. The technique learns a common subspace or uses domain adaptation layers to map disparate feature sets into a shared representation, enabling knowledge transfer without requiring identical input schemas across institutions.

Unlike standard federated learning, FTL explicitly handles asymmetric label distributions across clients. A source client may have a model trained on a broad set of disease classifications, while a target client only possesses data for a subset. FTL employs partial model sharing and classifier head adaptation to transfer relevant knowledge without forcing label alignment, preserving local diagnostic specificity.

FTL architectures typically split the model into a base feature extractor (shared across all clients) and domain-specific adaptation layers (trained locally). The base layers, pre-trained on large centralized datasets, capture universal patterns, while the local adaptation layers fine-tune representations to the target site's unique data distribution. This partitioning minimizes communication overhead and preserves data locality.

When direct model weight sharing is restricted, FTL leverages federated knowledge distillation. A source model's soft predictions or intermediate representations are transmitted to target clients as pseudo-labels or hint layers. Target models are trained to mimic these outputs on their local data, effectively transferring knowledge without exposing the source model's architecture or training data.

FTL incorporates covariate shift correction mechanisms to address the non-IID nature of clinical data. Techniques include: - Batch normalization recalibration on local data distributions - Adversarial domain classifiers that encourage domain-invariant feature learning - Importance-weighted loss functions that down-weight samples divergent from the source distribution

FTL integrates with differential privacy guarantees during the adaptation phase. Local gradient updates are clipped and noised before aggregation, ensuring that the fine-tuning process on sensitive patient data does not leak individual-level information. This is critical for cross-institutional model adaptation under HIPAA and GDPR constraints.

A federated learning objective that trains a model across multiple source domains to generalize to an entirely unseen target domain. Unlike standard federated transfer learning, which adapts to a known target, domain generalization prepares for zero-shot deployment on new hospital sites with no prior data access.

• Goal: Robustness to unknown distribution shifts
• Technique: Invariant risk minimization across source clients
• Healthcare use: Model trained on US and EU hospitals deployed in an Asian hospital without retraining

A decentralized training paradigm where a shared model learns to perform multiple related tasks simultaneously across different clients. Each hospital may have different label spaces—one predicts mortality, another predicts length of stay—and the shared representation leverages commonalities to improve generalization.

• Key benefit: Mitigates label space heterogeneity
• Architecture: Shared encoder with task-specific heads
• Example: A federated EHR model jointly predicting readmission risk and sepsis onset across silos

A family of techniques that tailor a global federated model to the specific data distribution of each local client. In healthcare, this addresses the reality that patient populations differ across hospitals, and a one-size-fits-all model may underperform for minority subgroups.

• Methods: Local fine-tuning, model interpolation, meta-learning
• Relationship to FTL: Federated transfer learning is a specialization of personalized FL where the source and target tasks differ
• Outcome: Each hospital retains a site-adapted model while benefiting from collaborative training

A communication-efficient federated approach where clients share abstract class prototypes—representative embedding vectors for each category—instead of raw gradients or model weights. This reduces bandwidth and enhances privacy by never transmitting individual-level information.

• Mechanism: Each client computes a mean embedding per class and sends only these prototypes to the server
• Advantage: Naturally handles heterogeneous label spaces across clients
• Relevance: Enables transfer-like adaptation when target clients have different class definitions than source clients

The set of strategies for managing statistical and system heterogeneity in decentralized clinical datasets where data distributions are not independent and identically distributed. Federated transfer learning is particularly vulnerable to non-IIDness when the source pre-training distribution differs significantly from the target silo distribution.

• Types of non-IID: Label distribution skew, feature distribution skew, concept drift
• Mitigations: FedProx, SCAFFOLD, FedNova
• Impact: Unaddressed non-IIDness can cause catastrophic forgetting during federated fine-tuning

A decentralized training paradigm where clients learn useful representations from unlabeled local data using pretext tasks, without requiring manual annotation at any site. These pre-trained representations serve as excellent initialization for downstream federated transfer learning on labeled target tasks.

• Pretext tasks: Contrastive learning, masked autoencoding, rotation prediction
• Advantage: Exploits vast pools of unlabeled clinical data across silos
• Workflow: Federated SSL pre-training → Federated transfer to labeled target task with limited samples