Federated Self-Supervised Learning

The core mechanism enabling learning without labels. Models solve artificially constructed tasks where the supervisory signal is derived from the data itself. Common medical imaging pretexts include:

• Contrastive learning: Maximizing agreement between differently augmented views of the same scan
• Masked image modeling: Predicting intentionally hidden patches in radiology images
• Rotation prediction: Determining the applied rotation angle to learn anatomical orientation features
• Jigsaw puzzle solving: Reordering shuffled image patches to learn spatial relationships These tasks force the model to learn clinically meaningful representations—such as tissue textures and organ boundaries—without requiring radiologist annotations.

A decentralized implementation of contrastive representation learning where local models learn to pull similar samples together and push dissimilar samples apart in embedding space. The process operates in two phases:

• Local phase: Each institution generates positive pairs via augmentation and computes contrastive loss on its own unlabeled data
• Aggregation phase: Only the learned encoder weights—not patient data or embeddings—are shared with the central server This approach has demonstrated performance within 1-3% of centralized supervised models on chest X-ray classification while maintaining complete data locality. Key challenge: Ensuring negative pairs are sufficiently diverse across sites without sharing samples.

A self-supervised framework that eliminates the need for negative pairs, making it particularly well-suited for federated medical environments where constructing meaningful negatives across institutions is problematic. The architecture uses:

• Online network: Processes augmented views and predicts representations
• Target network: An exponential moving average of the online network providing stable regression targets
• Predictor MLP: Prevents representational collapse without contrastive negatives In federated settings, only the online encoder is shared during aggregation. This avoids the communication overhead of synchronizing target networks and has shown strong results on histopathology patch classification across multiple cancer types.

FSSL directly addresses the pervasive label bottleneck in clinical AI where expert annotation is expensive and unevenly distributed. The paradigm enables:

• Universal pretraining: All institutions contribute unlabeled data to learn a shared representation space
• Local fine-tuning: Each site fine-tunes only the final classification head on its limited labeled samples
• Few-shot transfer: Representations learned on one modality (e.g., CT scans) transfer to related tasks with as few as 10-50 labeled examples This is critical for rare disease detection where no single institution has sufficient positive cases. The global model learns general anatomical and pathological features from all sites, then adapts to local label distributions.

Self-supervised objectives demonstrate inherent resilience to non-IID data distributions compared to supervised federated learning. Because pretext tasks operate on individual samples rather than class labels, they are less affected by:

• Label distribution skew: Different disease prevalence across hospitals
• Feature distribution skew: Varying scanner manufacturers and acquisition protocols
• Concept drift: Evolving clinical definitions over time Research shows that FSSL models pretrained on heterogeneous, unlabeled multi-institutional data converge more stably than supervised models trained on skewed labeled data. The learned representations capture universal visual features that transcend site-specific biases.

An adaptation of masked image modeling where local models learn to reconstruct intentionally obscured regions of medical images. The process:

• Masking: Random patches of input images (e.g., 75% of a retinal scan) are hidden
• Encoding: A vision transformer processes only visible patches
• Decoding: A lightweight decoder reconstructs the full image from encoded representations
• Federated aggregation: Only the encoder weights are averaged across institutions This approach excels at learning fine-grained anatomical structures and has proven effective for downstream segmentation tasks. The asymmetric encoder-decoder design keeps communication costs low, as the decoder remains local.

The engineering of self-supervised proxy objectives that force a model to learn meaningful representations from unlabeled data. In a federated context, these tasks must be computationally feasible on local hospital nodes.

• Contrastive Learning: Pulls augmented views of the same sample together while pushing apart dissimilar samples (e.g., SimCLR, MoCo).
• Masked Autoencoding: Randomly masks patches of an input (e.g., clinical time-series or image regions) and trains the model to reconstruct the missing content.
• Jigsaw Puzzles: Shuffles spatial or temporal segments and tasks the model with predicting the correct permutation, enforcing structural understanding.

The downstream adaptation phase where a federated pre-trained model is specialized to a supervised clinical task using limited labeled data. This bridges self-supervised pre-training and diagnostic utility.

• Linear Probing: Freezes the pre-trained backbone and only trains a local classification head, preserving global representations while personalizing output layers.
• Parameter-Efficient Tuning: Inserts lightweight adapter modules or low-rank matrices (LoRA) into the frozen model, drastically reducing communication costs during specialization.
• Semi-Supervised Fine-Tuning: Combines a small amount of labeled data with a large pool of unlabeled data at each site, leveraging consistency regularization to improve generalization.

Techniques to ensure self-supervised representations remain stable when local client data is statistically skewed, a common failure mode in healthcare federated learning.

• Group Normalization: Replaces Batch Normalization to avoid relying on cross-client batch statistics, which are unreliable when local distributions differ.
• Federated Domain Generalization: Trains the model to learn representations invariant to the specific hospital or scanner type, using local domain classifiers or adversarial objectives.
• Sharpness-Aware Minimization (SAM): An optimization technique that finds flatter minima in the loss landscape, which has been shown to improve generalization to unseen client distributions.