Federated Multimodal Imaging

The architecture employs parallel, independent feature extractors for each imaging modality before fusion. A PET scan and a CT scan from the same patient are processed by separate convolutional neural network branches locally at each institution. This design isolates modality-specific feature learning, allowing each encoder to specialize in the unique textural, metabolic, or anatomical patterns of its input domain without interference. The raw co-registered data never leaves the hospital; only the extracted latent representations are passed to the fusion layer.

Before federated training begins, each institution performs rigid and deformable image registration locally to spatially align multimodal scans. This step ensures that a PET voxel corresponds to the exact anatomical location in the paired CT scan. Performing this computationally intensive alignment on-site avoids transmitting raw DICOM volumes and preserves the spatial correspondence required for downstream fusion. The registration parameters are never shared, only the resulting aligned feature maps are used in local training loops.

The critical architectural component where latent representations from separate modality encoders are combined. Common fusion strategies include:

• Early Fusion: Concatenating feature maps at the input level before joint processing.
• Intermediate Fusion: Merging at a mid-network bottleneck layer.
• Late Fusion: Combining modality-specific predictions at the decision level. In a federated context, the fusion operation occurs inside the local model, ensuring that the combined multimodal representation is computed without exposing individual modality features to the central server.

A self-supervised objective applied locally to ensure that the learned representations from different modalities are semantically aligned. Using contrastive loss functions, the model pulls together feature vectors from corresponding PET/CT pairs and pushes apart non-matching pairs. This alignment is trained entirely on local data, teaching the global model to understand cross-modal relationships without ever seeing a centralized dataset. The result is a shared latent space where metabolic and anatomical information are directly comparable.

A robustness mechanism designed for real-world clinical environments where not every patient has a complete set of multimodal scans. During local training, entire modality branches are randomly dropped with a configured probability, forcing the fusion layer to learn robust representations from incomplete inputs. This simulates missing PET or MRI data at inference time, ensuring the federated model generalizes across institutions with varying imaging protocols and equipment availability without degrading diagnostic performance.

An attention mechanism that dynamically weights the contribution of each modality based on local data characteristics. Cross-attention layers allow the CT features to query the PET features and vice versa, learning which modality is most informative for a specific region or diagnosis. This attention is computed entirely within the local training loop, meaning the weighting logic adapts to each hospital's scanner biases and patient demographics without exposing the underlying attention maps to external servers.