Low-Rank Multimodal Fusion

Instead of computing the full outer product of modality embeddings—which creates an intractable high-dimensional tensor—LMF decomposes the fusion weight tensor into parallel low-rank matrices. Each modality embedding is projected through its own factor matrix, and the resulting vectors are combined via element-wise product. This approximates the full tensor interaction with linear growth in parameters relative to the number of modalities, reducing complexity from O(d^M) to O(M × d × r), where r is the rank.

In a federated healthcare setting with three modalities—imaging, genomics, and clinical text—a standard tensor fusion would require a weight tensor with d³ parameters. LMF reduces this to 3 × d × r parameters by factorizing the tensor into modality-specific low-rank components. For a typical embedding dimension of 512 and rank of 32, this represents a reduction from ~134 million parameters to ~49,000, making fusion feasible on edge devices and within bandwidth-constrained federated rounds.

Each data modality receives its own learnable low-rank projection matrix that maps its embedding into a shared latent subspace. These factor matrices act as modality-specific interpreters that extract the most salient interaction features before the element-wise fusion step. In federated learning, these matrices can be updated locally on each institution's data, capturing site-specific modality characteristics while the shared fusion representation remains globally consistent.

Despite the low-rank approximation, LMF retains the ability to model multiplicative interactions between all modality pairs. The element-wise product of the factorized projections implicitly captures second-order and higher-order relationships. This is critical in clinical contexts where, for example, a specific genomic mutation's significance depends on its co-occurrence with particular imaging phenotypes—interactions that additive fusion methods would miss entirely.

LMF's compact parameter footprint directly benefits federated learning deployments across hospitals. The reduced model size means lower communication overhead during gradient aggregation rounds and faster local training on hospital infrastructure. Additionally, the factorization structure naturally supports heterogeneous modality availability—if a rural clinic lacks genomic sequencing capabilities, its local factor matrix for that modality can be masked or imputed without destabilizing the global model.

• Early Fusion: Concatenates raw features but cannot model multiplicative interactions and suffers from high dimensionality.
• Late Fusion: Averages independent modality predictions but ignores cross-modal synergies entirely.
• Tensor Fusion: Models all interactions but is computationally prohibitive beyond two modalities.
• LMF: Approximates full tensor interactions with linear parameter scaling, making it the only practical choice for three or more modalities in resource-constrained federated environments.

The direct precursor to low-rank methods. Tensor Fusion Networks compute the full outer product of modality-specific embeddings to explicitly model high-order interactions between data streams like imaging and genomics.

• Captures multiplicative interactions between all feature dimensions
• Computationally prohibitive: an N-modality fusion creates an N-dimensional tensor
• Low-rank fusion directly addresses this exponential parameter explosion
• Serves as the theoretical upper bound on expressiveness that low-rank methods approximate

A mechanism where the representation of one modality guides the feature extraction of another. In a federated clinical context, cross-modal attention allows a radiology report to contextually inform the interpretation of a chest X-ray.

• Uses query-key-value projections across modalities
• Enables dynamic, context-dependent fusion rather than static concatenation
• Computationally lighter than full tensor products
• Often combined with low-rank constraints for further efficiency gains in multi-institutional settings

Independent neural network branches that extract salient features from a single data type before passing embeddings to a shared fusion module. These encoders are the preprocessing backbone of any multimodal system.

• Imaging: Vision Transformers or CNNs for pathology slides
• Genomics: 1D convolutional networks for variant calling data
• EHR: Clinical BERT models for structured and unstructured text
• In federated settings, these encoders are trained locally before fusion layers aggregate cross-modal representations

A shared latent vector space where representations of different modalities are mapped to enable direct comparison. Contrastive objectives pull matched pairs together while pushing mismatched pairs apart.

• Enables cross-modal retrieval: querying radiology reports using an image
• Foundation for zero-shot classification across modalities
• In federated learning, joint spaces must be aligned without sharing raw data
• Low-rank constraints can regularize the dimensionality of this shared space to prevent overfitting on small clinical cohorts

Techniques for enabling a multimodal model to make predictions when one or more expected inputs are absent at inference time. This is the norm in clinical environments, not the exception.

• Modality dropout: randomly dropping inputs during training to force robustness
• Generative imputation: using available modalities to reconstruct missing ones
• Low-rank fusion naturally handles missing modalities by factorizing the tensor into modality-specific factors that can be computed independently
• Critical for real-world federated deployments where not every hospital collects every data type

A decentralized training paradigm where a shared model learns to perform multiple related tasks simultaneously across different clients. This leverages task commonalities to improve generalization.

• Example: jointly predicting mortality risk and length of stay from multimodal ICU data
• Low-rank structures can be shared across tasks to reduce parameter count
• Each hospital may have different task labels available
• Complements multimodal fusion by allowing the model to learn representations useful for multiple clinical endpoints