Early Fusion

The defining mechanism of early fusion is the direct concatenation of feature vectors from different modalities before any processing occurs. For example, pixel values from an MRI scan are flattened and directly joined with structured lab values from an EHR to form a single, long input vector. This requires all modalities to be temporally aligned and represented in a compatible numerical format. The network's first hidden layer then sees the entire multimodal context simultaneously, allowing it to learn correlations between raw imaging features and clinical biomarkers from the very first weight matrix multiplication.

Because all data is combined before feature extraction, early fusion excels at capturing low-level, fine-grained interactions between modalities. A convolutional kernel in the first layer can learn patterns that span both imaging and genomic data simultaneously. This is critical for tasks where the relationship between modalities is subtle and spatially correlated, such as identifying how a specific genetic mutation manifests as a textural pattern in a histopathology slide. The model does not need to compress each modality into a high-level summary first; it can discover correlations directly in the raw signal space.

A primary technical challenge of early fusion is the curse of dimensionality. Concatenating high-dimensional raw data—such as whole-slide pathology images with millions of pixels and tens of thousands of gene expression values—creates an extremely wide input layer. This dramatically increases the number of trainable parameters in the first weight matrix, leading to:

• Massive memory consumption during training
• High risk of overfitting, especially with limited clinical samples
• Computational bottlenecks that slow convergence Dimensionality reduction techniques like PCA or autoencoders are often applied pre-fusion to mitigate this.

Early fusion imposes strict data alignment constraints. Every training sample must have all modalities present and perfectly synchronized. In a federated healthcare setting, this is a significant barrier because:

• A patient may have an MRI but no recent genomic sequencing
• Different hospitals collect different sets of modalities
• Temporal misalignment (e.g., a lab test from last year and an image from today) can introduce noise Missing modality handling is inherently difficult; the model cannot easily ignore an absent input when it expects a fixed-size concatenated vector.

In a federated learning context, early fusion presents unique architectural considerations. Since the fusion occurs at the input layer, the entire model—including the joint feature extractor—must be trained end-to-end. This means:

• Local clients must have all required modalities for every sample
• The global model architecture must be identical across all institutions
• Communication costs are driven by the size of the full model, which is often large due to the wide input layer However, when these conditions are met, early fusion allows the global model to learn joint representations that generalize across institutional boundaries without ever centralizing raw patient data.

Early fusion sits at one end of the fusion spectrum:

• Early Fusion: Combine raw inputs → single joint feature extractor → prediction
• Intermediate Fusion: Separate modality-specific encoders → combine at a hidden layer → joint layers → prediction
• Late Fusion: Separate models per modality → combine only at the decision/output layer Early fusion is preferred when cross-modal correlations are low-level and dense, while late fusion is better when modalities are independently informative. Intermediate fusion offers a balance, allowing modality-specific preprocessing before joint reasoning.

The defining mechanism of early fusion is the vector concatenation of heterogeneous data streams into a single input tensor. For example, a patient's structured EHR data (lab values, vitals) is flattened into a vector and directly appended to pixel intensities from a chest X-ray or one-hot encoded genomic variants. This unified vector is then fed into a shared neural network. The key advantage is that the model can learn joint representations where low-level features from one modality immediately influence the feature extraction of another, capturing subtle cross-modal correlations that might be lost if modalities were processed independently first.

In a federated setting for Alzheimer's disease prediction, early fusion can combine three data types at the input layer:

• MRI voxel intensities flattened into a 1D vector
• CSF biomarker levels (Aβ42, p-tau) as scalar values
• APOE genotype as a one-hot encoded categorical feature These are concatenated into a single feature vector and passed through a multi-layer perceptron (MLP) or 1D convolutional network. The network learns that a specific combination of hippocampal volume reduction, elevated p-tau, and APOE4 presence jointly predicts rapid cognitive decline—an interaction pattern that emerges naturally from joint input processing.

In a cross-silo federated learning network, each hospital holds multimodal patient data locally. The early fusion architecture operates as follows:

• Local preprocessing: Each site normalizes and flattens its own imaging, genomic, and clinical data into a standardized input vector format.
• Shared model architecture: All sites agree on an identical neural network topology that accepts the concatenated input dimension.
• Federated averaging: Only the weight updates from the jointly trained fusion network are shared with the aggregation server—never the raw multimodal patient data. This preserves privacy while enabling the model to learn from the full richness of multimodal data distributed across institutions.

Early fusion offers distinct benefits compared to late fusion (decision-level combination):

• Cross-modal feature interaction: The network learns interactions between modalities at the lowest level, not just correlations between final predictions.
• Single training pipeline: Only one model needs to be trained and maintained, simplifying the MLOps lifecycle in production clinical environments.
• Gradient flow: Backpropagation flows through all modalities simultaneously, enabling the network to learn which input features are most predictive across the entire multimodal space. However, this comes at the cost of requiring aligned, synchronized data—every training sample must have all modalities present, which is often challenging in fragmented clinical datasets.

Early fusion faces practical challenges in real-world healthcare deployments:

• Missing modality problem: If a patient is missing a genomic panel or an MRI, the entire sample becomes unusable unless imputation strategies are employed.
• Dimensionality mismatch: A 256×256 chest X-ray produces 65,536 pixel features, while lab values might contribute only 20 features. The high-dimensional modality can dominate the input space, requiring careful normalization or dimensionality reduction.
• Federated alignment complexity: Ensuring all sites preprocess data to identical input dimensions requires rigorous schema standardization across institutions, often necessitating a federated data quality validation layer.

Early fusion can be extended into Tensor Fusion Networks, which compute the outer product of modality-specific embeddings to explicitly model multiplicative interactions. While basic early fusion uses simple concatenation, tensor fusion creates a richer representation where every element of one modality's embedding is multiplied by every element of another's. For clinical applications, Low-Rank Multimodal Fusion approximates this expensive tensor product using matrix factorization, making it computationally feasible for federated settings with limited local compute resources. This allows hospitals to capture high-order interactions between imaging and genomic features without the exponential parameter explosion of full tensor products.

An architecture where modality-specific encoders extract features independently before they are combined at a hidden layer. Unlike early fusion, this allows each data stream to undergo initial domain-specific processing—such as a CNN for imaging and a transformer for clinical text—before cross-modal interactions are learned. This balances flexibility with joint representation learning.

A strategy where entirely separate models are trained on each modality and their predictions are combined only at the output stage via averaging, voting, or a meta-learner. This is the most privacy-friendly approach in federated settings, as modality-specific models can be trained in complete isolation. However, it cannot capture cross-modal interactions, limiting performance on tasks requiring holistic patient assessment.

A mechanism that dynamically weights the importance of different modalities using learned attention scores. For example, when diagnosing a skin lesion, the model might assign higher attention to dermoscopic images than to patient metadata. This is particularly valuable in clinical settings where the relevance of each data source varies per case, enabling the model to suppress noisy or irrelevant inputs.

An expressive fusion architecture that computes the outer product of modality-specific embeddings to explicitly model high-order interactions. While powerful, the resulting tensor grows exponentially with the number of modalities, making it computationally prohibitive. Low-rank multimodal fusion approximates this operation using matrix factorization to reduce complexity without sacrificing expressiveness.

A regularization strategy that randomly drops entire input modalities during training. This forces the model to learn robust representations that do not over-rely on any single data source. In federated healthcare, this is critical for handling missing modality scenarios—common when a patient's genomic data is available at one hospital but not another, or when imaging equipment varies across sites.

The process of establishing correspondences between different data types before fusion. In healthcare, this might involve aligning genomic sequences with histopathology images or synchronizing time-series EHR data with radiology timestamps. Proper alignment is a prerequisite for effective early fusion, as misaligned features introduce noise that degrades joint representation quality.