Late Fusion

Each data modality—such as imaging, genomics, and EHR tabular data—is processed by a separate, independently trained model. These encoders can be architecturally heterogeneous:

• A CNN or Vision Transformer for radiology scans
• A gradient-boosted tree or MLP for structured lab values
• A transformer for clinical text This decoupling allows teams to optimize each pipeline in isolation before integration.

Fusion occurs exclusively at the output stage by combining prediction scores, logits, or class probabilities from each unimodal model. Common aggregation rules include:

• Averaging: Taking the mean of probability vectors
• Weighted voting: Assigning confidence weights to each modality
• Stacked generalization: Training a meta-learner on concatenated unimodal predictions No raw features or intermediate representations are shared between modalities.

Late fusion aligns naturally with federated learning topologies because modality-specific models can be trained on separate clients or institutions. A hospital with imaging data trains the vision encoder, while a lab with genomic data trains the omics model. Only the final prediction vectors—not raw patient data or model weights—need to be exchanged for aggregation, preserving privacy by architecture.

Because each modality has an independent model, late fusion architectures inherently support missing modality scenarios. If a patient's genomic data is unavailable at inference time, the system simply omits that model's contribution and aggregates from the remaining available modalities. This is critical in fragmented clinical environments where complete multimodal data is rare.

The modular nature of late fusion provides inherent explainability. Clinicians can inspect:

• Which modality contributed most to a final diagnosis
• Individual unimodal predictions before aggregation
• Confidence scores per data source This transparency supports algorithmic explainability requirements in regulated healthcare settings, where understanding the basis of a decision is as important as the decision itself.

The primary trade-off of late fusion is the absence of intermediate cross-modal learning. Unlike early fusion or attention-based fusion, late fusion cannot learn correlations between modalities during feature extraction. For example, it cannot discover that a specific genomic marker correlates with a subtle texture pattern in histopathology slides. This limits performance on tasks requiring holistic multimodal reasoning.

Late fusion trains separate encoders for each modality—such as a 3D CNN for CT scans and a transformer for clinical notes—without any shared parameters or intermediate feature exchange. Each model produces its own prediction vector or confidence score independently. Only at the final output stage are these scores combined via averaging, weighted voting, or a meta-learner. This strict separation means that if one modality is unavailable at inference time, the remaining models can still produce a valid prediction—a critical property in fragmented clinical environments where not all tests are always ordered.

• Each modality model can be trained on different hardware or at different institutions
• No gradient flow between modalities during training
• Enables modality dropout at inference without retraining
• Simplifies regulatory validation since each model can be approved independently

Late fusion is inherently suited to federated healthcare networks because modality-specific models can be trained at different sites without ever exchanging raw features. A hospital with strong radiology data trains the imaging model locally; a research center with genomic expertise trains the omics model. Only the output scores—not the sensitive intermediate representations—are shared for final combination. This aligns with HIPAA minimum necessary principles and reduces the attack surface for model inversion or membership inference attacks.

• No cross-modal feature sharing reduces privacy risk
• Each institution contributes only its modality expertise
• Aggregation happens on lightweight prediction vectors, not high-dimensional embeddings
• Compatible with secure multi-party computation for score aggregation

The combination of modality-specific outputs can range from simple to sophisticated. Simple averaging takes the mean of probability distributions from each model. Weighted fusion assigns learned or domain-expert-determined importance to each modality—for example, giving histopathology higher weight than family history for cancer diagnosis. Meta-learning trains a lightweight classifier on top of concatenated prediction vectors, learning non-linear interactions between modality outputs. Bayesian fusion incorporates uncertainty estimates from each model, down-weighting uncertain predictions.

• Averaging: Computationally free, robust to single-modality noise
• Weighted: Allows clinical domain knowledge to influence fusion
• Meta-learner: Captures cross-modal output correlations
• Bayesian: Propagates epistemic uncertainty through the fusion step

Consider a pulmonary embolism detection system deployed across a hospital network. The imaging model processes CT pulmonary angiograms and outputs a probability of embolism. The EHR model analyzes lab results—D-dimer levels, troponin, blood gas values—and produces a risk score. The clinical notes model processes physician narratives using a clinical BERT variant. Each model runs on its respective department's infrastructure. Only the three probability scores are sent to a central inference server that computes a weighted ensemble prediction, triggering an alert if the combined score exceeds threshold. If the CT scanner is down, the system degrades gracefully using only lab and text modalities.

Late fusion cannot model cross-modal interactions at the feature level. If a subtle pattern in an MRI only becomes clinically significant when combined with a specific genomic marker, late fusion may miss it because each model processes its input in isolation. Early or intermediate fusion architectures can learn these joint representations but at the cost of privacy and modularity. Late fusion also requires each modality model to be independently capable of making a prediction, which demands sufficient label supervision for each modality separately—a challenge when annotations are sparse for certain data types.

• Cannot learn cross-modal feature interactions
• Requires each modality to be independently predictive
• May underperform joint fusion when modalities are highly complementary
• Performance ceiling bounded by the best single-modality model

Evaluating late fusion systems requires metrics beyond simple accuracy. Modality ablation studies measure performance when each modality is removed, quantifying each data source's marginal contribution. Oracle fusion compares the ensemble against the theoretical upper bound where the best modality is always selected. Calibration error assesses whether fused probability estimates reflect true likelihoods—critical for clinical decision-making. Inter-modal agreement metrics like Cohen's kappa measure how often modality-specific models disagree, flagging cases requiring human review.

• Ablation: Quantifies modality importance
• Oracle comparison: Measures fusion quality vs. theoretical maximum
• Expected calibration error (ECE): Ensures reliable confidence scores
• Disagreement analysis: Identifies ambiguous cases for clinician review

The architectural opposite of late fusion. Raw or minimally processed features from all modalities are concatenated at the input layer before being fed into a single joint model. This allows the network to learn cross-modal interactions from the very first layer.

• Strength: Captures fine-grained inter-modal correlations
• Weakness: Requires all modalities to be present and aligned; high dimensionality
• Example: Concatenating pixel values from MRI with tokenized clinical notes before a transformer encoder

A hybrid strategy where modality-specific encoders first extract independent features, which are then combined at one or more intermediate hidden layers for joint processing. This balances the expressiveness of early fusion with the flexibility of late fusion.

• Strength: Allows unimodal pre-training and hierarchical cross-modal interaction
• Weakness: More complex architecture design; still sensitive to missing modalities
• Example: Fusing chest X-ray CNN features with EHR embeddings at the second layer of a shared transformer

Uses cross-modal attention mechanisms to dynamically weight the importance of each modality for a given prediction. Unlike late fusion's static averaging, attention scores allow the model to focus on the most diagnostically relevant data source per sample.

• Strength: Interpretable modality importance; handles noisy inputs
• Weakness: Higher computational cost than simple averaging
• Example: A model that learns to weight genomic features higher for rare disease classification but imaging features higher for fracture detection

Maps representations from different modalities into a shared latent vector space where semantically similar concepts—regardless of their source modality—are close together. Late fusion often operates on these pre-aligned embeddings.

• Strength: Enables cross-modal retrieval and zero-shot transfer
• Weakness: Requires paired training data for alignment
• Example: CLIP-style training that aligns histopathology images with their corresponding pathology reports in a shared embedding space

A regularization technique that randomly drops entire modalities during training. This forces the fusion mechanism to avoid over-relying on any single data source and directly addresses late fusion's vulnerability to missing modalities at inference time.

• Strength: Produces models robust to missing data
• Weakness: May reduce peak performance when all modalities are present
• Example: Training with 30% probability of dropping genomic data so the model learns to rely on imaging and EHR alone when necessary

An expressive alternative to late fusion that computes the outer product of modality-specific embeddings to explicitly model multiplicative interactions. While late fusion only captures additive combinations, tensor fusion captures higher-order relationships.

• Strength: Models complex inter-modal interactions
• Weakness: Exponential growth in parameters; often requires low-rank approximation
• Example: Computing the tensor product of imaging, genomic, and clinical embeddings to capture three-way interactions for survival prediction