Early Fusion Architecture

The defining mechanism of early fusion is the vector concatenation of features from disparate modalities prior to the first hidden layer. For example, pixel values from a chest X-ray are flattened and directly appended to a one-hot encoded vector of a patient's genomic mutation status. This creates a single, high-dimensional input vector that is fed into a unified neural network, forcing the model to learn cross-modal correlations from the rawest possible state.

Because all modalities are processed by a single monolithic model, backpropagation flows uniformly from the output loss through all input features simultaneously. This allows the model to learn complex, low-level interactions between modalities that might be missed by architectures that process streams independently. A pixel in an MRI scan can directly influence the weight update for a lab value, enabling the discovery of subtle, non-linear biomarker-imaging relationships.

A primary engineering challenge is the explosion in feature space dimensionality. Concatenating a 256x256 grayscale image (65,536 features) with a 20,000-gene expression vector creates an input vector of 85,536 dimensions. This often requires an impractically large number of parameters in the first layer, leading to severe overfitting unless massive, balanced multi-modal datasets are available for training.

Early fusion demands strict data alignment. All input modalities must be available simultaneously and correspond to the exact same event or time point. In a clinical setting, this means a radiology scan, a blood draw, and a genomic biopsy must all be collected during the same patient encounter. This rigid synchronization requirement often limits the architecture's applicability in real-world workflows where data streams are asynchronous.

A single dominant modality can overwhelm the learning signal. If one input source has a significantly higher dimensionality or variance—such as a high-resolution whole-slide pathology image compared to a sparse set of clinical lab values—the model may ignore the weaker modality entirely. Without explicit regularization, the gradient from the dense modality drowns out the contribution of the sparser, but potentially equally diagnostic, data source.

Unlike late fusion, where separate encoders extract high-level features independently before a final classifier combines them, early fusion learns joint features from the ground up. This makes early fusion more expressive for capturing low-level cross-modal interactions but far more brittle to missing data. In late fusion, a missing modality simply produces no feature vector; in early fusion, a missing modality breaks the fixed input schema entirely.