Cross-Modal Attention

Unlike self-attention where queries, keys, and values originate from the same sequence, cross-modal attention uses queries from a target modality and keys/values from a source modality. This asymmetry allows a transcriptomics encoder to query a proteomics context, dynamically selecting relevant protein expression patterns to inform gene-level representations. The learned weight matrices project each modality into a shared dimensional space where compatibility scores can be computed.

Raw omics data must be converted into a sequence of embeddings before attention can be applied. Cross-modal architectures use distinct tokenization strategies per modality:

• Genomics: Gene expression vectors are linearly projected or processed by a shallow MLP into patch embeddings.
• Proteomics: Protein abundance values are embedded with positional encodings reflecting pathway membership.
• Metabolomics: Metabolite concentrations are tokenized with chemical structure fingerprints as auxiliary features. These modality-specific encoders ensure that heterogeneous input types are represented in a format suitable for cross-attention computation.

A dedicated fusion layer alternates or concatenates cross-attention operations between modality pairs. For example, a genomics-to-proteomics cross-attention block computes attention scores where gene tokens attend to all protein tokens, followed by a proteomics-to-genomics block in the reverse direction. The resulting context-aware representations are then aggregated—often via learned weighted sums or gating mechanisms—to produce a unified multi-modal embedding for downstream tasks like patient survival prediction.

The attention weight matrices produced during cross-modal computation serve as explanatory artifacts. By extracting the attention scores between specific gene tokens and protein tokens, researchers can identify which molecular relationships the model deemed important for a prediction. This provides a data-driven hypothesis generation mechanism: high-attention cross-modal pairs often correspond to known regulatory interactions or suggest novel mechanistic links between transcriptomic and proteomic layers.

Production biomarker systems often encounter samples where one omics modality is unavailable. Cross-modal attention architectures can be trained with modality dropout, where entire source modalities are randomly masked during training. This forces the model to learn robust representations that degrade gracefully. At inference, a missing proteomics input can be replaced with a learned modality-agnostic placeholder embedding, allowing the cross-attention mechanism to fall back on available modalities without catastrophic failure.

The quadratic complexity of attention becomes prohibitive when jointly modeling tens of thousands of genes and proteins. Cross-modal chunking strategies divide each modality into overlapping or contiguous blocks, computing attention within and across chunks independently. Techniques like Performer or Linformer approximations further reduce the computational burden by projecting the attention matrix into a lower-dimensional space, enabling cross-modal attention to scale to whole-genome multi-omics datasets on commodity GPU hardware.

A deep learning architecture leveraging self-attention mechanisms to model long-range dependencies between tokens representing different molecular features. Unlike static integration, it dynamically weighs interactions across the full sequence context.

• Processes concatenated tokens from genomics, proteomics, and metabolomics
• Uses cross-attention between modality-specific encoders
• Captures non-linear, context-dependent molecular relationships

A neural network layer that concatenates or combines learned representations from separate modality-specific encoders into a single joint representation. This is the critical architectural component where cross-modal attention operates before downstream prediction.

• Early fusion: Combine raw or lightly processed features
• Intermediate fusion: Fuse latent representations from hidden layers
• Late fusion: Average or concatenate final modality-specific predictions

A non-linear extension of canonical correlation analysis using deep neural networks to learn maximally correlated complex transformations of two datasets. It serves as a statistical alternative to attention-based fusion.

• Learns maximally correlated subspaces without direct feature alignment
• Handles non-linear cross-omics associations
• Often used as a pre-training step before supervised biomarker prediction

A low-dimensional vector representation of a biological sample encoding information from multiple integrated omics data types. Cross-modal attention produces these unified molecular fingerprints by selectively attending to relevant features across modalities.

• Serves as input for downstream classification or survival models
• Enables transfer learning across related biomarker tasks
• Visualized using UMAP or t-SNE for patient stratification

A neural network operating directly on graph-structured data, used to model molecular interactions by propagating feature information across biological networks. It provides a structured alternative to free-form cross-modal attention.

• Nodes represent genes, proteins, or metabolites
• Edges encode protein-protein interactions or pathway relationships
• Message passing aggregates information from neighboring nodes across omics layers

A machine learning approach that combines multiple kernel functions, each representing a different omics data type or similarity measure, to learn an optimal composite kernel. It provides a mathematically principled, non-neural alternative to attention-based fusion.

• Each omics modality contributes a similarity kernel
• Learns optimal weights for kernel combination
• Often outperforms simple concatenation on small sample sizes