Attention Mechanism

The foundational calculation where a query vector is compared against all key vectors to produce attention weights, which are then used to compute a weighted sum of value vectors. The scaling factor (1/√dₖ) prevents the dot products from growing too large, which would push the softmax function into regions of extremely small gradients. This operation is computed in parallel across all positions using matrix multiplication, making it highly efficient on GPUs.

Instead of performing a single attention function, the model projects the queries, keys, and values into h different subspaces (typically 8 or 16 heads). Each head can learn to attend to different types of relationships:

• One head might focus on proximal regulatory elements
• Another head might capture long-range enhancer-promoter interactions
• A third head might attend to co-expressed gene modules

The outputs are concatenated and projected back to the original dimension, allowing the model to jointly integrate information from multiple representation subspaces.

Self-attention computes relationships within a single sequence—every position attends to every other position in the same input. In genomics, this allows a gene's representation to be updated based on all other genes in the profile. Cross-attention mixes information between two different sequences, such as attending from a drug molecule representation to a protein target representation. This distinction is critical for multi-omics integration, where cross-attention can align cells across transcriptomic and proteomic modalities without requiring paired training data.

Attention is inherently permutation-invariant—it has no built-in notion of sequence order. Positional encodings inject information about the position of each token. In genomic applications, this can be extended beyond simple position to encode:

• Genomic coordinates (chromosome and base-pair location)
• Strand orientation
• Epigenomic context (chromatin accessibility, histone marks)

Sinusoidal encodings provide a deterministic pattern, while learned positional embeddings allow the model to discover task-specific positional features during training.

A key advantage for bioinformatics: attention weights can be extracted and visualized to understand which input elements the model considers important for a prediction. For example, when predicting a disease phenotype from multi-omics data, the attention matrix can reveal which genes, pathways, or regulatory regions drove the decision. This provides a form of built-in feature attribution that is more granular than post-hoc explanation methods like SHAP, though care must be taken as high attention weight does not always equate to high feature importance.

Standard attention scales quadratically with sequence length, which is prohibitive for whole-genome or single-cell datasets with tens of thousands of features. Efficient variants address this:

• Linformer: Projects the key-value dimension to a fixed low-rank representation, achieving linear complexity
• Performer: Uses kernel-based approximations to avoid computing the full attention matrix
• BigBird: Combines local, global, and random attention patterns for sparse coverage These enable attention-based models to scale to the high-dimensional, long-sequence data typical in multi-omics integration.

The foundational operation where a sequence computes a weighted representation of itself. Each token generates a Query (Q), Key (K), and Value (V) vector. The attention weights are derived from the scaled dot-product of Q and K, determining how much each token attends to every other token. This allows the model to capture long-range dependencies without the sequential bottleneck of recurrent architectures.

An extension of self-attention that runs multiple attention operations in parallel across different learned linear projections of Q, K, and V. Each attention head can specialize in a different type of relationship:

• Syntactic heads: Capture grammatical structure
• Semantic heads: Capture meaning and context
• Long-range heads: Focus on distant token interactions

The outputs are concatenated and projected, enabling the model to jointly attend to information from different representation subspaces.

The specific mathematical formulation used in the Transformer. The attention score between a query and a key is computed as: Attention(Q,K,V) = softmax(QK^T / √d_k)V. The scaling factor √d_k prevents the dot products from growing too large in magnitude, which would push the softmax function into regions of extremely small gradients, hindering training stability.

A variant where the Queries originate from one sequence (e.g., a decoder's target sequence) while the Keys and Values originate from another (e.g., an encoder's source representation). This is the core mechanism in sequence-to-sequence models, allowing the decoder to dynamically focus on relevant parts of the input when generating each output token. In multi-omics, this enables aligning gene expression data with protein interaction networks.

Since self-attention is inherently permutation-invariant—it has no notion of token order—positional encodings are added to input embeddings to inject sequence order information. The original Transformer uses sinusoidal functions of varying frequencies. Alternatives include learned positional embeddings and relative positional encodings (e.g., Rotary Position Embedding, RoPE), which encode the relative distance between token pairs directly in the attention computation.

The N × N matrix of pairwise attention weights computed for a sequence of length N. Each row represents how much a token attends to every other token. This matrix is often visualized as a heatmap to interpret model behavior. In biological applications, attention patterns can reveal gene regulatory interactions or protein binding sites, providing a form of mechanistic interpretability for multi-omics models.