Multi-Head Attention

Instead of computing a single attention function, multi-head attention linearly projects the queries, keys, and values h times with different, learned linear projections. Each of these h parallel attention heads operates in a lower-dimensional subspace, allowing the model to focus on different types of relationships simultaneously. In a genomic context, one head might learn to attend to GC-rich promoter regions while another simultaneously tracks splice donor/acceptor site syntax.

Each head computes attention using the formula: Attention(Q, K, V) = softmax(QK^T / √d_k)V. The scaling factor 1/√d_k prevents the dot products from growing too large in magnitude, which would push the softmax function into regions with extremely small gradients. This is critical for stable training on long genomic sequences where the dimensionality of the key vectors can be substantial.

The outputs of all h attention heads are concatenated and multiplied by a final output weight matrix W^O. This step synthesizes the diverse information captured by each head into a unified representation. For a genomic language model, this means the independent signals for enhancer activity, nucleosome positioning, and evolutionary conservation are fused into a single, context-rich nucleotide embedding for downstream prediction tasks.

Analysis of attention head behavior in models like Enformer and DNABERT reveals that individual heads often specialize in distinct biological functions without explicit supervision. Common emergent specializations include:

• Attending to transcription factor binding motifs
• Tracking long-range enhancer-promoter interactions
• Identifying open reading frame boundaries
• Detecting splice junction consensus sequences

The time and memory complexity of multi-head attention is O(n² · d) where n is sequence length and d is the model dimension. While the number of heads h increases the parameter count, the per-head dimension is typically reduced to d/h, keeping the total computational cost roughly constant compared to single-head attention with full dimensionality. This makes the mechanism parameter-efficient while dramatically increasing representational capacity.

In decoder-style architectures, a cross-attention variant of multi-head attention is used where the queries come from the decoder but the keys and values come from an encoder output. In genomics, this enables tasks like gene expression imputation, where a model can attend from a sparse set of measured genes (queries) to a full reference genome embedding (keys/values) to predict unmeasured transcript levels.

The foundational mechanism where a sequence computes a weighted representation of itself. For a genomic sequence, every nucleotide position generates a query, key, and value vector. The attention weight between position i and j is the scaled dot-product of the query at i and key at j, determining how much of position j's information is aggregated into the output at i. This allows the model to directly connect a distal enhancer to a promoter 50,000 base pairs away without sequential processing.

A visualization tool that renders the attention weight matrix as a heatmap, where the intensity of a cell (i, j) represents the model's focus from position i to j. In genomic models, these heatmaps are used as an interpretability method to validate that specific heads are attending to biologically relevant features, such as transcription factor binding motifs or splice junctions. A head consistently showing high attention at a known motif location provides evidence that the model has learned the regulatory grammar.

The dominant self-supervised pre-training objective for genomic BERT-like models. During training, a random percentage of input tokens (e.g., k-mers) are replaced with a [MASK] token. The multi-head attention mechanism must then predict the original nucleotide sequence from the unmasked bi-directional context. This forces the model to learn fundamental biological signals like sequence conservation and codon usage bias without requiring any manually labeled data.

A computational assay that leverages a trained multi-head attention model to predict the functional impact of every possible single-nucleotide variant. By systematically mutating each position in an input sequence and measuring the change in the model's predicted probability for the reference allele, one can generate a mutation effect map. A significant drop in probability indicates a potentially pathogenic variant, and the attention heads can be analyzed to see which surrounding regulatory elements were disrupted by the mutation.

An efficient approximation of full self-attention designed to overcome the quadratic memory cost (O(L²)) of processing long genomic sequences. Instead of attending to every position, each token attends to a restricted subset:

• Local Window: Attends only to neighboring tokens within a fixed range.
• Dilated Sliding Window: Attends with gaps to increase receptive field.
• Global Tokens: Designated tokens that attend to the entire sequence. This enables transformers to process whole chromosomes or megabase-scale loci that are intractable for dense attention.

A task where the attention patterns of a protein language model are analyzed to determine which pairs of amino acid residues are in close spatial proximity in the folded 3D structure. The underlying principle is that co-varying residues that must maintain physical contact during evolution will have high mutual attention weights. A dedicated contact prediction head can be fine-tuned on top of the multi-head attention outputs, transforming the learned sequence grammar into a residue-residue distance map foundational to de novo structure prediction.