Self-Attention

Self-attention computes a relevance score between every pair of positions in a sequence. For each token, the model generates a Query (Q) vector representing what it's looking for, a Key (K) vector representing what it contains, and a Value (V) vector representing its actual information content. The dot product of the Query with all Keys produces an attention weight matrix that quantifies how much focus each position should place on every other position.

• Scaled Dot-Product: Scores are divided by √dₖ (where dₖ is the key dimension) to prevent vanishing gradients from large dot products in high dimensions.
• Softmax Normalization: Raw scores are converted to a probability distribution, ensuring all attention weights for a given query sum to 1.
• Weighted Sum: The final output for each position is a weighted sum of all Value vectors, where weights are the normalized attention scores.

Unlike convolutional or recurrent architectures, self-attention creates direct pathways between any two positions in a sequence regardless of their distance. A token at position 1 can directly influence the representation at position 10,000 in a single operation.

• Constant Path Length: The number of operations between any two positions is O(1), compared to O(n) for recurrent networks.
• Genomic Significance: This is critical for modeling enhancer-gene interactions where regulatory elements can be located up to 1 megabase away from their target promoters.
• Global Context: Every output position incorporates information from the entire input sequence, enabling the model to resolve ambiguities that require distal context.

Self-attention weights are computed purely from the content of the tokens themselves, not from their relative positions. The model learns to attend based on semantic or functional similarity rather than proximity.

• Dynamic Routing: The attention pattern changes for every input sequence, unlike convolutional filters which apply the same static kernel everywhere.
• Positional Encoding Required: Because self-attention is permutation-invariant, separate positional encodings (sinusoidal, learned, or rotary) must be added to input embeddings to inject sequence order information.
• Biological Relevance: This allows a genomic transformer to attend to a promoter region based on its sequence motif content, not just its distance from a gene body.

Self-attention computes all pairwise interactions simultaneously using matrix multiplication, making it highly efficient on GPU and TPU hardware. The entire attention matrix is computed in a single forward pass.

• Matrix Formulation: Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V, where Q, K, and V are matrices of all tokens' vectors packed together.
• Training Efficiency: This parallelism dramatically reduces training time compared to sequential RNNs, enabling the scaling of models to hundreds of billions of parameters.
• Inference Consideration: While training is parallelized, autoregressive decoding still requires sequential generation, though the KV-cache stores previously computed keys and values to avoid recomputation.

Rather than computing a single attention function, the multi-head attention mechanism projects Q, K, and V into h different lower-dimensional subspaces, allowing the model to jointly attend to information from different representation subspaces.

• Parallel Heads: Each head can learn a distinct relational pattern—one head might focus on codon usage bias, while another captures transcription factor binding motifs.
• Concatenation and Projection: The outputs of all heads are concatenated and linearly projected back to the model dimension.
• Interpretability: Individual attention heads often exhibit interpretable behaviors, such as attending to syntactically or functionally related positions, making them valuable for motif discovery and biological hypothesis generation.

The primary limitation of full self-attention is its O(n²) memory and time complexity with respect to sequence length n, as it computes and stores an n×n attention matrix.

• Genomic Bottleneck: The human genome contains over 3 billion base pairs, making naive full self-attention computationally intractable for whole-genome analysis.
• Mitigation Strategies: Sparse attention patterns (local windows, dilated sliding windows, or global tokens), linearized attention approximations, and alternative architectures like State Space Models (SSMs) reduce complexity to O(n log n) or O(n).
• Practical Limits: Even with optimizations, most genomic transformers operate on sequence contexts of 1,000 to 100,000 tokens, requiring careful DNA tokenization strategies to maximize information density per token.