Positional Encoding

Every position in a sequence must map to a distinct encoding vector. If two positions share the same representation, the model cannot differentiate a TATA box at position -30 from one at position -200. In genomic models like Enformer, this property ensures that a transcription factor binding site's regulatory effect is correctly attributed to its exact genomic coordinate, preserving the spatial logic of cis-regulation.

Encoding values must remain within a stable numerical range regardless of sequence length. Unbounded values cause exploding gradients during training on long DNA sequences (e.g., 200kb inputs in Enformer). Sinusoidal encodings naturally satisfy this by confining values to [-1, 1], while learned absolute position embeddings require explicit normalization to prevent instability when processing entire chromosomes.

The encoding scheme must allow the attention mechanism to compute relative distances between tokens. In genomic context, this means the model should recognize that a CCAAT box and a GC box are separated by exactly 50 base pairs. Rotary Position Embeddings (RoPE) achieve this by encoding position as rotations, making the dot-product between query and key vectors depend only on their relative distance—critical for identifying periodic motifs like nucleosome positioning signals.

The encoding must generalize to sequence lengths unseen during training. A model trained on 512bp promoter regions should still function on 2kb enhancer sequences. ALiBi (Attention with Linear Biases) excels here by applying a fixed linear penalty to attention scores based on distance, requiring no learned parameters. This property is vital for genomic foundation models like Nucleotide Transformer that process variable-length inputs from short reads to full contigs.

The mapping from position index to encoding vector must be one-to-one and deterministic. For any two positions i and j, if i ≠ j, then encoding(i) ≠ encoding(j). This prevents positional collisions where a splice donor site and acceptor site become indistinguishable. Learned absolute position embeddings guarantee injectivity by design, storing a unique vector for each position up to the maximum sequence length.

Nearby positions should have similar encodings, reflecting the biological reality that adjacent nucleotides share regulatory context. A nucleotide at position 100 should be encoded more similarly to position 101 than position 10,000. Sinusoidal encodings with low-frequency dimensions provide this locality, enabling the model to learn that CpG islands and promoter-proximal elements cluster in contiguous genomic neighborhoods.

The original fixed encoding scheme from the Attention Is All You Need paper. It uses sine and cosine functions of different frequencies to generate unique position vectors without adding learnable parameters.

• Mechanism: Each dimension of the encoding corresponds to a sinusoid with a different wavelength, forming a geometric progression from 2π to 10000·2π.
• Key Property: Allows the model to easily learn to attend by relative positions, since the linear relationship between any two positions can be expressed as a linear function of the encoding.
• Genomic Application: Effective for capturing periodic regulatory motifs in DNA, such as the 10.5 bp helical repeat of transcription factor binding sites.

A widely adopted method that encodes position by rotating the query and key vectors in the attention mechanism, rather than adding information to the input embeddings.

• Mechanism: Applies a rotation matrix to the query and key vectors where the rotation angle is proportional to the token's absolute position.
• Key Advantage: The dot-product attention score naturally depends only on the relative distance between tokens, providing superior length generalization.
• Genomic Relevance: Used in models like DNABERT-2 to handle the extreme variability in intergenic distances and long-range enhancer-promoter interactions.

A straightforward approach where a unique, trainable vector is assigned to each absolute position in the sequence, treated identically to token embeddings during backpropagation.

• Mechanism: A standard embedding matrix of size [max_seq_len, d_model] is initialized randomly and updated via gradient descent.
• Limitation: Cannot extrapolate to sequence lengths unseen during training, which is a critical constraint for variable-length genomic scaffolds.
• Usage: Often employed in BERT-based genomic models where a fixed maximum context window, such as 512 or 2048 nucleotides, is predefined.

The core computational unit that positional encoding serves. It computes a weighted sum of values, where the weights are derived from the compatibility between a query and a set of keys.

• Self-Attention: Without positional encoding, this operation is permutation-invariant—shuffling the input tokens produces the identical output.
• Scaled Dot-Product: The standard formulation is Attention(Q,K,V) = softmax(QK^T / √d_k)V, where positional information is fused into Q and K.
• Genomic Context: Enables the model to integrate distal regulatory signals by directly comparing a promoter region token with an enhancer token 50,000 base pairs away.

An alternative paradigm that directly injects a learnable bias term into the attention logits based on the pairwise distance between tokens, bypassing input-level encoding entirely.

• T5 Bias: A scalar bias is learned for each discrete relative distance bucket, added to the pre-softmax attention scores.
• ALiBi: A non-learned, linearly decreasing penalty is applied to attention scores for distant tokens, providing strong length extrapolation.
• Genomic Utility: Naturally models the distance-dependent decay of regulatory interactions, such as the decreasing contact probability in Hi-C maps with linear genomic separation.

The preprocessing step that segments a raw DNA string into discrete units before positional encoding is applied. The choice of tokenizer fundamentally shapes the positional context.

• k-mer Tokenization: Overlapping k-mers (e.g., 3-mers, 6-mers) convert a sequence into a series of overlapping words, where position encodes the stride shift.
• Byte Pair Encoding (BPE): Adaptively merges frequent nucleotide subwords, creating a variable-length vocabulary that can capture common regulatory motifs as single tokens.
• Single Nucleotide: Treating each A, C, G, T as a token provides maximum resolution but requires the positional encoding to learn all motif-level syntax from scratch.