Positional Encoding

The core self-attention mechanism in transformers is permutation-invariant—it computes weighted sums of values without any notion of token order. Without positional encoding, the sequence 'AGCT' and 'TCGA' would produce identical representations. Positional encoding breaks this symmetry by adding a unique signal to each position, allowing the model to distinguish upstream promoters from downstream enhancers and learn the sequential grammar of regulatory DNA.

The original transformer formulation uses sinusoidal functions of varying frequencies:

• Each position pos and dimension i receives sin(pos / 10000^(2i/d)) or cos(pos / 10000^(2i/d))
• The deterministic nature allows extrapolation to sequence lengths unseen during training
• The relative position can be expressed as a linear function of the encodings, enabling the model to attend to relative offsets naturally
• In genomics, this supports processing variable-length loci without retraining

Instead of fixed sinusoidal functions, many genomic models like DNABERT use learned position embeddings:

• A trainable embedding matrix of shape [max_seq_len, hidden_dim] is initialized randomly
• The model learns task-specific positional patterns during pre-training
• This captures genomic biases like the periodicity of nucleosome positioning or the 3-nucleotide periodicity of codons
• Limitation: Cannot extrapolate beyond the maximum training length without interpolation

RoPE encodes absolute position by rotating the query and key vectors in the attention computation. The rotation angle is a function of position, so the dot product between query and key naturally depends on their relative distance. This property is critical for genomic models that must generalize to longer sequences—RoPE's relative encoding allows length extrapolation without performance degradation, making it the preferred choice for models like HyenaDNA and recent genomic transformers.

Rather than adding position information to the input embeddings, relative positional bias modifies the attention scores directly:

• A learned scalar bias is added to the attention logit based on the relative distance between tokens
• This is used in architectures like T5 and Enformer variant implementations
• In genomics, this explicitly models that regulatory elements have distance-dependent interaction strengths—enhancers act over variable genomic distances
• The bias matrix can capture the decaying influence of sequence context with increasing nucleotide separation

DNA is double-stranded, and a regulatory element on the forward strand is functionally equivalent to its reverse complement on the reverse strand. Strand-aware positional encoding ensures:

• The position signal respects the 5' to 3' directionality of each strand
• Reverse complement augmentation is paired with position reversal to maintain biological consistency
• Models like the Nucleotide Transformer incorporate strand information to prevent the network from learning spurious directional artifacts in symmetric regulatory motifs