Rotary Position Embedding (RoPE)

RoPE encodes position by applying a rotation matrix to the query and key vectors in self-attention. The rotation angle is proportional to the token's absolute position. The dot product between a rotated query at position m and a rotated key at position n depends only on their relative distance m - n, not their absolute positions. This property emerges naturally from the trigonometric identity cos(m-n) = cos(m)cos(n) + sin(m)sin(n), making RoPE inherently relative without requiring explicit relative position biases or learned embeddings.

RoPE applies rotations at multiple frequencies across the embedding dimensions. Each consecutive pair of dimensions (2i, 2i+1) is rotated at a different frequency θ_i = 10000^{-2i/d}, where d is the head dimension.

• Low dimensions (high frequencies): Capture fine-grained, short-range positional relationships
• High dimensions (low frequencies): Encode coarse, long-range positional structure

This multi-resolution approach mirrors the sinusoidal position encoding design but integrates it directly into the attention computation rather than adding it to token embeddings.

A critical advantage of RoPE for genomics is its ability to generalize to sequence lengths unseen during training. Because the rotation is a continuous function of position, the model can extrapolate to longer sequences without requiring retraining or position interpolation.

• Linear scaling: Apply a scaling factor to the rotation frequencies to extend the context window
• NTK-aware scaling: Non-uniformly scale high frequencies less than low frequencies to preserve short-range resolution while extending long-range capacity
• YaRN (Yet another RoPE extensioN): Combines NTK-aware scaling with a temperature adjustment to the attention softmax

This property is essential for processing whole chromosomes or megabase-scale genomic loci that exceed typical training lengths.

RoPE induces a natural decay in attention weights as relative distance increases. The dot product between rotated query and key vectors includes a factor that diminishes with larger positional gaps. This creates an inductive bias where:

• Nearby tokens receive higher attention weights, reflecting local sequence context like splice junctions or codon boundaries
• Distant tokens receive progressively lower weights, but the decay is smooth rather than hard-coded

This learned locality bias is particularly valuable for genomic sequences where proximal regulatory elements (promoters, enhancers within a few kilobases) typically exert stronger influence than distal ones, while still allowing the model to capture long-range chromatin loops when the sequence signal warrants it.

RoPE is parameter-free—it requires no learned position embedding table, unlike absolute position embeddings or relative position biases.

• Memory savings: Eliminates the max_seq_len × d_model parameter matrix for position embeddings
• No lookup overhead: Rotation is computed on-the-fly with simple trigonometric operations
• Fused implementation: Modern attention kernels fuse the rotation into the linear projection step, making the overhead negligible compared to the attention computation itself

For genomic models processing variable-length sequences (from short promoter regions to multi-megabase loci), this parameter-free design avoids the need to pad or truncate to a fixed maximum length dictated by a learned embedding table.

RoPE has been adopted by several state-of-the-art genomic architectures:

• HyenaDNA: Uses RoPE in its implicit long-convolution layers to process up to 1 million nucleotides in a single forward pass
• Caduceus: Employs RoPE in its bidirectional Mamba-based architecture for whole-genome variant effect prediction
• Evo: Integrates RoPE into its StripedHyena architecture for modeling both DNA and protein sequences at scale

In these models, RoPE enables the attention mechanism to learn strand symmetry (the relative orientation of forward and reverse strands) and periodic patterns corresponding to helical turns of DNA (~10.5 bp per turn), which are fundamental structural properties of the double helix.

The original transformer method that adds fixed or learned sinusoidal vectors directly to token embeddings before the first attention layer. Each position in the sequence receives a unique encoding vector.

• Mechanism: PE(pos, 2i) = sin(pos/10000^(2i/d))
• Limitation: Fails to capture relative distance between tokens
• Genomic context: Used in early models like DNABERT but struggles with variable-length sequences
• Key weakness: Cannot extrapolate beyond the maximum sequence length seen during training

Encodes the pairwise distance between tokens directly into the attention computation rather than adding position information to input embeddings. The model learns to attend based on relative offsets.

• Approaches: Learned bias matrices, T5-style relative bias, Shaw et al. (2018)
• Advantage: Naturally handles sequences longer than training examples
• Genomic application: Critical for capturing enhancer-promoter interactions across variable genomic distances
• Trade-off: Adds computational overhead to the attention mechanism

A simple relative encoding method that adds a static, non-learned bias to attention scores that decreases linearly with distance. No positional embeddings are added to the input.

• Formula: Bias = -m * |i - j| where m is a head-specific slope
• Key property: Excellent length extrapolation without fine-tuning
• Genomic relevance: Enables processing of whole chromosomes without retraining
• Efficiency: Zero parameter overhead for position encoding

A straightforward approach where each position index is assigned a trainable embedding vector that is added to the token embedding. The model learns position representations from data.

• Implementation: Standard embedding lookup table of size max_len × d_model
• Limitation: Cannot generalize beyond the maximum trained position
• Genomic use case: Effective for fixed-length promoter region analysis
• Variants: Factorized or axial position embeddings for 2D genomic feature maps

A hybrid method that uses a learned continuous function to map relative positions to attention biases, combining the flexibility of learned approaches with the extrapolation benefits of relative encoding.

• Mechanism: A small MLP maps relative distance to a scalar bias
• Advantage: Smooth interpolation for unseen distances
• Genomic application: Handles variable-length intronic regions between exons
• Comparison to RoPE: Achieves similar extrapolation with different theoretical foundations

An enhancement to RoPE that introduces exponential decay in the rotation magnitude across dimensions, stabilizing attention scores for very long sequences and improving length generalization.

• Innovation: Combines rotation with dimension-dependent scaling
• Formula: Applies ζ_j = γ^(j/d) scaling to RoPE rotations
• Genomic significance: Enables whole-genome context windows exceeding 100k tokens
• Adoption: Used in state-of-the-art long-context models for DNA sequence modeling