Rotary Position Embedding (RoPE)

RoPE encodes absolute position information by rotating query and key vectors by multiples of a frequency matrix. Critically, the dot-product between a rotated query and key depends only on their relative position difference, not their absolute positions.

• The rotation angle is proportional to the token's position index
• Attention score naturally captures relative distance
• Eliminates the need for separate relative position bias terms

RoPE formulates position encoding as a rotation in complex vector space. Each dimension pair in the embedding is treated as the real and imaginary parts of a complex number, then multiplied by a rotation factor e^(i·m·θ) where m is position and θ is a frequency.

• Uses Sinusoidal frequencies θ_i = 10000^(-2i/d)
• Rotation is applied to queries and keys before attention
• Preserves vector norm, maintaining embedding magnitude stability

A defining advantage of RoPE is the ability to extrapolate to longer sequences than those seen during training. Because the rotation is a continuous function of position, the model can generalize to novel position indices.

• Performance degrades gracefully beyond training length
• Enables processing of megabase-scale genomic sequences
• Critical for whole-genome context in DNA language models
• Supports techniques like NTK-aware scaling for further extension

RoPE is implemented as an element-wise operation on the query and key tensors before the attention matrix computation. It adds negligible overhead compared to learned absolute position embeddings.

• No additional parameters to learn or store
• Easily vectorized and fused into attention kernels
• Compatible with FlashAttention for memory-efficient training
• Applied once per layer, not per attention head individually

RoPE naturally induces a long-range decay property in attention scores: as the relative distance between two tokens increases, the expected attention weight decreases. This mirrors biological reality where proximal regulatory elements often have stronger interactions.

• Decay rate is controlled by frequency parameters θ_i
• Higher frequencies decay faster, lower frequencies persist longer
• Provides an inductive bias favoring local genomic context
• Still allows strong long-range enhancer-promoter connections

In DNA language models like those using the HyenaDNA or Enformer architectures, RoPE is applied to the query and key projections within self-attention layers to handle sequences spanning hundreds of thousands of base pairs.

• Enables modeling of distal regulatory elements
• Works with both standard attention and subquadratic operators
• Preserves the model's ability to learn strand symmetry
• Often combined with reverse complement augmentation for DNA-specific inductive biases