Rotary Position Embedding RF

RoPE encodes position by applying a phase rotation to the query and key vectors in the attention mechanism. The rotation angle is a function of the absolute position, but the dot product between a rotated query and key depends only on their relative position difference. This is achieved by representing each dimension pair as a complex number and multiplying by e^(i·m·θ), where m is the position and θ is a frequency basis.

• The attention score q_m^T k_n becomes a function of (m - n) only
• Naturally captures temporal offsets between signal samples
• Avoids the absolute position bias of learned embeddings

RoPE's mathematical formulation aligns perfectly with complex baseband representations of RF signals. Since IQ samples are inherently complex numbers with magnitude and phase, RoPE can be applied directly without artificial decomposition. The rotation operation corresponds to a frequency shift in the baseband signal.

• Each complex token x = I + jQ is rotated by e^(j·m·θ)
• Preserves the magnitude while shifting the phase
• Matches the physics of frequency translation and Doppler effects
• Enables the model to learn translation-invariant features in the time-frequency domain

RoPE uses a set of geometrically spaced frequencies θ_i = base^(-2i/d) that create a multi-scale positional representation. Higher frequency components capture fine-grained local structure, while lower frequencies model long-range dependencies. This induces a natural decay in attention scores as the relative distance increases.

• Similar to the Fourier basis used in signal processing
• Long-range decay is controlled by the base hyperparameter (typically 10,000)
• Can be tuned to match the coherence time of the wireless channel
• Provides a smooth inductive bias for temporal locality in signal sequences

Unlike learned absolute position embeddings, RoPE can generalize to sequence lengths not seen during training. By adjusting the frequency basis or applying NTK-aware scaling, the rotary embeddings can be extended to longer contexts without retraining. This is critical for processing extended RF captures.

• Supports dynamic sequence length inference
• NTK-aware interpolation rescales frequencies: θ_i' = θ_i · s^(-2i/d)
• Enables processing of variable-duration signal bursts
• No need to pad or truncate to a fixed training length

RoPE is applied directly to the query and key projections before the attention dot product in each head. For RF applications, this means each attention head can learn to attend to different frequency offsets or time lags simultaneously. The rotation is applied per-head, allowing diverse positional relationships to be captured.

• Applied as: q' = RoPE(q), k' = RoPE(k) before softmax(q'k'^T / √d)
• Each head can specialize in different delay spreads or Doppler shifts
• Compatible with causal masking for real-time streaming inference
• Preserves the linear complexity of standard attention

Because RoPE operates through multiplication in the complex domain, it preserves the phase coherence between tokens that is essential for RF signal processing. The relative rotation between two tokens encodes their temporal separation while maintaining the inter-symbol phase relationships critical for modulation recognition and channel equalization.

• Maintains coherent combining of multipath components
• Preserves phase continuity across token boundaries
• Enables the model to learn delay-and-sum beamforming-like operations
• Critical for tasks like direction of arrival estimation and equalization

An extension of the standard attention mechanism that operates natively on complex numbers, preserving the magnitude and phase relationships inherent in IQ baseband signals. Unlike real-valued attention, which processes in-phase (I) and quadrature (Q) components as separate real channels, complex-valued attention computes dot products in the complex domain. This is the foundational operation that RoPE enhances by applying rotations in the complex plane to encode relative position, making the pairing particularly expressive for physical-layer signal processing.

A transformer model adapted to process raw in-phase and quadrature (IQ) sample sequences directly. Key design choices include:

• Tokenization: How to chunk the complex time-series into discrete tokens
• Positional encoding: Where RoPE provides a principled method by rotating each token's complex embedding proportionally to its time index
• Attention masking: Often causal for real-time streaming applications

The IQ Transformer is the primary architecture where RoPE for RF is deployed, replacing traditional signal processing chains with a single learned model.

A method for injecting positional information into a transformer by encoding the frequency index of each spectral token. While standard positional encodings use sinusoidal functions of the token index, RoPE generalizes this by applying rotations in the complex plane whose frequency is tied to the token's position. For RF applications, this means the model can learn to attend to tokens based on their relative frequency offset, naturally capturing harmonic relationships and spectral correlations without requiring absolute position memorization.

A preprocessing module that converts a raw time-series signal into a sequence of tokens representing localized time-frequency patches. This enables a standard transformer backbone to process spectral content efficiently. When combined with RoPE, the tokenizer can embed two-dimensional positional information—both temporal offset and frequency offset—by applying independent rotations in separate subspaces of the complex embedding. This 2D RoPE variant allows the attention mechanism to be sensitive to both when and at what frequency a signal feature occurred.

An attention masking pattern that restricts a transformer model to only attend to past and present time steps, making it suitable for real-time, streaming signal processing tasks. RoPE's relative position encoding is particularly advantageous here because:

• The relative time offset between any two tokens is encoded directly in their dot-product attention score
• No absolute position indices need to be recomputed when the sequence length grows
• The model naturally generalizes to sequence lengths unseen during training, critical for continuous streaming RF applications like adaptive equalization and interference cancellation.

An adaptation of the Vision Transformer (ViT) that treats a spectrogram image as a grid of patches, applying self-attention to learn spatial and temporal features. When RoPE is applied to this architecture, each patch's 2D position (time bin, frequency bin) is encoded via independent rotations in orthogonal subspaces of the query and key vectors. This allows the model to learn translation-equivariant features in the time-frequency plane, improving generalization for tasks like emitter identification and modulation classification across varying signal parameters.