Multi-Head Spectrum Attention

The input spectrum representation is projected into h distinct query, key, and value subspaces. Each head computes attention independently, allowing the model to focus on different types of spectral relationships simultaneously.

• Head 1 might attend to adjacent frequency bins for local interference patterns
• Head 2 could capture harmonic relationships across widely separated bands
• Head 3 might focus on temporal evolution of specific carriers

The outputs are concatenated and linearly projected, preserving the dimensionality of the original input while enriching the representation with multi-faceted spectral context.

Unlike convolutional approaches that are limited by kernel size, multi-head attention computes pairwise interactions between every time-frequency bin in the input sequence. This global receptive field is critical for spectrum analysis.

• Captures non-adjacent carrier aggregation patterns
• Identifies cyclostationary features spanning distant time steps
• Detects cross-band interference that local filters would miss

The attention matrix explicitly models which frequency components at which time steps are most relevant to each other, providing a learned spectral correlation map.

Standard attention operates on real-valued vectors, but spectrum data is inherently complex-valued (IQ samples). Multi-head spectrum attention can be extended to operate natively in the complex domain.

• Attention weights computed using Hermitian inner products preserve phase relationships
• Rotary Position Embedding (RoPE) applies rotations in the complex plane for relative frequency encoding
• Magnitude and phase are processed jointly rather than as separate real channels

This preserves the quadrature relationship between I and Q components that is essential for modulation recognition and channel estimation tasks.

For real-time spectrum monitoring applications, a causal attention mask restricts each time step to attend only to past and present tokens. This enables streaming processing without requiring future samples.

• Supports online signal classification with bounded latency
• Enables predictive spectrum occupancy forecasting
• Maintains temporal causality for cognitive radio decision engines

The same pre-trained model can be used for both offline analysis (full bidirectional attention) and online deployment (causal masking) by simply toggling the attention mask pattern.

Multi-head cross-attention enables the fusion of heterogeneous spectrum representations. One modality provides the query, while another provides the keys and values.

• Fuse time-domain IQ samples with frequency-domain spectrograms
• Combine outputs from multiple antenna elements in MIMO systems
• Integrate metadata (carrier frequency, bandwidth) with raw signal data

This mechanism allows the model to learn which aspects of one representation are most informative for interpreting another, creating a unified spectral understanding.

The core attention computation uses scaled dot-product attention with an optional learnable frequency bias added to the attention logits before softmax.

• Learnable frequency-distance bias: Tokens closer in frequency space receive higher initial attention weights
• Band-type embeddings: Different spectral bands (licensed, unlicensed, guard) receive distinct learnable biases
• Attention scaling: The scaling factor 1/√d_k prevents gradient saturation in high-dimensional spectrum token representations

This inductive bias encodes the prior knowledge that nearby frequencies are more likely to be correlated, accelerating convergence while still allowing the model to discover long-range dependencies.

The mathematical backbone of native RF attention. Unlike standard attention operating on real numbers, this mechanism preserves the magnitude and phase relationships inherent in IQ baseband signals. By treating the Key, Query, and Value vectors as complex numbers, the dot-product attention calculation naturally captures phase rotation and coherent interference patterns that would be lost if the signal were split into separate real and imaginary channels. This allows a single attention head to learn filters that are sensitive to specific phase offsets, which is critical for modulation recognition.

The preprocessing module that bridges raw waveform data and the transformer backbone. This component converts a continuous stream of IQ samples into a sequence of discrete tokens representing localized time-frequency patches. Common implementations include:

• Patchified Spectrogram: Dividing a spectrogram image into 2D grid patches, flattening each into a token vector.
• Convolutional Tokenizer: Using a learned 2D convolution to project overlapping time-frequency windows into embedding space.
• Delay-Doppler Embedding: Encoding propagation path characteristics directly as tokens for channel processing. The tokenizer's stride and resolution define the model's receptive field and computational granularity.

Standard positional encodings assume sequential order. For spectrum data, this specialized encoding injects information about the absolute frequency index of each token. This allows the model to understand the ordering of subcarriers or frequency bins, which is essential for learning frequency-dependent phenomena like selective fading or adjacent channel interference. Rotary Position Embedding (RoPE) is particularly well-suited here, as its rotation-based encoding naturally aligns with the complex-plane representation of frequency shifts, encoding relative frequency offsets directly into the attention calculation.

A mechanism for integrating heterogeneous signal representations within the attention framework. Instead of self-attention, one sequence (e.g., time-domain tokens) generates the Query, while another sequence (e.g., frequency-domain tokens) generates the Key and Value. This allows the model to explicitly fuse information from multiple sensor modalities or dual-domain representations. For example, a model can use cross-attention to correlate a raw IQ stream with a simultaneously captured spectrogram, learning joint representations that are more robust to noise than either domain alone.

An attention masking pattern that restricts the self-attention mechanism to only attend to past and present time steps. This is critical for real-time, streaming signal processing applications—such as cognitive radio decision engines or neural receivers—where future samples are unavailable. By applying a lower-triangular mask to the attention matrix, the model prevents information leakage from the future, ensuring that predictions at time t depend only on observations up to time t. This transforms the transformer from a bidirectional context aggregator into a causal, autoregressive sequence processor.

A discrete, learnable token representing an individual multipath component in a wireless channel. Characterized by its delay, Doppler shift, and complex gain, each path is embedded into a vector that serves as input to a transformer. This tokenization strategy allows a model like a Transformer Channel Estimator to process a wireless channel as an unordered set of paths, using self-attention to model the interactions between them. This is a powerful alternative to grid-based channel representations, naturally handling sparse scattering environments common in mmWave and sub-THz systems.