Cross-Attention Spectrum Fusion

Unlike self-attention, cross-attention uses queries generated from one modality (e.g., time-domain IQ samples) to selectively retrieve relevant information from keys and values of a second modality (e.g., a frequency-domain spectrogram). This asymmetry allows one representation to act as a dynamic filter for the other, creating a learned soft-alignment between heterogeneous signal views without requiring manual feature engineering or rigid synchronization.

The attention matrix inherently performs a content-dependent gating function. When fusing a high-resolution time-domain stream with a lower-resolution spectral feature map, the cross-attention weights can learn to:

• Amplify transient time-domain events that correlate with specific frequency signatures
• Suppress noise components that lack cross-modal corroboration This results in a fused representation that is more robust than either input alone, particularly in low-SNR environments.

Cross-attention operates on sequences of token vectors, abstracting away the native structure of each modality. A raw IQ stream tokenized via a Time-Frequency Tokenizer and a spectrogram processed by a Patchified Spectrogram encoder can be fused seamlessly. This token-level interface enables the fusion of arbitrary signal representations—including Delay-Doppler Embeddings or Propagation Path Tokens—within a single unified architecture.

During backpropagation, the cross-attention mechanism creates a bottleneck for gradient flow between the two modality-specific encoders. The query-generating encoder receives gradients only through the attention weights, not through the value pathways. This architectural property allows for asynchronous pre-training: each encoder can be pre-trained independently (e.g., using Masked Spectrum Modeling on one modality) before fine-tuning the fusion layer end-to-end.

Cross-attention can fuse representations with mismatched temporal or frequency resolutions without resampling. A short time-domain burst can attend to a long spectral history, or a single spectral snapshot can query a sequence of temporal tokens. The attention mechanism implicitly handles the alignment, learning which time-frequency correspondences are salient for the task—critical for fusing Channel State Information Transformers with raw waveform data in massive MIMO systems.

The cross-attention weight matrices provide a built-in explainability mechanism. By visualizing which tokens from modality B received high attention weights when processing a specific token from modality A, engineers can identify which time-frequency regions drove a classification or detection decision. This is essential for Explainable RF AI applications in mission-critical spectrum sensing, where operators must validate that a signal classifier is attending to physically meaningful features rather than spurious correlations.

An extension of the standard attention mechanism that operates natively on complex numbers, preserving the magnitude and phase relationships inherent in IQ baseband signals. This is critical for cross-attention fusion because it allows the model to learn correlations between the real and imaginary parts of two distinct signal representations without losing the physical significance of the phase information. Standard real-valued attention would treat the I and Q components as separate, unrelated features, destroying the complex structure.

A preprocessing module that converts a raw time-series signal into a sequence of tokens representing localized time-frequency patches. This tokenizer is the essential bridge that enables a standard transformer backbone to perform cross-attention between time-domain and frequency-domain representations. By projecting both modalities into a shared token space, the model can compute attention weights that reveal how a specific temporal event corresponds to a specific spectral component.

An attention mechanism that simultaneously models dependencies across spatial dimensions (e.g., antenna elements) and temporal dimensions (e.g., symbol periods) in a multi-antenna signal. When used in a cross-attention context, this allows the model to fuse information from different sensor modalities—such as fusing radar and communication waveforms—by attending to correlations across both space and time in a single, unified operation.

A discrete, learnable token representing an individual multipath component, characterized by its delay, Doppler shift, and complex gain. In a cross-attention spectrum fusion architecture, these tokens can serve as a shared latent representation. One branch of the model might encode raw IQ samples into path tokens, while another encodes a spectrogram; cross-attention between these token sets allows the model to resolve ambiguities by fusing time-domain multipath structure with frequency-domain energy distribution.

An adaptation of the Vision Transformer (ViT) that treats a spectrogram image as a grid of patches, applying self-attention to learn spatial and temporal features. This architecture is a natural candidate for one branch of a cross-attention fusion system. Its patchified output tokens can be cross-attended with tokens from a raw IQ transformer branch, allowing the model to jointly reason about high-level spectral shapes and low-level waveform transients.

The application of Rotary Position Embedding (RoPE) to RF signal tokens, encoding relative temporal or frequency offsets through rotation in the complex plane. This is particularly well-suited for cross-attention fusion because it provides a mathematically consistent way to encode positional information across different signal representations. A time-domain token and a frequency-domain token representing the same event can be encoded with relative positional rotations that are consistent in both domains.