Spectrum Transformer

The critical preprocessing step that converts a raw time-series signal or spectrogram into a sequence of discrete tokens suitable for a transformer backbone. A patchified spectrogram divides the 2D time-frequency representation into a grid of non-overlapping patches, each flattened into a token vector. Alternatively, a Time-Frequency Tokenizer applies learned convolutional projections to localized time-frequency patches, preserving spectral locality while creating a compact, information-dense token sequence. This tokenization enables the self-attention mechanism to operate on manageable sequence lengths while retaining the structural integrity of the frequency-domain data.

Standard positional encodings are insufficient for spectral data because they do not capture the ordered, physical meaning of frequency bins. Frequency-Domain Positional Encoding injects positional information by encoding the frequency index of each spectral token, allowing the model to understand the ordering of subcarriers or frequency bins. A more advanced variant, Rotary Position Embedding RF (RoPE), encodes relative temporal or frequency offsets through rotation in the complex plane, making it particularly well-suited for complex-valued signal representations where phase relationships must be preserved.

The application of multi-head self-attention directly to spectrum data allows the model to jointly attend to information from different frequency sub-bands and time slots simultaneously. Each attention head can learn to capture distinct correlation patterns:

• Inter-frequency correlations: Identifying harmonics or spectral masks
• Temporal dependencies: Tracking signal evolution over time
• Cross-band interference: Detecting interactions between adjacent channels This mechanism replaces hand-crafted feature extractors with a learned, data-driven approach to discovering relevant signal structure.

For real-time, streaming signal processing tasks where future samples are unavailable, Causal Temporal Attention applies an attention masking pattern that restricts the model to only attend to past and present time steps. This autoregressive constraint transforms the transformer into a causal sequence model suitable for:

• Real-time anomaly detection in spectrum monitoring
• Online modulation classification
• Streaming interference mitigation
• Low-latency cognitive radio decision engines The causal mask ensures the model never peeks into the future, maintaining strict causality for deployment in live signal processing pipelines.

A powerful mechanism that uses cross-attention to fuse information from two distinct signal representations. Common fusion patterns include:

• Time-Frequency Fusion: Combining time-domain waveform features with frequency-domain spectral features for richer representations
• Multi-Sensor Fusion: Integrating outputs from multiple antenna elements or sensor modalities
• Multi-Resolution Fusion: Merging features extracted at different spectrogram resolutions Cross-attention allows one representation to serve as the query while the other provides keys and values, enabling the model to dynamically align and integrate heterogeneous signal information.

Masked Spectrum Modeling is a self-supervised pre-training technique where random portions of a spectrogram or frequency-domain sequence are masked, and the transformer is trained to reconstruct the missing content. This approach:

• Learns robust, generalizable representations of signal structure without labeled data
• Captures the statistical regularities of the electromagnetic environment
• Enables effective fine-tuning on downstream tasks with limited labeled samples
• Functions analogously to masked language modeling in NLP, but applied to the spectral domain This technique is particularly valuable in RF domains where labeled data is scarce and expensive to collect.

The critical preprocessing bridge between raw IQ samples and the transformer backbone. This module converts a continuous time-series signal into a sequence of discrete tokens by segmenting a spectrogram into localized time-frequency patches.

• Patchification: Divides the 2D spectrogram into a grid of non-overlapping patches, each flattened into a 1D token vector.
• Learned Embedding: A linear projection maps each flattened patch to the transformer's hidden dimension.
• Preservation of Structure: Unlike 1D tokenization of raw samples, this retains the 2D locality of spectral energy, allowing the attention mechanism to learn correlations across both time and frequency axes.

Standard sinusoidal positional encodings are insufficient for spectral data. This method injects structural information by encoding the absolute frequency index of each token, allowing the model to understand subcarrier ordering or frequency bin relationships.

• Spectral Order: Encodes the center frequency of each token, enabling the model to distinguish between low-band and high-band features.
• Learnable Variants: Often implemented as a learnable embedding matrix indexed by frequency bin, jointly optimized during training.
• Rotary Position Embedding (RoPE): A natural fit for complex-valued RF data, encoding relative frequency offsets through rotation in the complex plane, preserving phase relationships.

A self-supervised pre-training strategy directly analogous to Masked Image Modeling (MIM) in vision. Large portions of a spectrogram are randomly masked, and the transformer is trained to reconstruct the missing time-frequency content.

• Pretext Task: Forces the model to learn the statistical structure of legitimate signals, including cyclostationary patterns and modulation signatures.
• Downstream Transfer: The pre-trained encoder provides a robust initialization for supervised tasks like signal classification or anomaly detection, especially in data-scarce environments.
• Masking Strategies: Random patch masking, block masking (to simulate interference), or frequency-band masking are used to create challenging reconstruction targets.

A mechanism for fusing heterogeneous signal representations by using one modality as the query and another as the key/value in a cross-attention block. This is essential for multi-modal RF processing.

• Time-Frequency Fusion: Fuses features from a 1D temporal convolutional stream with a 2D spectrogram transformer stream, allowing the model to leverage both raw waveform transients and spectral texture.
• Multi-Sensor Fusion: Combines outputs from different antenna polarizations or separate sensing receivers, where cross-attention aligns features across modalities before a joint classification head.
• Conditional Processing: A guidance signal (e.g., a known transmission protocol) can attend to the spectral tokens to condition the decoding process.

An alternative paradigm that models the spectrum as a graph rather than a grid. Nodes represent transmitters or frequency bands, and edges represent interference or spatial correlation.

• Message Passing: Nodes iteratively exchange information with neighbors to learn a global representation of the spectral environment, naturally handling irregular spatial deployments.
• Interference Graph Construction: Edges are weighted by the mutual interference between transmitter-receiver pairs, serving as input to a GNN for distributed power control and link scheduling.
• Hybrid Architectures: Graph layers can be combined with transformer layers, where the GNN handles spatial relationships between nodes and the transformer processes the temporal sequence of each node's spectral state.

A masking pattern that restricts the self-attention mechanism to only attend to past and present time steps. This is non-negotiable for real-time, streaming signal processing where future samples are physically unavailable.

• Autoregressive Decoding: Enables the Spectrum Transformer to operate as a streaming predictor for tasks like real-time interference prediction or sequential signal detection.
• Mask Implementation: A lower-triangular mask is applied to the attention scores, zeroing out weights for future tokens.
• Latency Constraints: Often paired with efficient attention variants (e.g., flash attention) to meet the strict sub-millisecond deadlines of physical-layer processing.