Spectrogram Vision Transformer (Spectrogram ViT)

The spectrogram is divided into a grid of non-overlapping 2D patches, each flattened into a vector and linearly projected into a fixed-dimensional embedding space. Unlike CNNs that process local pixel neighborhoods, this tokenization treats each time-frequency tile as a discrete semantic unit. A learnable positional encoding is added to preserve the sequential and spectral order of patches, enabling the model to distinguish between a signal at 900 MHz early in time versus the same frequency later. This approach converts raw STFT magnitudes into a sequence suitable for transformer processing.

The core mechanism computes pairwise attention scores between every patch in the spectrogram, allowing the model to directly relate distant temporal events and disparate frequency components. A signal hop at 2.4 GHz can be directly associated with its precursor at 900 MHz, even if separated by seconds. Each attention head specializes in a different relational pattern:

• Temporal heads: Track signal evolution and dwell time
• Spectral heads: Correlate harmonics and intermodulation products
• Cross heads: Link time-frequency signatures of specific emitters This global receptive field is critical for classifying complex, agile waveforms.

A special learnable [CLS] token is prepended to the patch sequence. After passing through all transformer encoder layers, this token's final hidden state serves as a global representation of the entire spectrogram for classification. Alternative pooling strategies include:

• Mean pooling: Averages all patch embeddings for a more distributed representation
• Attention pooling: Learns a weighted combination of patch outputs
• Max pooling: Captures the most salient activations The [CLS] token approach, inherited from BERT, concentrates task-relevant information during training through the self-attention mechanism itself.

Standard ViT uses fixed-resolution patches, but hierarchical architectures like the Swin Transformer compute self-attention within local windows and progressively merge patches in deeper layers. This creates a multi-scale feature pyramid analogous to CNNs:

• Early layers: Fine-grained time-frequency details (transients, micro-Doppler)
• Middle layers: Intermediate structures (modulation patterns, sweeps)
• Late layers: Global context (emitter type, protocol identification) Shifted window partitioning alternates window boundaries between layers to enable cross-window connections without the quadratic cost of full global attention, making it efficient for high-resolution spectrograms.

Inspired by masked autoencoders, this self-supervised pre-training strategy randomly masks a high proportion of spectrogram patches (e.g., 75%) and trains the model to reconstruct the missing time-frequency content. The asymmetric encoder-decoder design processes only visible patches in the encoder, dramatically reducing compute. This forces the model to learn the underlying structure of RF signals—harmonics, modulation shapes, and temporal coherence—without labeled data. The pre-trained encoder is then fine-tuned on downstream tasks like automatic modulation classification or specific emitter identification with limited labeled samples.

Traditional absolute position embeddings struggle with variable-length spectrograms. Rotary Position Embedding (RoPE) encodes relative position information directly into the attention computation by rotating query and key vectors based on their patch indices. This provides:

• Translation invariance: A signal pattern at any time offset receives similar attention
• Extrapolation: The model generalizes to spectrogram lengths unseen during training
• Decay with distance: Attention naturally decreases for more distant patches RoPE is particularly effective for RF signals where the absolute temporal position is arbitrary but relative timing between pulses carries critical information.

The process of dividing a spectrogram image into a grid of fixed-size patches (e.g., 16×16 pixels) and projecting each patch into a high-dimensional vector space via a linear transformation. This converts the 2D time-frequency representation into a 1D sequence of tokens suitable for transformer processing. A learnable positional encoding is added to each token to preserve the spatial and temporal relationships between patches.

A primary downstream application of the Spectrogram ViT. The model autonomously identifies the modulation scheme of a received waveform—such as BPSK, QPSK, 16-QAM, or 64-QAM—by analyzing the visual patterns in the spectrogram. The ViT's ability to capture global time-frequency dependencies makes it robust to variations in symbol rate, carrier frequency offset, and pulse shaping that challenge traditional feature-based classifiers.

An alternative approach that processes raw IQ data directly in the complex domain, preserving phase information that is inherently lost during spectrogram conversion. While Spectrogram ViTs operate on real-valued image data, CVNNs use complex-valued weights and activation functions to learn richer representations. The choice between these approaches involves a trade-off between the mature image-model ecosystem and the preservation of complete signal information.