Transformer-Based Signal Classification

The foundational operation that allows the model to weigh the importance of every time step in an input sequence relative to every other time step. For RF classification, this means the model can directly correlate a preamble burst at the start of a transmission with a distortion pattern occurring milliseconds later, bypassing the vanishing gradient problem inherent in RNNs. Scaled dot-product attention computes compatibility scores between queries, keys, and values derived from the input IQ samples or spectrogram frames. This global receptive field is critical for identifying protocol-aware jamming where the interference pattern is synchronized with specific parts of the target waveform.

An extension of self-attention that projects the input into multiple parallel attention heads, each operating in a distinct representational subspace. In signal classification, one head might learn to attend to frequency-domain patterns while another focuses on temporal envelope characteristics. This parallelization allows the model to jointly capture diverse interference signatures—such as a chirp's instantaneous frequency slope and its periodic gating interval—within a single layer. The outputs of all heads are concatenated and linearly projected, providing a richer feature representation than any single attention map could achieve.

Since the self-attention operation is permutation-invariant, transformers require an explicit mechanism to encode the order of the input sequence. For RF data, sinusoidal positional encodings or learned position embeddings are added to the input token representations before the first attention layer. This allows the model to distinguish between a jamming pulse that occurs at the beginning of a time slot versus one that occurs at the end. In spectrogram-based approaches, 2D positional encodings can separately encode time and frequency axis positions, preserving the spatiotemporal structure of the interference pattern.

A preprocessing step adapted from Vision Transformers (ViT) where the input spectrogram is divided into a grid of non-overlapping 2D patches. Each patch is flattened and linearly projected into a fixed-dimensional embedding vector, forming the input token sequence for the transformer encoder. This approach reduces sequence length compared to pixel-level processing while preserving local time-frequency texture. A learnable class token is prepended to the sequence, and its final hidden state serves as the aggregate representation for the downstream classification head, enabling the model to output a single interference category prediction.

In cooperative spectrum sensing deployments, a transformer decoder can use cross-attention to fuse information from heterogeneous sensor nodes. The decoder treats the output of one sensor's encoder as keys and values while using another sensor's representations as queries. This allows the model to learn correlations between, for example, a high-gain directional antenna's observations and an omnidirectional sensor's background noise floor. The resulting fused representation enables more robust classification of spatially distributed interference sources, such as a mobile jammer moving through the monitored environment.

A multi-stage architecture that processes the input at progressively coarser resolutions, analogous to the downsampling stages in a CNN. Each stage applies a block of self-attention layers followed by a patch merging operation that concatenates features from neighboring patches and reduces the spatial dimensions. This design is particularly effective for wideband spectrum monitoring, where interference may span vastly different bandwidths. Early stages capture fine-grained micro-Doppler signatures, while deeper stages model macro-level spectral occupancy patterns, providing a multi-scale feature hierarchy for classification.