Transformer Network

The core computational unit that allows every position in an input sequence to attend to all other positions simultaneously. For RF signal processing, this means the model can directly relate a transient event at the start of a burst to a distortion pattern occurring milliseconds later, without the sequential bottleneck of RNNs. The mechanism computes Query (Q), Key (K), and Value (V) matrices from the input, producing a weighted sum where weights are determined by the compatibility between Q and K pairs. This enables the network to dynamically focus on the most discriminative signal features regardless of their temporal distance.

An extension of self-attention that runs multiple attention operations in parallel, each with its own learned linear projections. Each head can specialize in different aspects of the signal:

• Head 1: Attends to phase discontinuities
• Head 2: Focuses on envelope rise-time characteristics
• Head 3: Captures frequency offset patterns
• Head 4: Identifies I/Q imbalance signatures

The outputs are concatenated and projected, allowing the model to jointly attend to information from different representation subspaces. This is critical for emitter fingerprinting where hardware impairments manifest across multiple signal dimensions simultaneously.

Since the Transformer processes all positions in parallel rather than sequentially, it has no inherent notion of token order. Positional encodings inject information about the relative or absolute position of each element in the sequence. For RF applications, this is adapted to encode temporal position within a signal burst. Common approaches include:

• Sinusoidal encodings: Fixed functions of position and dimension
• Learned embeddings: Trainable position vectors
• Rotary Position Embedding (RoPE): Encodes relative position through rotation matrices, particularly effective for capturing phase relationships in IQ samples

Each attention sub-layer is followed by a position-wise feed-forward network applied identically to every position. This consists of two linear transformations with a non-linear activation (typically GELU or ReLU) in between. While attention mixes information across positions, the feed-forward layers process each position's representation independently, acting as a non-linear feature transform. In signal identification, these layers learn to detect specific combinations of features that indicate particular hardware impairments, such as the joint presence of a specific carrier frequency offset and a particular amplitude compression pattern.

Each sub-layer (attention and feed-forward) is wrapped with a residual connection followed by layer normalization. The residual path allows gradients to flow directly through the network during training, enabling the successful optimization of very deep architectures. The original "post-LN" design applies normalization after the residual addition, while the modern "pre-LN" variant applies it before each sub-layer for improved training stability. For RF fingerprinting models processing long signal sequences, pre-LN Transformers demonstrate faster convergence and reduced sensitivity to learning rate selection.

The full Transformer consists of an encoder stack and a decoder stack. For signal identification tasks, the encoder-only variant (similar to BERT) is most common. The encoder processes the entire input signal sequence bidirectionally, allowing each position to attend to both past and future context. This is ideal for emitter classification where the entire signal burst is available. Key architectural choices include:

• Number of layers: Typically 6-12 for RF tasks
• Hidden dimension: 256-768, scaled to signal complexity
• Attention heads: 8-16, enabling diverse feature specialization

The fundamental computational unit that allows a Transformer to weigh the relevance of every element in an input sequence relative to every other element. Unlike recurrent models that process tokens sequentially, self-attention computes pairwise interactions in parallel, capturing long-range dependencies in IQ sample streams or spectrogram time-slices. The mechanism projects inputs into Query, Key, and Value matrices, computing attention scores via scaled dot-product operations. For RF fingerprinting, this enables the model to correlate a transient event at the start of a burst with a subtle distortion pattern occurring milliseconds later.

An extension of self-attention that runs multiple attention operations in parallel, each with its own learned linear projections. Each head can specialize in different aspects of the signal: one head might attend to phase discontinuities, another to envelope shaping, and a third to frequency offset patterns. The outputs are concatenated and projected back to the model dimension. This parallel decomposition allows the network to simultaneously extract complementary RF fingerprint features from the same input representation without interference.

Since self-attention is permutation-invariant and has no inherent notion of sequence order, positional encodings inject information about token position into the input embeddings. The original Transformer uses sinusoidal functions of varying frequencies, while modern variants often employ learned position embeddings. For signal processing applications, this encoding can be adapted to represent temporal offsets in IQ samples or frequency bin positions in spectral representations, preserving the sequential structure critical to waveform analysis.

Each Transformer layer contains a position-wise feed-forward network applied identically to every sequence element after attention. This two-layer MLP with a non-linear activation—typically GELU or ReLU—provides the model with additional representational capacity. The FFN can be interpreted as a per-token feature transformation that processes the attention-weighted context. In emitter identification, this stage refines the aggregated signal features into more discriminative representations for downstream classification.

Two architectural stabilizers that enable training deep Transformer stacks. Residual connections add the input of each sub-layer to its output, creating direct gradient highways that mitigate vanishing gradients. Layer normalization standardizes activations across the feature dimension, stabilizing training dynamics. The original post-LN design has largely been superseded by pre-LN variants that apply normalization before each sub-layer, improving convergence for the deep architectures used in complex signal classification tasks.

In encoder-decoder Transformer architectures, cross-attention allows the decoder to attend to the encoder's output representations. Queries come from the decoder's previous layer, while Keys and Values derive from the encoder's final output. For signal identification, this mechanism is valuable in multi-modal fusion scenarios—for example, allowing a decoder to cross-reference learned RF features with metadata embeddings or protocol state information to produce a final emitter classification.