Self-Attention Mechanism

Unlike recurrent networks that process signals sequentially, self-attention computes a weighted representation of the entire input sequence in parallel. For a raw IQ sample stream, this means the model can directly relate a transient burst at the start of a time slot to a fading dip occurring much later, capturing long-range dependencies that define complex modulation patterns without the degradation of temporal information.

The mechanism operates by projecting each input element into three distinct vectors: a Query (Q), a Key (K), and a Value (V). The query of a target signal element is compared against the keys of all other elements via a dot product to produce attention scores. These scores are used to aggregate the value vectors, effectively synthesizing a new representation that is a selective summary of the entire signal context.

The core mathematical operation is defined as Attention(Q, K, V) = softmax(QK^T / √d_k)V. The dot product of Q and K computes a raw relevance score. Dividing by the square root of the key dimension (√d_k) is critical to prevent the softmax function from entering regions with extremely small gradients, ensuring stable training. The softmax normalizes scores into a probability distribution over the sequence.

Instead of performing a single attention function, the model runs multiple attention operations in parallel with different, learned linear projections of Q, K, and V. This allows the model to jointly attend to information from different representation subspaces. In modulation recognition, one head might focus on phase transitions while another attends to amplitude variations, creating a richer feature set for classification.

Because self-attention is inherently permutation-invariant and has no notion of sequence order, positional information must be explicitly injected. This is achieved by adding sinusoidal functions of different frequencies or learned embeddings to the input vectors. For a signal classifier, this encoding allows the model to distinguish a linearly increasing phase ramp from a random jitter pattern, preserving the critical temporal structure of the modulation.

The standard self-attention mechanism has a quadratic complexity of O(n²) with respect to the sequence length n, as it computes a pairwise interaction between every element. This is a primary engineering constraint when applying transformers to long, high-sample-rate IQ streams. Research into efficient attention variants, such as linear or sparse attention, is critical for deploying these models in real-time spectrum classification systems.

An extension of the basic self-attention mechanism that runs multiple attention operations in parallel. Each 'head' learns to attend to different representational subspaces, allowing the model to jointly focus on information from different positions and feature dimensions.

• Computes scaled dot-product attention across multiple learned linear projections of queries, keys, and values
• Enables the model to simultaneously capture short-term transient features and long-term structural patterns in a signal
• Typical configurations use 8 or 16 heads, with each head operating on a reduced dimension to maintain computational parity with single-head attention
• Outputs from all heads are concatenated and linearly projected to produce the final multi-head representation

A mechanism that injects information about the absolute or relative position of tokens in a sequence into the transformer model. Since self-attention is inherently permutation-invariant, positional encodings are essential for the model to understand temporal order in signal data.

• Sinusoidal encodings use fixed sine and cosine functions of different frequencies to represent position, allowing the model to extrapolate to sequence lengths unseen during training
• Learned positional embeddings treat position as a trainable parameter, often preferred for fixed-length signal segments
• In modulation recognition, positional information helps the model distinguish between identical signal patterns occurring at different time indices within an IQ sequence

The core mathematical operation at the heart of the self-attention mechanism. It computes a weighted sum of values, where the weight assigned to each value is determined by a compatibility function between a query and the corresponding key.

• Formula: Attention(Q, K, V) = softmax(QK^T / √d_k)V, where d_k is the dimension of the keys
• The scaling factor 1/√d_k prevents the dot products from growing too large in magnitude, which would push the softmax function into regions with extremely small gradients
• Computationally efficient when implemented as matrix multiplications on GPUs or specialized neural processing units
• Provides a differentiable mechanism for the model to learn which parts of the input signal are most relevant for the classification task

The three learned linear transformations that project each input element into distinct representational spaces before the attention computation. This concept is borrowed from information retrieval systems.

• Query (Q): Represents what the current position is 'looking for' in other positions
• Key (K): Represents what each position 'offers' as a match to queries
• Value (V): Represents the actual information that will be aggregated if a position is attended to
• In signal classification, these projections allow the model to dynamically route information—for example, a query from a symbol transition region can attend to keys representing steady-state symbol features

A critical distinction in attention mechanisms that defines the source of the query, key, and value inputs.

• Self-Attention: All three projections (Q, K, V) are derived from the same input sequence. Used in encoder layers to build rich contextual representations of a single signal segment
• Cross-Attention: Queries come from one sequence (e.g., decoder), while keys and values come from another (e.g., encoder output). Used in decoder layers to align generated outputs with input signal features
• In modulation classification tasks, pure encoder architectures using only self-attention are common, as the goal is to produce a single class label rather than a generated sequence