Attention Mechanism

Attention mechanisms compute a context vector as a weighted sum of input elements, where weights are dynamically generated based on the current query. Unlike fixed convolutional kernels or recurrent connections, these weights adapt to each input instance.

• Query-Key-Value (QKV) Framework: Queries represent what to look for, keys represent what each input offers, and values represent the actual content
• Softmax Normalization: Raw compatibility scores are transformed into a probability distribution, ensuring weights sum to 1
• Content-Based Addressing: The model retrieves information based on semantic similarity rather than positional proximity

Example: In few-shot modulation recognition, a query sample of an unknown signal dynamically attends to support set examples with similar IQ constellation patterns, regardless of their position in the sequence.

The core scoring function computes compatibility between queries and keys using a dot product scaled by the square root of the key dimension. This scaling prevents the softmax from entering regions of extremely small gradients when dimensionality is large.

• Formula: Attention(Q,K,V) = softmax(QK^T / √d_k)V
• Scaling Factor: Dividing by √d_k maintains variance at approximately 1, preventing saturated softmax outputs
• Computational Efficiency: Matrix multiplication enables highly parallelized implementation on GPUs and NPUs

Real-World Impact: This mechanism allows a few-shot classifier to simultaneously compare a query signal against hundreds of support samples in a single matrix operation, achieving sub-millisecond attention computation for real-time spectrum classification.

Rather than computing a single attention function, multi-head attention projects queries, keys, and values into multiple lower-dimensional subspaces, allowing the model to attend to different representation aspects simultaneously.

• Parallel Heads: Typically 8-16 independent attention operations run concurrently
• Subspace Diversity: Each head can specialize in different feature relationships—one head might focus on phase patterns while another attends to amplitude variations
• Concatenation and Projection: Head outputs are concatenated and linearly projected back to the model dimension

Signal Processing Application: In modulation classification, separate heads can simultaneously attend to cyclostationary features, constellation geometry, and temporal envelope characteristics, providing complementary evidence for the final classification decision.

Since attention mechanisms are inherently permutation-invariant—they treat input as a set without order—positional information must be explicitly injected. This is critical for sequential data like IQ sample streams.

• Sinusoidal Encodings: Fixed encodings using sine and cosine functions of varying frequencies, enabling the model to attend to relative positions
• Learned Positional Embeddings: Trainable vectors that capture task-specific temporal patterns
• Relative Position Representations: Encodings that model pairwise distances between elements rather than absolute positions

Why It Matters: Raw IQ samples are inherently temporal; without positional encoding, the attention mechanism cannot distinguish between a preamble sequence and a payload burst, degrading few-shot classification accuracy on time-varying modulation formats like GMSK.

Cross-attention extends the standard self-attention mechanism by computing compatibility between elements from two different sequences—typically a query sample and a support set. This is the foundational operation in matching networks and relation networks for few-shot learning.

• Asymmetric Inputs: Queries come from the test sample, while keys and values come from the support set
• Pairwise Comparison: Every query element attends to every support element, building a fine-grained similarity map
• Contextualized Embeddings: The output is a support-conditioned representation of the query, enriched with relevant information from labeled examples

Modulation Recognition Example: A query IQ sequence cross-attends to support samples of known modulations (BPSK, QPSK, 16QAM). The resulting attention weights highlight which support samples—and which specific time segments within them—are most relevant for classification.

Attention masks control which input elements can interact, enabling causal constraints, padding handling, and sparse attention patterns that improve computational efficiency without sacrificing representational capacity.

• Causal Masking: Prevents attending to future time steps, essential for autoregressive signal prediction
• Padding Masks: Excludes zero-padded elements from attention computation, preventing artificial correlations
• Sparse Patterns: Local window attention, strided attention, and global token attention reduce O(n²) complexity to O(n√n) or O(n log n)

Efficiency Impact: For long IQ sequences (10k+ samples), sparse attention reduces memory consumption from gigabytes to megabytes, enabling deployment on edge FPGA platforms for real-time spectrum monitoring without cloud connectivity.

The specific computational core of the Transformer architecture. It computes a weighted sum of values, where the weight assigned to each value is determined by a compatibility function between a query and a key.

• Formula: Attention(Q,K,V) = softmax(QK^T / √d_k)V
• Scaling Factor: Dividing by √d_k prevents the dot products from growing too large, which would push the softmax function into regions of extremely small gradients.
• Mechanism: The query is matched against all keys to produce attention scores, which are then used to aggregate the corresponding values.

A mechanism that runs multiple scaled dot-product attention operations in parallel. This allows the model to jointly attend to information from different representation subspaces at different positions.

• Parallel Heads: Instead of a single attention function, queries, keys, and values are linearly projected h times.
• Subspace Focus: Each head can learn to focus on different aspects of the signal, such as phase shifts, amplitude variations, or frequency components.
• Concatenation: The outputs of all heads are concatenated and projected again, providing a richer representation than a single attention head.

An attention mechanism where the queries, keys, and values are all derived from the same input sequence. It computes a representation of a sequence by relating different positions within that sequence.

• Intra-Sequence Dependencies: Captures long-range dependencies between signal samples without regard to their distance in the input sequence.
• Permutation Equivariance: The operation is inherently permutation equivariant, meaning it processes sets of features without assuming a fixed spatial or temporal order.
• Application: Used in encoder layers to build rich contextual representations of IQ sample sequences before cross-attending with support set examples.

An attention mechanism that mixes information between two distinct sequences. Queries are derived from one sequence, while keys and values come from another.

• Few-Shot Role: In matching networks, the query sample (unlabeled) attends to the support set (labeled) to compute a weighted class prediction.
• Feature Alignment: Allows the model to dynamically align features from a novel query signal with the most relevant examples in the support set.
• Asymmetric Design: Unlike self-attention, cross-attention explicitly models the relationship between a primary input and a context or memory source.

An alternative compatibility function to dot-product attention, also known as Bahdanau attention. Instead of a dot product, it computes a score using a feed-forward network with a single hidden layer.

• Formula: score(q, k) = v^T tanh(W_1 q + W_2 k)
• Flexibility: Theoretically more expressive than dot-product attention, as the non-linear tanh activation allows for more complex compatibility functions.
• Computational Cost: Generally slower and more memory-intensive than highly optimized scaled dot-product attention, leading to its replacement in most modern architectures.

The practice of interpreting the learned attention distribution as a measure of feature importance or model explanation. In signal classification, this can highlight which temporal segments of an IQ sample the model deems salient.

• Visualization: Heatmaps of attention weights over time can show if the model focuses on preamble sequences, pilot tones, or transition regions.
• Caution: Raw attention weights are not always faithful explanations of model decisions. They correlate with feature importance but do not guarantee causal attribution.
• Debugging Tool: Useful for identifying if a model is relying on spurious correlations or noise artifacts in the signal.