Complex-Valued Attention

Replaces the standard real-valued dot product with a complex inner product to compute attention scores. The query, key, and value projections are complex linear transformations, and the attention weight is computed as the squared magnitude of the complex dot product. This preserves phase alignment between query and key vectors, allowing the model to attend based on both amplitude correlation and phase coherence.

Standard softmax operates on real-valued logits, discarding phase information. Complex-valued attention applies softmax to the magnitudes of the complex attention scores while retaining the phase component for value aggregation. The output is a complex-weighted sum where both the amplitude weighting (from softmax) and phase rotation (from the complex scores) are preserved, enabling the model to learn phase-sensitive alignments.

Training complex-valued networks requires optimization in the complex domain. Wirtinger calculus treats the complex variable and its conjugate as independent, enabling gradient computation for non-holomorphic activation functions. This allows standard backpropagation to work with complex-valued parameters by computing gradients with respect to both the real and imaginary parts separately, ensuring stable convergence.

Rotary Position Embedding (RoPE) is naturally suited for complex-valued attention. It encodes relative position by rotating query and key vectors in the complex plane by an angle proportional to their position index. For complex-valued signals, this provides a mathematically elegant way to inject temporal or frequency-domain positional information without breaking the complex algebraic structure of the representations.

A common architectural pattern splits complex-valued attention into parallel streams: one processing magnitude (signal energy) and another processing phase (signal structure). The two streams may use different attention mechanisms—e.g., standard dot-product for magnitude and a phase-sensitive operation for the angle—before recombining. This allows the model to learn distinct representations for amplitude-based and phase-based signal features.

Standard residual connections add real-valued tensors. Complex-valued attention uses complex residual connections that preserve the full IQ representation throughout the network depth. This prevents information loss from premature conversion to real-valued magnitudes and ensures that phase coherence is maintained across multiple attention layers, which is critical for tasks like channel equalization and beamforming.

Complex-valued attention directly processes the channel state information (CSI) matrices from antenna arrays, learning optimal beamforming weights. Unlike real-valued networks that split complex numbers, this approach preserves the phase coherence between antenna elements, leading to higher beamforming gain and reduced interference in multi-user scenarios. The attention mechanism learns to focus on the most critical spatial paths.

In a fully learned neural receiver, complex-valued attention replaces the entire traditional processing chain. The model attends to a sequence of received IQ symbols to perform joint channel estimation, equalization, and demapping in one shot. The attention mechanism's ability to model long-range dependencies in the complex domain allows it to mitigate severe inter-symbol interference that conventional methods struggle with.

Complex-valued attention enables the design of autoencoder-based transceivers that map source data directly to channel symbols in the complex domain. The encoder uses self-attention to create robust complex-valued representations, while the decoder uses cross-attention to reconstruct the original data from a noisy received signal. This joint optimization outperforms separate source and channel coding, especially in low-SNR regimes.

Complex-valued attention models are used to design and process radar waveforms for autonomous systems. The network learns to attend to specific delay-Doppler bins in the complex ambiguity function, improving target detection and range-velocity estimation. By operating natively on complex baseband samples, the model preserves the phase history critical for micro-Doppler signature analysis and synthetic aperture radar (SAR) imaging.

Complex-valued attention mechanisms excel at RF fingerprinting by learning to attend to the subtle, unique hardware impairments embedded in the phase and magnitude of a transmitter's IQ signal. The model focuses on transient and steady-state phase noise patterns that are discarded by real-valued networks, enabling highly robust specific emitter identification (SEI) for secure device authentication at the physical layer.

In goal-oriented semantic communication, complex-valued attention is the core of a system that transmits the meaning of a message, not its exact bits. The transmitter's attention mechanism learns to map semantic features to a robust complex-valued signal. The receiver uses cross-attention to reconstruct the intended meaning from a distorted signal, achieving dramatic bandwidth savings by ignoring task-irrelevant information.

A transformer architecture that processes raw in-phase and quadrature (IQ) samples directly. It uses specialized tokenization to preserve the complex baseband structure, making it the primary consumer of complex-valued attention. Key aspects:

• Operates on complex-valued tokens without separating real and imaginary parts
• Uses complex layer normalization to stabilize training
• Enables end-to-end learning from antenna samples to bits

The application of Rotary Position Embedding (RoPE) to RF signal tokens. RoPE encodes relative temporal or frequency offsets through rotation in the complex plane, making it mathematically natural for complex-valued attention. Benefits include:

• Preserves the complex inner product structure
• Naturally captures phase rotations from carrier frequency offsets
• Enables length extrapolation for variable-duration signals

The extension of gradient-based learning to complex parameters using Wirtinger calculus. Unlike real-valued backpropagation, gradients are computed with respect to the complex variable and its conjugate independently. Critical properties:

• The chain rule operates in the complex domain
• Activation functions must be holomorphic or use split-complex approaches
• Enables end-to-end training of complex attention weights

A design pattern in complex attention where the magnitude and phase of the attention weights are computed or constrained separately. This allows:

• Magnitude to represent signal energy and relevance
• Phase to encode relative timing and carrier offsets
• Independent regularization of amplitude and angular components
• More interpretable attention patterns for signal processing engineers

A stabilization technique that normalizes complex-valued activations using a complex mean and covariance matrix. Unlike real layer norm, it must account for the correlation between real and imaginary parts. The process:

• Computes a 2×2 covariance matrix per feature
• Whitens using the inverse square root of the covariance
• Applies learnable complex scale and shift parameters

A mechanism using complex-valued cross-attention to fuse information from two distinct signal representations. Common pairings include:

• Time-domain IQ samples with frequency-domain spectra
• Multiple antenna streams in MIMO systems
• Pilot symbols with data symbols for channel estimation
• The complex attention weights preserve phase coherence between modalities