Channel State Information Transformer

Unlike standard transformers that operate on real numbers, CSI transformers use complex-valued self-attention to preserve both magnitude and phase relationships. The attention score between two channel elements is computed as:

• Magnitude attention: Captures the strength of correlation between spatial paths
• Phase attention: Preserves the relative phase rotation critical for coherent combining

This dual-path mechanism allows the model to learn that two channel coefficients with identical magnitude but opposite phase carry fundamentally different information for beamforming.

CSI matrices have a 2D structure across antenna ports (spatial) and subcarriers (frequency). Standard 1D positional encodings fail to capture this grid topology. CSI transformers employ:

• 2D Rotary Position Embedding (RoPE): Encodes relative antenna spacing and subcarrier offset through rotation in the complex plane
• Learnable spatial-frequency embeddings: Trainable vectors assigned to each (antenna, subcarrier) coordinate

This dual encoding enables the attention mechanism to understand that adjacent subcarriers on the same antenna are more correlated than distant subcarriers on different antennas.

Raw CSI feedback in massive MIMO can involve thousands of subcarriers across 64+ antennas, creating prohibitively long sequences. CSI transformers address this through hierarchical tokenization:

• Coarse tokens: Represent channel clusters across blocks of subcarriers, capturing macroscopic frequency selectivity
• Fine tokens: Encode detailed variations within each block for high-fidelity reconstruction

A cross-attention mechanism fuses these two resolutions, allowing the decoder to reconstruct the full CSI matrix from a compressed latent representation with minimal information loss.

The primary use case for CSI transformers is channel feedback compression in FDD massive MIMO. The architecture introduces a deliberate bottleneck:

• The encoder at the user equipment compresses the CSI matrix into a low-dimensional latent vector using cross-attention pooling
• The decoder at the base station reconstructs the full channel using a series of self-attention and upsampling layers
• Compression ratios of 8x to 64x are achievable while maintaining higher beamforming gain than traditional codebook-based methods like Type-II CSI

Dual-polarized antenna arrays create two correlated but distinct channel views. CSI transformers model this through cross-polarization attention:

• Separate token streams for co-pol and cross-pol channel components
• A dedicated cross-attention module learns the polarization leakage patterns
• This enables the model to exploit polarization diversity for improved rank prediction and multi-layer transmission

The result is more accurate rank indicator (RI) and channel quality indicator (CQI) prediction compared to treating polarizations independently.

CSI evolves smoothly over time due to limited Doppler spread. Rather than encoding each CSI snapshot independently, CSI transformers use temporal differential encoding:

• The model processes the difference between the current CSI and a predicted value from the previous time step
• A lightweight temporal fusion layer combines the differential token with a latent state vector carrying historical channel information
• This reduces the entropy of the input, enabling higher compression ratios for periodic CSI reporting with minimal reconstruction error

An extension of the standard attention mechanism that operates natively on complex numbers, preserving the magnitude and phase relationships inherent in CSI matrices. Unlike real-valued attention that processes in-phase (I) and quadrature (Q) components as separate real channels, this mechanism computes dot-product similarity in the complex domain, enabling the model to learn more expressive representations of wave interference and spatial signatures. This is critical for accurately modeling the physics of multipath propagation.

A transformer model that performs channel estimation by processing received pilot signals. Using self-attention, it interpolates the channel response across time and frequency with higher accuracy than conventional methods like Minimum Mean Square Error (MMSE). Key advantages include:

• Superior performance in high-mobility scenarios with severe Doppler spread.
• Implicit learning of channel statistics without explicit model assumptions.
• Robustness to pilot contamination in multi-cell massive MIMO deployments.

An attention mechanism that simultaneously models dependencies across both spatial dimensions (antenna elements at the base station) and temporal dimensions (OFDM symbol periods) in a multi-antenna signal. By flattening the spatio-temporal grid into a sequence of tokens, the model learns correlations between specific antenna ports and delay taps. This unified processing is essential for capturing the full structure of a massive MIMO channel tensor for tasks like precoding and user scheduling.

A discrete, learnable token representing an individual multipath component, characterized by its delay, Doppler shift, and complex gain. Instead of processing a raw CSI matrix as a grid of pixels, the channel is decomposed into a set of these physically meaningful tokens. The transformer then applies self-attention over this set to model interactions between propagation paths, leading to highly efficient and interpretable channel representations, particularly useful in delay-Doppler domain processing for OTFS modulation.

A critical application where a transformer autoencoder compresses the downlink CSI matrix at the user equipment (UE) into a low-dimensional latent vector before transmission over a feedback link. The base station uses a decoder transformer to reconstruct the channel. This learned approach significantly outperforms traditional codebook-based methods, achieving higher reconstruction accuracy at much lower overhead. Key metrics include normalized mean squared error (NMSE) and cosine similarity of the reconstructed precoding vectors.

The application of Rotary Position Embedding (RoPE) to RF signal tokens, encoding relative temporal or frequency offsets through rotation in the complex plane. This method is particularly well-suited for complex-valued CSI representations because the rotation operation naturally preserves the magnitude while shifting the phase. It allows the attention mechanism to be aware of the relative distance between subcarriers or time slots, capturing the Toeplitz structure of the channel covariance matrix with strong inductive bias.