Transformer Codec

Unlike RNN-based decoders that process sequences step-by-step, the Transformer Codec uses multi-head self-attention to compute relationships between all positions in a received codeword simultaneously. This allows the model to directly capture dependencies between distant bits—critical for modern codes like LDPC and polar codes where errors propagate across the entire block. The attention mechanism computes weighted sums of all input positions, enabling the decoder to focus on relevant parts of the corrupted sequence regardless of temporal distance.

Traditional belief propagation decoders process nodes sequentially, creating a computational bottleneck. The Transformer Codec processes the entire coded sequence in parallel during each iteration, dramatically reducing latency. Key benefits include:

• Simultaneous token processing: All positions updated in one forward pass
• GPU-friendly architecture: Fully leverages modern accelerator hardware
• Constant inference time: Decoding depth fixed by transformer layers, not convergence criteria This parallelism makes transformer decoders particularly attractive for high-throughput applications like 5G NR and beyond.

Since self-attention is permutation-invariant, the Transformer Codec injects learned positional encodings that encode the structural properties of the underlying error-correcting code. These encodings can capture:

• Tanner graph topology for LDPC codes
• Polarization order for polar codes
• Codebook geometry for arbitrary block codes By learning these representations from data rather than hard-coding them, the model adapts to the specific code structure and channel characteristics jointly.

The Transformer Codec employs cross-attention mechanisms to condition the decoding process on channel state information (CSI) and received signal statistics. Rather than treating CSI as a simple input feature, cross-attention allows the decoder to dynamically query the channel estimate at each decoding step. This enables:

• Adaptive decoding: Model adjusts confidence based on channel quality
• Soft information fusion: Log-likelihood ratios integrated with learned representations
• Multi-modal conditioning: Combines pilot-based estimates with blind channel features

When applied to sequential decoding or autoregressive code generation, the Transformer Codec uses masked self-attention to enforce causal constraints. This prevents the model from attending to future positions during generation, ensuring that each decoded bit depends only on previously decoded bits and the received signal. This causal masking is essential for:

• Successive cancellation decoding of polar codes
• Autoregressive neural channel coding
• Streaming applications where bits must be output sequentially

Transformer Codecs are increasingly combined with Graph Neural Network (GNN) decoders to create hybrid architectures. While GNNs excel at message passing along the Tanner graph's explicit edges, transformers capture global dependencies that GNNs may miss. This synergy produces:

• Structured attention: GNN layers provide local structure, transformers provide global context
• Improved convergence: Fewer iterations needed compared to pure belief propagation
• Generalization: Hybrid models perform well across different code rates and block lengths This combination represents the state-of-the-art in learned channel decoding.

The core mechanism enabling transformer codecs to process long-range dependencies in received codewords. Unlike recurrent decoders that process bits sequentially, self-attention computes pairwise relationships between all positions in a received sequence simultaneously.

• Computes attention weights between every pair of received symbols
• Captures global channel memory and correlated noise patterns
• Enables parallel decoding, reducing latency compared to sequential RNN-based decoders
• Multi-head attention learns diverse error pattern representations across different subspaces

A hybrid architecture that unrolls the Tanner graph structure of linear block codes into a transformer network. Each transformer layer corresponds to one iteration of belief propagation, with self-attention replacing the standard check-to-variable node message passing.

• Preserves the graph topology of the code as an inductive bias
• Learns to weight and combine messages from connected nodes adaptively
• Overcomes short-cycle traps that degrade classical belief propagation
• Achieves near-maximum-likelihood performance on low-density parity-check (LDPC) codes

A critical component that injects sequence order information into the permutation-invariant self-attention mechanism. For channel decoding, positional encodings must capture both absolute bit positions and relative distances within the codeword structure.

• Sinusoidal encodings provide smooth distance-aware representations
• Learned position embeddings adapt to specific codebook structures
• Encodings help the model distinguish systematic bits from parity bits
• Enables the transformer to respect the algebraic structure of the underlying code

A technique that adapts transformer decoders for rate-compatible punctured codes where certain coded bits are intentionally not transmitted. The attention mask prevents the model from attending to missing symbol positions.

• Zeroes out attention weights for punctured bit positions
• Allows a single model to decode multiple code rates without retraining
• Critical for hybrid automatic repeat request (HARQ) systems
• Maintains computational efficiency by skipping masked positions in the attention computation

A mechanism that fuses channel state information (CSI) into the decoding process through cross-attention layers. The decoder attends to estimated channel parameters while processing the received codeword, enabling channel-aware decoding.

• Queries from received symbols attend to key-value pairs from CSI estimates
• Handles time-varying channels by attending to time-localized channel snapshots
• Outperforms simple concatenation of CSI with input symbols
• Enables pilot-assisted and blind decoding within a unified architecture

A specific architecture that adapts the vision transformer (ViT) paradigm to channel decoding. The received codeword is split into patches of symbols, which are linearly projected into embedding vectors and processed by a standard transformer encoder.

• Treats blocks of received symbols analogously to image patches
• Uses a classification token to aggregate global information for bit-wise predictions
• Achieves state-of-the-art performance on polar codes and BCH codes
• Demonstrates that general-purpose transformer architectures transfer effectively to physical layer tasks