Causal Temporal Attention

The core mechanism is a lower-triangular mask applied to the self-attention matrix. This mask sets all attention weights corresponding to future time steps to negative infinity before the softmax operation, forcing the model to compute each output solely from past and present tokens. This is mathematically equivalent to the decoder mask in autoregressive language models but applied to the time dimension of signal sequences.

Unlike bidirectional attention, which requires the entire sequence to be available before processing, causal attention enables sample-by-sample or chunk-by-chunk streaming. As each new IQ sample arrives, the model can immediately generate an output—such as an equalized symbol or a detection decision—without recomputing the entire context. This is critical for physical-layer processing where decisions must be made within strict latency budgets.

Causal temporal attention is often combined with causal convolutions in hybrid architectures. A temporal convolutional network (TCN) with dilated causal convolutions can efficiently capture local temporal patterns, while the self-attention layers model long-range dependencies. The key constraint is that all operations—convolutions, pooling, and attention—must respect the causal ordering, ensuring no information leakage from future time steps.

A critical advantage of causal attention is train-test consistency. During training, teacher forcing can be applied efficiently because the mask naturally prevents the model from attending to future targets. At inference time, the identical causal mask is used, ensuring the model never encounters a distribution shift between training and deployment. This contrasts with bidirectional models that require complex inference-time adaptations for streaming.

Causal temporal attention is frequently paired with Rotary Position Embedding (RoPE) to encode relative temporal offsets. RoPE applies a rotation to the query and key vectors based on their time indices, and the dot-product attention naturally captures relative timing. When combined with a causal mask, RoPE ensures that the model's understanding of temporal distance respects the arrow of time, making it particularly effective for processing complex-valued IQ sequences.

During autoregressive streaming inference, causal attention enables key-value (KV) caching. The keys and values computed for past time steps are stored in a memory buffer and reused for all subsequent predictions. This avoids redundant recomputation and reduces the per-step complexity from O(N²) to O(N), where N is the sequence length. For continuous signal processing, this cache grows linearly with time and may require sliding window truncation to bound memory usage.