Hierarchical Temporal Transformer

The core architectural principle is the explicit processing of a signal at multiple temporal resolutions simultaneously. Unlike a standard transformer that operates on a single, fixed token rate, this architecture creates a hierarchy of representations. Fine-grained, high-resolution tokens capture transient events like individual symbol transitions or pulse edges, while coarse-grained, low-resolution tokens model long-term structural patterns such as modulation schemes, protocol framing, or channel coherence time. This is typically achieved through token pooling, strided attention, or dedicated cross-scale attention layers that allow information to flow between the different temporal scales, enabling the model to understand both the 'micro' and 'macro' structure of a waveform.

To build the temporal hierarchy, the model employs a token aggregation strategy that progressively reduces sequence length. Common methods include:

• Strided Token Merging: Concatenating or averaging adjacent tokens at a fixed stride, directly reducing temporal resolution.
• Convolutional Pooling: Applying a 1D convolution with a stride > 1 between transformer layers to downsample the sequence length while learning a compressed representation.
• Learnable Query Pooling: Using a set of learned query vectors that cross-attend to a segment of fine-grained tokens, dynamically summarizing the segment into a single coarse token. This process creates a pyramid of feature maps, where lower levels have high temporal fidelity and upper levels capture abstract, long-range context.

A critical component is the mechanism for fusing information between the different temporal scales. A unidirectional flow from fine-to-coarse is insufficient for tasks requiring precise temporal localization. Effective architectures use bidirectional cross-attention where:

• Fine-to-Coarse Attention: Injects high-resolution details into the coarse representation, ensuring abstract features are grounded in precise signal events.
• Coarse-to-Fine Attention: Broadcasts global context and long-term structural priors back to the fine-grained tokens, informing the interpretation of local, ambiguous signal segments. This fusion allows the model to, for example, use the detected modulation scheme (a coarse feature) to disambiguate a specific symbol decision (a fine feature).

For real-time signal processing applications, the architecture is adapted with causal temporal attention masks. This ensures that at each time step, a token can only attend to itself and previous time steps, preventing any look-ahead bias. The hierarchical structure is particularly advantageous here: a coarse token representing the last N seconds of context can attend to all its constituent fine tokens, providing a large receptive field without violating causality. This enables the model to operate as a streaming processor on a continuous flow of IQ samples, making decisions with low latency while maintaining awareness of long-term signal history.

When applied to raw IQ baseband signals, the Hierarchical Temporal Transformer is often paired with a Complex-Valued Attention mechanism or a Time-Frequency Tokenizer. The tokenizer first converts the complex time-series into a sequence of tokens representing time-frequency patches. The hierarchical model then processes these patches, learning correlations across both time and frequency at multiple scales. This is a powerful combination for tasks like automatic modulation classification and specific emitter identification, where discriminative features exist at vastly different time-frequency resolutions, from microsecond-scale hardware impairments to millisecond-scale modulation patterns.

The hierarchical structure is naturally suited for self-supervised pre-training via Masked Spectrum Modeling. A high proportion of fine-grained tokens (e.g., 75% of time-frequency patches) are masked. The encoder processes the visible tokens through the hierarchy, and a lightweight decoder is tasked with reconstructing the masked content using the coarse, context-rich representations from the top of the hierarchy. This forces the model to learn the underlying statistical structure of the signal domain without requiring labeled data, resulting in a powerful, general-purpose feature extractor that can be fine-tuned for multiple downstream tasks with very few labeled examples.

The foundational masking mechanism that makes real-time processing possible. By restricting the self-attention computation so that a token at time step t can only attend to tokens at time steps ≤ t, the model is prevented from peeking into the future. This is critical for streaming physical layer applications like online equalization and demodulation, where decisions must be made with strictly causal information.

A preprocessing module that converts a raw 1D complex waveform into a sequence of 2D tokens suitable for a transformer backbone. It typically applies a Short-Time Fourier Transform (STFT) to generate a spectrogram, which is then divided into overlapping or non-overlapping patches. Each patch is flattened and linearly projected into an embedding vector, allowing the model to jointly learn local spectral correlations and global temporal dynamics.

An adaptation of RoPE that encodes relative temporal or frequency offsets directly into the complex-valued signal tokens. By applying a rotation matrix to the query and key vectors, the dot-product attention score naturally depends on the relative distance between tokens. This is particularly elegant for IQ samples, as the rotation operation aligns perfectly with the complex plane representation of the signal's phase.

A mechanism for integrating heterogeneous signal representations. For example, a model might process raw IQ samples with one encoder and a spectrogram with another. Cross-attention then allows the time-domain path to query the frequency-domain path, fusing fine-grained temporal events with their spectral signatures. This is highly effective for automatic modulation classification where both domains carry complementary discriminative information.

A self-supervised pre-training strategy analogous to Masked Language Modeling. Large portions of a spectrogram or frequency-domain sequence are randomly masked, and the transformer is trained to reconstruct the missing content from the surrounding context. This forces the model to learn robust, high-level representations of signal structure without requiring any labeled data, enabling powerful few-shot transfer to downstream tasks like specific emitter identification.

An extension of the attention mechanism that operates simultaneously over the spatial (antenna elements) and temporal (symbol periods) dimensions of a MIMO signal. Instead of processing space and time sequentially, the model computes attention across all antenna-time tokens, directly capturing complex spatial multiplexing and beamforming patterns. This forms the core of advanced neural receivers like DeepRx MIMO.