Time-Frequency Tokenizer

The tokenizer begins with an STFT to decompose the raw IQ or real-valued time-series into a 2D time-frequency representation. This step localizes spectral content into overlapping or non-overlapping windows, producing a complex-valued spectrogram. The magnitude and phase components are then separated or processed jointly, depending on the downstream architecture. This frontend is critical for exposing the non-stationary characteristics of communication signals, such as frequency hops or transient pulses, that a raw waveform tokenizer would miss.

Inspired by the Vision Transformer (ViT), the 2D spectrogram is divided into a grid of non-overlapping time-frequency patches. Each patch—a small matrix of spectral bins spanning a short time window—is flattened into a 1D vector. A learned linear projection then maps this vector to a fixed-dimensional embedding space, creating a patch token. This process aggressively compresses the raw spectrogram while preserving local correlations between adjacent frequency bins and time steps.

Standard positional encodings are insufficient for spectral data. The tokenizer injects a learnable frequency-domain positional encoding (FPE) that encodes the absolute center frequency of each patch. This allows the transformer backbone to understand the ordering of subcarriers or frequency bins. For multi-channel systems, a separate antenna index embedding can be added. This encoding ensures the model distinguishes between identical spectral patterns occurring at different carrier frequencies, a crucial capability for wideband spectrum analysis.

For applications requiring phase coherence, such as beamforming or digital pre-distortion, the tokenizer preserves the complex nature of the signal. Instead of separating magnitude and phase into two real channels, the patch embedding and subsequent linear projection operate in the complex domain using complex-valued neural network layers. This preserves the IQ imbalance and phase rotation information that would be lost in a real-valued conversion, enabling the transformer to learn physically meaningful representations of the wireless channel.

To avoid boundary artifacts and capture transient events that fall on patch borders, the tokenizer employs overlapped patching in the time dimension. Adjacent time-frequency patches share a percentage of their temporal bins. This redundancy ensures that a short pulse or frequency hop is fully captured within at least one token. The overlap ratio is a tunable hyperparameter, balancing the trade-off between token sequence length and the risk of missing critical, short-duration signal features.

A single patch size cannot optimally capture both narrowband tones and wideband bursts. A hierarchical tokenizer generates tokens at multiple resolutions by using different STFT window lengths and patch sizes in parallel. Fine-scale tokens capture rapid micro-Doppler modulations, while coarse-scale tokens represent long-term spectral occupancy. These multi-scale token sequences are fused using cross-attention or concatenated before being fed to the transformer, providing a rich, scale-invariant representation of the electromagnetic environment.

The primary input representation for a Time-Frequency Tokenizer. A raw spectrogram is divided into a grid of non-overlapping or overlapping 2D patches (e.g., 16x16 time-frequency bins). Each patch is flattened into a 1D vector and projected linearly to create a sequence of spectral tokens for a standard transformer encoder.

• Input: A spectrogram of shape (F, T).
• Process: Reshape into N patches of size (P_F, P_T).
• Output: A sequence of N token embeddings.
• Key Benefit: Reduces sequence length from F*T to N, making self-attention computationally tractable.

A specialized positional encoding that injects information about a token's absolute frequency index and relative temporal offset into the transformer. Unlike standard sinusoidal encodings for text, this method encodes the physical meaning of the token's location in the time-frequency plane.

• Frequency Encoding: A learned or fixed embedding for each frequency bin index.
• Time Encoding: A standard or rotary positional encoding for the time step.
• Combination: The two encodings are typically summed or concatenated before being added to the patch token.
• Purpose: Allows the model to distinguish a 1 GHz signal from a 2 GHz signal, preserving spectral structure.

An application of Rotary Position Embedding to complex-valued or real-valued RF tokens. RoPE encodes relative position by rotating the query and key vectors in the attention mechanism, which is particularly elegant for signals with inherent phase relationships.

• Mechanism: Applies a rotation matrix to token embeddings based on their time or frequency index.
• Relative Encoding: The dot-product attention score naturally depends only on the relative distance between tokens.
• RF Advantage: Preserves the complex-plane geometry of IQ samples and naturally handles the periodic nature of frequency-domain data.
• Use Case: Ideal for tokenizers processing raw IQ sequences or complex-valued spectrogram patches.

An attention masking pattern that restricts the self-attention mechanism to only attend to past and present time steps. This is essential when the Time-Frequency Tokenizer is used in a streaming or real-time processing pipeline where future samples are unavailable.

• Mask: A lower-triangular matrix applied to the attention scores.
• Constraint: A token at time t can only attend to tokens at times ≤ t.
• Application: Real-time signal classification, adaptive filtering, and online anomaly detection.
• Contrast: Standard bidirectional attention is used for offline, recorded signal analysis.

An extension of the standard attention mechanism that operates natively on complex numbers, preserving both magnitude and phase. When the tokenizer outputs complex-valued tokens (e.g., from raw IQ patches), this attention variant prevents information loss from separating real and imaginary components.

• Complex Dot-Product: Computes attention scores using complex-valued queries and keys.
• Complex Softmax: Applies softmax to the real-valued magnitudes of the complex scores.
• Weighted Sum: Aggregates complex value vectors using the real-valued attention weights.
• Benefit: Maintains the phase coherence critical for physical-layer signal processing tasks like beamforming and equalization.

A learned vector representation derived from the autocorrelation function of a signal segment. This embedding captures periodicities and cyclostationary features—such as symbol rates and carrier offsets—that are not immediately obvious in raw time-frequency patches.

• Pre-computed Feature: The autocorrelation is calculated for each token's corresponding signal segment.
• Embedding: A small neural network projects the autocorrelation vector into a dense embedding.
• Fusion: This embedding is concatenated or added to the standard patch token.
• Discriminative Power: Provides a strong inductive bias for tasks like automatic modulation classification and specific emitter identification.