Waveform Reconstruction Transformer

A self-supervised pre-training strategy where random segments of a time-domain waveform are masked, and the transformer learns to reconstruct the missing samples. This forces the model to learn the underlying statistical structure of the signal. During inference, the same mechanism is used for denoising and inpainting by treating noise or gaps as the masked region. This approach is directly analogous to Masked Language Modeling (MLM) in NLP but applied to continuous-valued IQ samples.

Standard self-attention operates on real numbers, but RF waveforms are inherently complex-valued (IQ data). This mechanism extends attention to operate natively in the complex domain, preserving critical phase and magnitude relationships. The attention scores are computed using complex-valued queries, keys, and values, allowing the model to learn transformations that are sensitive to both amplitude and phase rotation—essential for coherent waveform reconstruction.

Raw waveforms contain structure at multiple timescales, from nanosecond-level carrier cycles to millisecond-level symbol patterns. A hierarchical architecture processes the signal at multiple resolutions simultaneously:

• Fine-grained tokens capture instantaneous sample correlations
• Coarse-grained tokens model envelope and modulation patterns Cross-attention between scales allows the model to use global context to inform local reconstruction decisions.

After initial reconstruction training, a discriminator network is introduced to distinguish between real clean waveforms and the transformer's reconstructions. The generator (reconstruction transformer) is then fine-tuned to fool the discriminator, pushing it to produce perceptually indistinguishable waveforms. This is critical for removing artifacts like spectral splatter or unnatural phase discontinuities that simple MSE loss functions fail to penalize.

For real-time applications like live audio denoising or interference cancellation, the transformer uses a causal attention mask. This restricts each output sample to depend only on past and present input samples, never future ones. Combined with a sliding window buffer, this enables low-latency, sample-by-sample reconstruction without waiting for the entire waveform to be received.

The autoencoder variant compresses the input waveform into a low-dimensional latent representation using a transformer encoder, then reconstructs it with a decoder. The bottleneck forces the model to learn a highly compressed, semantic representation of the signal. This is used for source separation—by forcing multiple mixed signals through separate bottlenecks, the model learns to disentangle and reconstruct each source independently.

A self-supervised pre-training technique directly analogous to the masked autoencoding objective used in Waveform Reconstruction Transformers. During training, large contiguous blocks of a spectrogram or frequency-domain sequence are intentionally masked. The model learns to reconstruct the missing content, forcing it to build a robust, internal representation of signal structure, propagation physics, and emitter characteristics without requiring labeled data.

An extension of standard self-attention that operates natively on complex numbers, preserving both magnitude and phase. This is a critical sub-component for high-fidelity waveform reconstruction because it allows the model to learn transformations that are equivariant to phase rotation. Key benefits include:

• Direct manipulation of IQ baseband signals without splitting into real/imaginary channels
• Superior modeling of interference patterns through constructive/destructive phase relationships
• More expressive gradient flow during backpropagation for signal recovery tasks

A preprocessing module that converts a raw time-series waveform into a sequence of discrete tokens representing localized time-frequency patches. This tokenizer is the critical bridge enabling a standard transformer backbone to process spectral content efficiently. It typically employs a Short-Time Fourier Transform (STFT) followed by learned linear projections or convolutional patch embedding, creating a 2D grid of tokens that the Waveform Reconstruction Transformer can attend to globally.

An attention masking pattern that restricts the model to attend only to past and present time steps, strictly prohibiting look-ahead. For waveform reconstruction, this is essential when deploying the model for real-time streaming denoising or interference cancellation. Unlike offline processing which can use full bidirectional context, causal masking ensures the reconstructed waveform at time t depends only on samples received up to time t, enabling low-latency operation on FPGAs or edge accelerators.

A fully learned neural receiver architecture that replaces the entire traditional signal processing chain—channel estimation, equalization, and demapping—with a single end-to-end deep learning model. While the Waveform Reconstruction Transformer focuses on recovering clean waveforms from corrupted inputs, DeepRx extends this philosophy to the complete demodulation pipeline, outputting log-likelihood ratios directly from raw IQ samples. It often uses a convolutional or transformer backbone with a similar reconstruction-oriented loss function.