Patchified Spectrogram

The core mechanism involves dividing a spectrogram image into a regular grid of non-overlapping or overlapping patches. Each 2D patch, representing a localized time-frequency region, is flattened into a 1D vector and linearly projected into a fixed-dimensional token embedding. This process converts the continuous spectrogram into a discrete sequence of tokens that a standard transformer encoder can process, analogous to how the Vision Transformer treats an image. The patch size is a critical hyperparameter that determines the granularity of the learned features.

Unlike flattening a spectrogram into a 1D sequence of frequency vectors, patchification explicitly preserves the 2D spatial locality of time-frequency energy. A single patch encapsulates both temporal evolution and spectral content within its boundaries. This allows the transformer's self-attention mechanism to learn correlations between distinct signal events—such as a frequency hop or a pulsed transmission—that are separated in time and frequency but structurally related, making it superior for tasks like emitter identification and signal classification.

Since the self-attention mechanism is permutation-invariant, the flattened patch tokens must be injected with positional information to retain the spectrogram's topology. A learned 2D positional encoding is added to each token, encoding its original row (frequency bin) and column (time frame) coordinates. This allows the model to distinguish between a signal at a low frequency early in time versus a high frequency later, enabling the learning of complex cyclostationary patterns and transient behaviors.

The choice of patch stride defines the tokenization scheme. Non-overlapping patches (stride = patch size) are computationally efficient and produce fewer tokens. Overlapping patches (stride < patch size) generate a higher-resolution token sequence with redundant information, which can improve performance on fine-grained tasks like weak signal detection by providing multiple perspectives on the same time-frequency event. This creates a trade-off between computational complexity and model accuracy.

Advanced architectures employ a hierarchical patchification scheme, generating tokens at multiple scales simultaneously. A coarse patch captures broadband, long-duration events, while a fine patch captures narrowband, transient pulses. These multi-scale tokens are fed into a hierarchical transformer or a feature pyramid network, allowing the model to learn both global spectral usage patterns and local signal minutiae, which is essential for complex tasks like automatic modulation classification in congested environments.

The patchified representation is directly compatible with self-supervised learning techniques like Masked Spectrum Modeling. By randomly masking a subset of spectrogram patches and training a transformer autoencoder to reconstruct the missing time-frequency content from the visible context, the model learns robust, general-purpose representations of signal structure without requiring labeled data. This is a powerful method for leveraging vast amounts of unlabeled RF recordings.