Masked Autoencoder (MAE)

The MAE architecture is fundamentally asymmetric, meaning the encoder and decoder have vastly different capacities. The encoder is a large, heavy Vision Transformer (ViT) that operates exclusively on the small subset of visible, unmasked patches—typically only 15-25% of the input DNA sequence. This design drastically reduces pre-training compute and memory. The decoder is a lightweight, shallow transformer responsible for reconstructing the original nucleotides from the combined latent representations and mask tokens. Because the decoder is discarded after pre-training, its efficiency is critical; the heavy lifting is done by the encoder, which learns a rich, context-aware latent space without ever processing the full sequence.

Unlike BERT's masked language modeling, which typically masks 15% of tokens, MAE employs an aggressive masking ratio of 75-85%. This is possible because genomic sequences exhibit high redundancy and local continuity; a single nucleotide is highly predictable from its immediate neighbors. By removing the vast majority of patches, the model is forced to learn holistic, long-range dependencies rather than memorizing local k-mer statistics. The masking is performed randomly and uniformly, creating a challenging reconstruction task that prevents the encoder from simply interpolating from adjacent visible tokens. This high-ratio strategy is the key mechanism that drives the learning of meaningful latent representations.

MAE does not operate on individual nucleotides. The input DNA sequence is first divided into non-overlapping patches—fixed-length contiguous segments, such as 6-base or 9-base windows. Each patch is linearly projected into a dense embedding vector, which serves as the input token. This patchification step is critical: it creates a semantically meaningful unit that captures local sequence motifs (e.g., a hexamer binding site) and reduces the total sequence length by a factor of the patch size. The model then randomly shuffles these patches, masks the majority, and feeds only the visible patch embeddings to the encoder, preserving the spatial order of the surviving tokens.

The decoder's sole purpose is to reconstruct the original nucleotide sequence from the latent representation. It receives the full set of tokens: the encoded visible patches placed at their correct positions, and shared, learnable mask tokens filling the positions of the masked patches. Crucially, positional encodings are added to all tokens so the decoder knows the spatial location of each missing patch. The decoder is intentionally shallow—often only 2-4 transformer blocks—and operates on a narrow embedding dimension. Its output is a linear projection to the patch size multiplied by the nucleotide vocabulary (A, C, G, T), with a mean squared error (MSE) loss computed only on the masked patches.

A critical adaptation for genomics is enforcing reverse complement invariance during reconstruction. DNA is double-stranded, and a regulatory element on the forward strand is functionally identical to its reverse complement on the opposite strand. The MAE decoder must predict the correct nucleotide for the masked patch regardless of strand orientation. This is achieved by applying a strand-invariant loss function or by augmenting the training data with reverse complement pairs. The model learns that the latent representation of a promoter, for example, should decode to the same functional element whether presented in its forward or reverse complement orientation, ensuring biologically coherent embeddings.

After self-supervised pre-training on a large, unlabeled genomic corpus, the decoder is discarded entirely. The pre-trained encoder—now a powerful, general-purpose genomic feature extractor—is repurposed for downstream tasks. A task-specific head (e.g., a linear classifier for variant effect prediction or a multi-task head for chromatin profile prediction) is attached to the encoder's output. The entire model can then be fine-tuned end-to-end on labeled data, or the encoder can be frozen to serve as a static embedding generator. This transfer learning paradigm allows the MAE to excel on tasks with limited labeled data, as the encoder has already internalized the statistical grammar of the genome.