Self-Supervised Epigenomic Learning

The dominant pretext task adapted from natural language processing. A random subset of nucleotides or epigenomic signal bins is masked, and the model must predict the original values from the surrounding context.

• Input Corruption: Typically 15-20% of the input sequence is masked or replaced with a special token.
• Bidirectional Context: Unlike autoregressive models, the architecture attends to both upstream and downstream flanking regions to resolve the masked locus.
• Learned Representations: Forces the model to internalize motif syntax, spacing constraints, and long-range regulatory logic.
• Example: The Nucleotide Transformer uses this strategy across 3,202 diverse human genomes to learn variant effect prediction.

A discriminative SSL approach that learns representations by maximizing agreement between differently augmented views of the same genomic region while repelling representations of unrelated loci.

• Positive Pairs: Generated via stochastic data augmentation—random cropping, reverse complementing, or adding Gaussian noise to epigenomic tracks.
• Negative Sampling: Critical for preventing dimensional collapse; hard negative mining selects challenging non-matching sequences from the same chromosome.
• Output: Produces a smooth epigenomic latent space where functionally similar elements (e.g., enhancers active in the same tissue) cluster together.
• Architecture: Often employs a Siamese network with a momentum encoder, as in the MoCo and SimCLR frameworks adapted for genomic sequences.

An autoregressive SSL objective where the model predicts the next nucleotide or k-mer in a sequence, treating the genome as a linear language.

• Causal Masking: The attention mechanism is restricted to only previous tokens, preserving the sequential nature of DNA.
• Tokenization: Raw nucleotides are typically grouped into k-mers (k=6 is common) to create a manageable vocabulary size and capture local motif patterns.
• Genomic Perplexity: A metric analogous to language modeling that measures how well the model captures the statistical regularities of the genome.
• Limitation: Purely autoregressive models may struggle with enhancer-promoter interactions that operate over hundreds of kilobases, as the causal constraint limits bidirectional context.

A reconstruction-based SSL method where the model learns to recover clean epigenomic signal profiles from intentionally corrupted input data.

• Corruption Strategies: Adding Gaussian noise, dropping entire signal bins, or simulating experimental artifacts like batch effects and coverage fluctuations.
• Bottleneck Architecture: Forces the model to compress the noisy input into a compact latent representation before decoding, capturing the essential biological signal.
• Multi-Assay Training: A single denoising autoencoder can be trained simultaneously on ATAC-seq, ChIP-seq, and DNase-seq tracks, learning shared representations of chromatin state.
• Application: Excels at chromatin profile imputation, predicting missing assay data for unmeasured cell types by reconstructing from a shared latent space.

A spatial pretext task where the model must predict the correct ordering of shuffled genomic segments, learning the global syntax of regulatory element positioning.

• Input Preparation: A long sequence is divided into fixed-size patches, which are randomly permuted before being fed to the model.
• Positional Reasoning: The model must infer the original order by recognizing coherent regulatory logic—promoters must precede gene bodies, enhancers must be proximal to their targets.
• Feature Learning: Develops an understanding of chromatin domain boundaries, topologically associating domains (TADs), and the non-random organization of the 3D genome.
• Transfer Benefit: Pre-training with this objective significantly improves downstream performance on 3D genome folding prediction tasks like Hi-C contact map generation.

SSL strategies that jointly embed different molecular modalities—DNA sequence, epigenomic tracks, and gene expression—into a shared representation space without explicit labels.

• Cross-Modal Prediction: The model learns to predict one modality from another, such as inferring chromatin accessibility from raw DNA sequence or RNA expression from histone modification patterns.
• Contrastive Alignment: Pairs of corresponding modalities from the same genomic locus are pulled together in the embedding space, while non-matching pairs are pushed apart.
• Zero-Shot Transfer: A model pre-trained with multi-modal alignment can perform cross-cell-type generalization, predicting regulatory activity in unseen cell types by leveraging the learned relationships between sequence and epigenomic state.
• Example: The Enformer architecture implicitly learns this alignment by predicting thousands of epigenomic and transcriptomic tracks simultaneously from a single DNA input.