Motif Discovery

The core mechanism relies on analyzing self-attention weights from transformer heads. By examining which nucleotides the model attends to when making predictions, researchers can extract position weight matrices (PWMs) that represent binding motifs. Attention heads in models like DNABERT and Enformer often specialize in recognizing specific transcription factor binding sites, with different heads capturing distinct motif features such as core binding sequences or flanking nucleotide preferences.

Learned nucleotide embeddings from genomic language models encode rich contextual information about sequence function. Motif discovery can be performed by:

• Clustering similar k-mer embeddings to identify conserved sequence patterns
• Projecting embeddings into lower-dimensional space using UMAP or t-SNE to visualize motif families
• Comparing embedding similarity between known motifs and novel sequences to discover de novo binding sites This approach captures subtle sequence features that position weight matrices may miss.

A systematic approach where every nucleotide in a sequence is virtually mutated, and the change in model prediction score is measured. This generates a mutation effect map that reveals which positions are critical for function. Positions with high sensitivity to mutation correspond to functional nucleotides within motifs. This technique, applied to models like Enformer and the Nucleotide Transformer, can identify both the core motif and its flanking context dependencies without any prior motif annotation.

Pre-trained genomic transformers can discover motifs without task-specific fine-tuning. By using the model's masked language modeling (MLM) objective, researchers score how well a sequence conforms to learned regulatory grammar. Sequences with high likelihood scores contain patterns the model recognizes as biologically meaningful. This zero-shot capability enables motif discovery in newly sequenced genomes where no experimental binding data exists, leveraging evolutionary conservation patterns learned during pre-training.

Attention heatmaps provide interpretable visualizations of motif locations. By plotting attention weights as a heatmap over the input sequence, researchers can directly observe:

• Which sequence regions the model focuses on for specific predictions
• The boundaries of regulatory elements like enhancers and promoters
• Cooperative binding patterns where multiple motifs interact Tools like Tangermeme and custom visualization libraries enable genome-browser-style views of attention patterns across long sequences.

Transformer models trained on multiple species can identify evolutionarily conserved motifs that persist across divergent genomes. By comparing attention patterns or embedding similarities between orthologous regions in different species, researchers can distinguish functionally constrained motifs from neutral sequence patterns. This approach has revealed deeply conserved regulatory logic in processes like developmental gene regulation and stress response pathways that traditional alignment-based methods struggle to detect.

The fundamental mechanism that makes motif discovery possible in transformers. Self-attention computes a weighted representation of every position in a sequence by dynamically assessing its relevance to all other positions.

• Enables capture of long-range dependencies between distal regulatory elements
• Attention weights directly highlight nucleotides involved in binding interactions
• Raw attention maps serve as the primary signal for de novo motif extraction
• Multi-head variants allow simultaneous discovery of different motif types

A visualization of self-attention weights used as the primary interpretability tool for motif discovery. Heatmaps reveal which specific nucleotides the model focuses on when making predictions.

• Identifies potential transcription factor binding sites without prior annotation
• Column-wise aggregation reveals position weight matrices (PWMs)
• Can distinguish between direct binding motifs and higher-order sequence grammar
• Used to validate that models learn biologically relevant features rather than artifacts

A systematic computational technique that introduces virtual mutations into a sequence and measures the resulting change in model predictions. This generates a comprehensive effect map for every possible single-nucleotide change.

• Saturating mutagenesis reveals motif boundaries at single-nucleotide resolution
• Identifies which positions within a motif are most critical for function
• Enables discovery of syntax rules governing motif spacing and orientation
• Complements attention-based methods by providing causal rather than correlational evidence

A measure of the degree to which nucleotide positions remain unchanged across evolutionary time. This signal is implicitly learned by transformer models during self-supervised pre-training on diverse genomes.

• Positions with high attention weights often correspond to evolutionarily constrained elements
• Cross-species model comparisons validate discovered motifs as functionally important
• Conservation patterns help distinguish functional motifs from statistical noise
• Provides an orthogonal validation signal independent of attention-based discovery