Attention Heatmap

An attention heatmap is a color-coded representation of the self-attention weight matrix, where each cell (i, j) indicates how strongly token i attends to token j. Darker or warmer colors typically signify higher attention scores. For a sequence of length L, the heatmap is an L×L grid, revealing pairwise interaction strengths across the entire input. In genomic models, this translates directly to nucleotide-nucleotide or amino acid-amino acid interaction maps.

Each attention head in a multi-head architecture learns distinct relational patterns, producing unique heatmaps. Common patterns include:

• Diagonal attention: tokens attend to neighbors, capturing local motifs like splice sites
• Vertical stripes: specific positions (e.g., a start codon) are attended to by all tokens
• Block diagonal: attention within functional domains or exons
• Long-range off-diagonal: distal enhancer-promoter interactions Analyzing head diversity reveals the model's learned biological grammar.

In protein language models, attention heatmaps have been shown to recapitulate structural contact maps without any explicit 3D supervision. Heads in deeper layers of models like ESM-2 produce heatmaps that correlate strongly with true residue-residue proximity in folded proteins. This emergent property validates that the model has internalized biophysical constraints governing protein folding, making heatmaps a direct bridge between sequence and structure.

Attention heatmaps enable de novo motif discovery by identifying sequence positions that receive disproportionately high attention across many input examples. By extracting and aligning the high-attention regions, researchers can uncover:

• Transcription factor binding sites
• Splice donor/acceptor sites
• Conserved regulatory elements This approach requires no prior annotation, allowing the model to surface novel functional elements from raw sequence data.

Attention patterns evolve across transformer layers, forming an information processing hierarchy:

• Shallow layers: capture local syntax, k-mer patterns, and immediate neighbors
• Middle layers: integrate regional context, such as entire exons or protein domains
• Deep layers: encode long-range, functionally meaningful interactions like enhancer-gene links or inter-domain contacts Tracking this flow reveals how the model progressively builds its understanding of biological sequence context.

Attention heatmaps are powerful but require careful interpretation:

• Attention is not explanation: high attention does not guarantee causal importance; a token may be attended to for syntactic rather than functional reasons
• Saturation in deep layers: attention can become diffuse and less interpretable in final layers
• Head redundancy: multiple heads may encode similar patterns, complicating analysis
• Aggregation choices: averaging across heads or layers can obscure meaningful signals Complementary methods like integrated gradients or in-silico mutagenesis should validate heatmap-derived hypotheses.

The fundamental mechanism that generates the attention weights visualized in a heatmap. Self-attention computes a weighted representation of every position in a sequence by dynamically assessing the relevance of all other positions. In genomics, this allows the model to link a distal enhancer to a promoter thousands of base pairs away. The raw attention scores, before softmax normalization, form the basis of the heatmap visualization.

An extension of self-attention that runs multiple attention operations in parallel. Each attention head learns a distinct relational pattern, producing its own heatmap. One head might focus on codon usage bias, while another captures transcription factor binding motifs. Analyzing heatmaps across different heads and layers is critical for understanding the hierarchical features a genomic language model has learned.

The process of using attention heatmaps to identify recurring, biologically meaningful sequence patterns directly from raw DNA. By examining which k-mers receive consistently high attention weights across many input sequences, researchers can computationally discover novel transcription factor binding sites or splice junctions without prior biological annotation. This transforms the model from a black-box predictor into a discovery tool.

In protein language models (pLMs), attention heatmaps can be interpreted as contact maps—predictions of which amino acid residues are in close spatial proximity within the folded 3D structure. The pattern of high attention between distant positions in the linear sequence often corresponds directly to physical contacts in the tertiary structure. This emergent property was foundational to the breakthrough in de novo structure prediction.

The regulatory and clinical imperative to understand why an AI made a specific prediction. For a variant effect predictor, an attention heatmap can serve as evidence by highlighting the exact nucleotides the model focused on when classifying a mutation as pathogenic. This visual justification is essential for building clinician trust and meeting FDA submission requirements for AI-based diagnostic tools.

A computational technique that systematically introduces virtual mutations into a sequence and measures the resulting change in model prediction. When paired with attention heatmaps, this method reveals not just where the model is looking, but why a specific nucleotide is critical. A high-attention position where a virtual mutation causes a massive drop in predicted binding affinity is a strong candidate for a functionally essential base.