Network Dissection

The framework relies on a curated set of human-interpretable concepts that serve as ground-truth labels for evaluating hidden units. In genomic models, these concepts include:

• Transcription factor binding motifs (e.g., CTCF, SP1, GATA family)
• Splice donor and acceptor sites
• CpG islands and methylation-prone regions
• Open chromatin regions from ATAC-seq data
• Conserved non-coding elements across species

Each concept is represented as a binary segmentation mask over the input sequence, indicating which nucleotides belong to the concept region.

For each hidden unit (neuron or channel) in a target convolutional or transformer layer, the framework computes an activation map across a dataset of genomic sequences. The key steps are:

• Forward-propagate each sequence through the model up to the target layer
• Record the spatial activation pattern of each unit
• Upsample activation maps to match the input nucleotide resolution using bilinear interpolation
• Threshold activations to produce a binary detection mask for each unit

This process transforms continuous activations into discrete region proposals that can be compared against concept masks.

The alignment between a unit's detection mask and a concept's segmentation mask is quantified using the Intersection-over-Union (IoU) metric:

codeCopy
IoU = |Detection ∩ Concept| / |Detection ∪ Concept|

A unit is considered a detector for a specific concept if its IoU exceeds a predefined threshold (typically > 0.04 for genomic applications). This threshold accounts for the inherent sparsity and variable length of biological sequence elements compared to the fixed receptive fields of convolutional filters.

Network Dissection produces a global interpretability score for the model by measuring the fraction of units that emerge as concept detectors:

• Number of unique concepts detected: How many distinct biological motifs or regions are captured by at least one unit
• Detector coverage: The percentage of all units in a layer that align with any annotated concept
• Concept selectivity: Whether a unit responds exclusively to one concept or fires across multiple unrelated sequence patterns

This quantification enables direct comparison of interpretability across different model architectures, training regimes, and layer depths.

Applying Network Dissection across all layers of a genomic model reveals the hierarchical emergence of biological concepts:

• Early layers: Detect simple nucleotide patterns such as GC content, dinucleotide frequencies, and short k-mer repeats
• Middle layers: Emerge as detectors for canonical transcription factor binding motifs and splice junction signals
• Deep layers: Align with complex, composite concepts such as enhancer-promoter interactions, chromatin state combinations, and tissue-specific regulatory modules

This progression mirrors the hierarchical organization of biological sequence information and validates that the model learns meaningful representations.

The framework requires a broadly and densely labeled dataset (Broden) that pairs genomic sequences with multiple concept annotations:

• Multi-concept per sequence: Each input sequence is annotated with all applicable biological concepts simultaneously
• Diverse genomic contexts: Sequences span promoters, introns, intergenic regions, and coding exons
• Balanced representation: Equal sampling across concept classes to prevent frequency bias in detector identification
• Strand-aware annotation: Concepts are labeled on both forward and reverse strands to capture orientation-specific unit tuning

For genomic applications, Broden is typically constructed from ENCODE, Roadmap Epigenomics, and FANTOM5 consortium data.

A systematic perturbation approach that computationally mutates every nucleotide position to all three alternative bases and records the prediction change.

• Produces a mutation effect map showing which positions are functionally intolerant
• Used to validate whether units identified by Network Dissection genuinely track pathogenic variants
• Computationally expensive but provides causal evidence rather than correlational

Architectures that first predict human-specified biological concepts from genomic data, then use only those concepts for the final prediction.

• Enforces interpretability by construction rather than post-hoc analysis
• Concepts might include 'promoter region,' 'splice donor,' or 'nucleosome occupancy'
• Contrasts with Network Dissection, which discovers concepts emergent in a black-box model

Quantitative benchmarks that evaluate whether an interpretability method accurately reflects the model's true decision logic.

• ROAR (Remove And Retrain): Iteratively retrains after removing top-attributed features to measure degradation
• AOPC (Area Over Perturbation Curve): Measures prediction drop as salient nucleotides are sequentially masked
• Essential for validating that Network Dissection's unit labels correspond to causal mechanisms, not spurious correlations