Attribution Sanity Checks

The foundational sanity check that cascades randomization from the output layer backward. If an attribution method is faithful, its output must degrade when model parameters are destroyed.

• Cascading Randomization: Weights are re-initialized layer-by-layer, starting from the top. Attribution maps should diverge from the original at the first randomization step.
• Similarity Metrics: Use rank correlation (Spearman) or structural similarity indices to quantify the divergence between original and randomized attributions.
• Failure Mode: If an attribution map remains unchanged after full randomization, the method is acting as an edge detector or sequence complexity analyzer, not a model explainer.

A complementary check where the model is trained on a version of the dataset where input sequences have been randomly permuted relative to their labels. A valid attribution method must produce fundamentally different explanations for a model trained on nonsense data.

• Label Permutation: Shuffle the mapping between genomic sequences and their target labels before training.
• Expected Outcome: Attribution maps from the data-randomized model should show no biologically coherent motifs or known transcription factor binding sites.
• Comparison Baseline: Quantify the difference using TF-MoDISco motif recovery rates between the real and randomized models.

Verification that the attribution method correctly handles constant or near-constant input sequences where no single nucleotide should be deemed important.

• Zero-Entropy Inputs: A sequence of all adenine (poly-A) should yield uniformly near-zero attribution scores.
• Synthetic Constructs: Test with sequences containing a single strong motif embedded in random background. The method must isolate only the motif.
• Gradient Saturation: Ensure the method does not produce artificially high attributions in regions where the model's output is saturated and gradients vanish.

Quantitative evaluation of how accurately an attribution map reflects the model's true decision logic, measured through systematic perturbation experiments.

• AOPC (Area Over the Perturbation Curve): Sequentially mask the most salient nucleotides and measure the drop in prediction probability. A steep drop indicates high faithfulness.
• ROAR (RemOve And Retrain): Retrain the model after removing the top-k attributed features. If the method is faithful, retraining on the remaining features should recover performance.
• Infidelity Measure: Compute the expected mean-squared error between a perturbation to the input and the corresponding perturbation to the attribution map.

A domain-specific sanity check that validates attributions against established biological knowledge bases, ensuring explanations are not just faithful to the model but also to molecular biology.

• Motif Database Overlap: Compare high-attribution subsequences against known motifs in JASPAR or ENCODE databases using Tomtom.
• Evolutionary Conservation: Overlay attribution maps with PhyloP or GERP conservation scores. Functional regulatory elements should show both high attribution and high conservation.
• Saturation Mutagenesis Concordance: Validate that nucleotide-level attribution scores correlate with experimentally measured Deep Mutational Scan (DMS) effect sizes.

A meta-check ensuring that the attribution method's output is stable across different implementations, software libraries, and hardware backends.

• Cross-Library Validation: Compare outputs from Captum, DeepExplain, and SHAP implementations. Results should be numerically identical within floating-point tolerance.
• Deterministic Reproducibility: Verify that running the same method on the same model and input with the same random seed produces bit-for-bit identical attributions.
• Numerical Precision: Test under FP32, FP16, and BF16. Significant divergence under reduced precision indicates an unstable implementation unsuitable for production.

The foundational sanity check for any attribution method. This test compares the attribution maps generated by a trained model against those from a model with randomly initialized weights. If the attribution method is truly sensitive to the model's learned parameters, the maps should differ significantly. A failure to detect this difference indicates the method is merely reacting to the input structure, not the model's logic.

Quantitative measures that evaluate how accurately an attribution map reflects the true decision-making logic of a genomic model. The core principle is perturbation-based validation: if the attributed nucleotides are truly important, removing or altering them must cause a predictable drop in model confidence. Key implementations include:

• AOPC (Area Over the Perturbation Curve): Measures the drop in prediction probability as the most salient nucleotides are sequentially perturbed.
• ROAR (RemOve And Retrain): Iteratively retrains a model after removing the most relevant features to test if the attribution map identifies genuinely learned dependencies.

A formal metric that quantifies the expected error between a significant perturbation of the input sequence and the corresponding perturbation of the attribution map. Mathematically, it measures the difference between the dot product of the attribution map and the perturbation vector and the actual change in the model's output. A low infidelity score indicates that the attribution map is a locally faithful linear surrogate to the model's complex decision function.

A high-throughput experimental method that assays the functional impact of thousands of genomic variants, serving as the ground-truth benchmark for validating attribution methods. By comparing computationally predicted variant effects from attribution maps against the empirical DMS measurements, developers can rigorously assess if their interpretability tools are identifying biologically causal nucleotides. This closes the loop between in-silico explanation and real-world molecular phenotype.

The systematic study of how uncertainty in the output of a genomic model can be apportioned to different sources of uncertainty in its nucleotide inputs. Unlike point-estimate attributions, sensitivity analysis provides a distribution of importance scores, often using techniques like Monte Carlo dropout or Bayesian neural networks. This allows practitioners to distinguish between high-confidence attributions and spurious correlations driven by noise.

An extension of the basic parameter randomization test where layers of the genomic model are randomized sequentially from the top down. If an attribution method is faithful, the similarity between the original and randomized attribution maps should progressively degrade as randomization cascades closer to the input. This test verifies that the method is sensitive to the hierarchical feature representations learned by deep architectures like Basset or Enformer.