Sensitivity Analysis

Sensitivity analysis operates at two distinct scopes:

• Local Sensitivity: Measures the impact of infinitesimal perturbations around a single specific input sequence, often computed via the partial derivative of the output with respect to each input nucleotide.
• Global Sensitivity: Assesses the impact of input variability across the entire domain of possible sequences, identifying factors that dominate model behavior irrespective of the specific input. This distinction is critical for determining whether an explanation is valid for a single patient's variant or a general biological mechanism.

The foundational method for sensitivity analysis involves systematically perturbing one input feature while holding all others constant.

• In genomic models, this translates to performing in-silico mutagenesis (ISM) by computationally substituting a single nucleotide at each position and recording the change in the model's prediction score.
• The resulting delta score quantifies the sensitivity of the output to that specific base pair. While computationally straightforward, OAT methods fail to capture epistatic interactions where the effect of one mutation depends on the state of another.

Global sensitivity is often quantified using variance-based methods, such as Sobol' indices, which decompose the total variance of the model output into fractions attributable to individual inputs and their higher-order interactions.

• First-order index: Measures the direct contribution of a single nucleotide position to output variance.
• Total-effect index: Measures the total contribution, including all interactions with other positions. This provides a rigorous statistical framework for identifying which genomic regions most control the model's predictive uncertainty, essential for regulatory compliance.

For differentiable genomic models like deep neural networks, the gradient of the output with respect to the input provides a direct, computationally efficient measure of local sensitivity.

• Saliency maps are generated by computing the absolute value of these gradients, visually highlighting nucleotides where a small change would cause the largest impact on the prediction.
• This is the core mechanism behind attribution methods like Integrated Gradients and DeepLIFT. Gradient-based sensitivity assumes a locally linear approximation of the model's decision function, which can be a limitation for highly non-linear genomic landscapes.

Sensitivity analysis directly underpins the evaluation of explanation quality through faithfulness metrics:

• Area Over the Perturbation Curve (AOPC): Sequentially perturbs the most 'sensitive' nucleotides identified by an attribution map and measures the drop in prediction probability. A steep drop indicates a faithful map.
• Remove and Retrain (ROAR): Retrains the model from scratch after removing the most sensitive features, testing if the original attribution map truly identified the causal features the model relied upon. These metrics transform sensitivity from a descriptive tool into a quantitative benchmark for interpretability.

A raw sensitivity score is incomplete without a measure of its statistical confidence. Attribution uncertainty quantifies the variability of an importance score.

• Techniques like Bayesian neural networks or Monte Carlo dropout can be used to compute a distribution of sensitivity scores for each nucleotide, yielding credible intervals.
• A high-sensitivity nucleotide with a wide confidence interval is a less reliable explanation than one with a tight interval. This is vital for clinical genomics, where a variant effect prediction must be accompanied by a confidence score to inform medical decisions.

Quantifies the confidence interval around a variant effect score by propagating input perturbations through the model. This distinguishes truly pathogenic variants from benign ones when prediction scores are similar.

• Uses Monte Carlo dropout or ensemble methods to generate prediction distributions
• Critical for clinical reporting where a single point estimate is insufficient
• Enables risk stratification based on prediction variance, not just mean effect

Systematically adds noise to one-hot encoded nucleotide sequences to measure how model predictions degrade. This reveals which sequence positions the model is most sensitive to.

• Gaussian noise injection into embedding layers
• Identifies brittle dependencies on single nucleotides
• Used to validate that models rely on biological motifs rather than artifacts

Decomposes output variance using Sobol sensitivity analysis to attribute uncertainty to specific input features or interactions between features. This is a model-agnostic approach.

• Computes first-order and total-order sensitivity indices
• Identifies epistatic interactions where nucleotide pairs jointly influence predictions
• Computationally intensive but provides rigorous variance decomposition

Evaluates whether feature attribution maps are stable under input perturbations. A reliable saliency map should not change dramatically when non-essential nucleotides are altered.

• Measures Jaccard similarity between attribution maps from original and perturbed inputs
• Low stability indicates the model's decision boundary is poorly defined
• Used to validate interpretability methods before regulatory submission

Tests model sensitivity to adversarial perturbations—small, biologically plausible sequence changes that cause large prediction shifts. This reveals vulnerabilities in genomic classifiers.

• Generates adversarial examples constrained by evolutionary substitution matrices
• High sensitivity to adversarial examples indicates overfitting to spurious correlations
• Essential for validating models used in clinical decision support

Applies sensitivity analysis to calibrate model confidence so that predicted probabilities reflect true empirical frequencies. A well-calibrated model's 80% confidence predictions are correct 80% of the time.

• Uses expected calibration error (ECE) as a quantitative metric
• Temperature scaling adjusts output logits based on sensitivity patterns
• Required for models that inform clinical action thresholds

The general class of methods that assign a relevance score to each input nucleotide for a specific neural network prediction. While sensitivity analysis studies output variance from input perturbations, feature attribution methods like Integrated Gradients and DeepLIFT decompose a single prediction backward through the network to identify which bases drove the decision.

• Produces per-nucleotide importance maps
• Foundation for all downstream interpretability
• Distinct from but complementary to global sensitivity analysis

A systematic perturbation technique that computationally mutates every nucleotide in a sequence to quantify its impact on model predictions. ISM is the most direct implementation of local sensitivity analysis in genomics, measuring the change in output when each position is individually altered.

• Exhaustive single-nucleotide variant scanning
• Generates delta score matrices
• Computationally expensive for long sequences
• Gold standard for variant effect prediction

Quantitative measures that evaluate how accurately an attribution map reflects the true decision-making logic of a genomic model. These metrics are essential for validating sensitivity analysis results through perturbation experiments that test whether removing high-importance features actually changes predictions.

• ROAR: Retrain after removing top features
• AOPC: Area Over the Perturbation Curve
• Infidelity Measure: Expected perturbation-prediction error
• Critical for regulatory compliance documentation

A unified framework based on Shapley values from cooperative game theory that assigns each genomic feature an importance score for a particular prediction. SHAP connects sensitivity analysis to game-theoretic optimal credit allocation, satisfying properties like local accuracy and consistency.

• KernelSHAP: Model-agnostic approximation
• DeepSHAP: High-speed variant for deep networks
• Computes additive feature importance
• Bridges global and local sensitivity perspectives

The task of using deep learning models and their attribution maps to computationally score the functional consequence of single-nucleotide substitutions on molecular phenotypes. Sensitivity analysis directly enables this by quantifying how output changes when specific bases are altered.

• Maps delta scores to pathogenicity
• Benchmarked against Deep Mutational Scans (DMS)
• Critical for clinical variant interpretation
• Integrates with ACMG classification guidelines

The quantification of statistical confidence or credible intervals associated with importance scores assigned to genomic features. Sensitivity analysis must account for uncertainty in both the input perturbations and the model's response to produce reliable, reproducible interpretations.

• Bayesian approaches for posterior distributions
• Bootstrap resampling of attribution maps
• Essential for high-stakes clinical applications
• Prevents over-interpretation of noisy gradients