Sensitivity Analysis

Sensitivity analysis operates by locally perturbing the input features of a single data instance. This involves systematically modifying one or more features—such as slightly increasing a numerical value or masking a token—and observing the resulting change in the model's output score. The core assumption is that a faithful explanation should identify features whose perturbation causes the largest change in prediction. Common perturbation techniques include:

• Occlusion: Setting a feature to zero or a baseline value.
• Noise Injection: Adding small Gaussian noise.
• Feature Swapping: Replacing a feature with values from a different instance.
• Gradient-based Sampling: Perturbing features in the direction of the gradient.

The primary output of sensitivity analysis is a set of quantitative metrics that score the explanation's alignment with the model's true behavior. These metrics transform subjective assessment into objective, comparable numbers. Key metrics derived from sensitivity analysis include:

• Infidelity: Measures the expected squared error between the explanation's importance scores and the actual change in model output when the input is perturbed. Low infidelity indicates high faithfulness.
• Sensitivity-n: Measures the correlation between the magnitude of feature importance scores and the magnitude of prediction change when those features are perturbed.
• Monotonicity: Assesses whether progressively removing features in order of decreasing importance (according to the explanation) causes the model's prediction confidence to drop monotonically.

A defining characteristic is that sensitivity analysis is model-agnostic. It treats the machine learning model as a black-box function, requiring only the ability to query it with inputs and receive outputs. This makes it applicable to any model type—deep neural networks, gradient-boosted trees, or proprietary APIs—and any explanation method, including SHAP, LIME, and Integrated Gradients. The validation does not rely on internal model weights or architectures, focusing solely on the input-output relationship. This external perspective is crucial for auditing systems where internal states are inaccessible.

Sensitivity analysis provides a distinct, complementary perspective to gradient-based attribution methods like Saliency Maps or Integrated Gradients. While gradient methods analyze the local slope of the model's output function, sensitivity analysis measures the actual change in output due to finite perturbations. This is critical because gradients can be saturated or noisy, especially for models with non-linear activation functions like ReLU, where a zero gradient does not necessarily mean the feature is unimportant. Sensitivity analysis validates whether the theoretical importance suggested by gradients manifests in actual output changes.

A robust explanation method should produce stable attributions under small, semantically meaningless input changes. Sensitivity analysis is used to test explanation robustness by applying perturbations that should not alter a human's reasoning for the prediction. If an explanation changes drastically under minor noise, it indicates instability and low trustworthiness. This test helps identify explanation methods vulnerable to adversarial attacks aimed at manipulating interpretability outputs without changing the prediction, a significant concern for regulatory audits and high-stakes deployments.

The results of sensitivity analysis are highly dependent on the choice of baseline (the reference input for perturbations) and perturbation distribution. For example, occluding a pixel in an image requires defining what value replaces it (e.g., black, gray, or mean pixel value). These choices are not neutral; they embed assumptions about the data manifold. Consequently, a complete sensitivity analysis report must specify:

• Baseline Input: The reference point (e.g., zero vector, blurred image, average instance).
• Perturbation Magnitude: The size of the change (e.g., ε for noise).
• Perturbation Distribution: The statistical distribution of the noise or replacement values.
• Number of Samples: The Monte Carlo samples used to estimate expected changes.

Sensitivity analysis is used to benchmark explanation methods like SHAP, LIME, and Integrated Gradients. By applying small, controlled perturbations to input features and observing changes in both the prediction and the attribution scores, data scientists can quantify an explanation's local fidelity. A robust method will show attribution scores that change predictably with the perturbation, confirming the explanation accurately reflects the model's local behavior.

In high-stakes applications like fraud detection, sensitivity analysis probes model decision boundaries. Analysts systematically adjust transaction features (e.g., amount, location, time) to see if small, plausible changes cause the model to flip its prediction from 'fraudulent' to 'legitimate'. This identifies brittle logic and ensures explanations highlight the truly decisive risk factors, not spurious correlations. It directly supports algorithmic explainability and interpretability requirements for financial regulators.

For medical imaging and diagnostic vision models, sensitivity analysis validates that saliency maps (e.g., from Occlusion Sensitivity) focus on clinically relevant anatomy. By perturbing pixels outside the highlighted region, engineers verify the prediction remains stable, confirming the explanation's sufficiency. Conversely, perturbing pixels within the highlighted region should cause a significant prediction change, confirming its necessity. This process is critical for achieving a high faithfulness score in life-critical systems.

Sensitivity analysis refines counterfactual explanations. After generating a counterfactual (e.g., 'Your loan was denied because income is $5K too low'), analysts test its minimality by making even smaller feature adjustments. If a smaller change also flips the prediction, the original counterfactual is not minimal. This iterative process, often automated, produces more actionable and legally compliant explanations for credit decisions, supporting enterprise AI governance frameworks.

This application is a form of adversarial testing for the explanations themselves. Attackers can craft inputs where the model's prediction is correct, but the accompanying explanation is misleading or highlights irrelevant features. Sensitivity analysis can uncover these vulnerabilities by showing that the explanation is not stable—small, imperceptible perturbations (adversarial examples) cause drastic, unjustified changes in the feature attributions, revealing a failure in explanation robustness.

In Retrieval-Augmented Generation (RAG) architectures, sensitivity analysis evaluates how changes in retrieved document chunks affect the final answer and the model's cited sources. By perturbing or swapping retrieved passages, engineers measure the stability of the answer and the faithfulness of the attribution. A robust system will show answer changes only when core supporting evidence is altered. This is a key component of RAG evaluation metrics for answer engine architecture.

A foundational technique for sensitivity analysis and explanation validation. It involves systematically modifying or removing input features to observe the resulting changes in the model's output. This is the core mechanism behind many sensitivity tests.

• Purpose: To empirically test the causal relationship between specific features and the model's prediction.
• Method: Common approaches include feature occlusion (setting values to zero or mean), adding Gaussian noise, or sampling from a marginal distribution.
• Example: In an image classifier, occluding the region containing a 'dog's head' should cause a large drop in the 'dog' class probability if that feature is truly important.

A quantitative metric that measures how accurately an explanation reflects the true reasoning process of the underlying model. Sensitivity analysis is a primary method for calculating faithfulness.

• Calculation: Often computed as the correlation between the importance scores assigned by an explanation (e.g., SHAP values) and the actual impact on the model output when those features are perturbed.
• High Faithfulness: If features ranked as highly important by an explanation cause large prediction changes when perturbed, the explanation is considered faithful.
• Role in Validation: A core target metric for sensitivity analysis; the goal is to prove an explanation method yields high faithfulness scores.

An explanation metric that quantifies the failure of an explanation to accurately reflect model behavior under perturbation. It is a direct measure of unfaithfulness.

• Definition: Formally, infidelity measures the expected squared error between the explanation's attribution-based prediction and the actual model output change when the input is perturbed.
• Interpretation: A low infidelity score is desirable, indicating the explanation reliably predicts how the model will react to changes.
• Relation to Sensitivity: Infidelity is computed by performing many local sensitivity analyses—perturbing the input according to a distribution and comparing the explanation's estimate of the change to the real change.

Measures the consistency of explanations generated for similar inputs or under small, semantically-preserving perturbations. It assesses the robustness of the explanation method itself.

• Why it Matters: An unstable explanation method produces vastly different importance scores for two nearly identical inputs, undermining trust.
• Sensitivity Test: Stability is evaluated by applying small perturbations (e.g., minor image rotations, synonym replacement in text) and measuring the variation in the resulting explanations (e.g., using Rank-Biased Overlap or Cosine Similarity).
• Distinction from Faithfulness: Stability evaluates the explanation method's consistency, while faithfulness evaluates its accuracy relative to the model.

The property of an explanation method to produce consistent and stable attributions when the input or model is subjected to minor, semantically-preserving changes. It is the qualitative goal measured by the stability score.

• Threats to Robustness: Adversarial examples can be crafted to drastically alter explanations without changing the model's prediction or the input's meaning to a human.
• Engineering Goal: Building explanation methods that are Lipschitz continuous—where small input changes lead to proportionally small changes in the explanation.
• Validation via Sensitivity: Robustness is validated through extensive sensitivity analysis across a distribution of natural and adversarial perturbations.

A critical sanity check for any feature attribution or sensitivity analysis method. It tests whether the explanation method is truly detecting model logic versus producing arbitrary patterns.

• Procedure: 1. Generate explanations using the trained model. 2. Randomize the model's weights (destroying its learned knowledge). 3. Generate explanations for the same inputs using the randomized model.
• Passing the Test: The explanations from the trained and randomized models should be significantly different. If they are similar, the explanation method is not sensitive to the model's actual function.
• Foundation for Trust: This test establishes a baseline; a sensitivity analysis method that fails the randomization test provides no valid insight.