Perturbation Analysis

This mechanism involves systematically removing or masking individual input features (e.g., setting a word's embedding to zero or blurring an image patch) and observing the resulting drop in the model's prediction confidence. It is the most direct form of perturbation.

• Purpose: To empirically test if a feature deemed important by an explanation (like a saliency map) is actually critical for the prediction.
• Example: In an image classifier for 'dog', occluding the pixel region containing the dog's head should cause a significant prediction score decrease.
• Key Metric: The prediction delta quantifies the change, with larger deltas indicating more important features.

Ablation extends occlusion by iteratively removing groups of features based on an explanation's importance ranking. Features are ablated in order of descending attributed importance.

• Purpose: To evaluate the completeness and faithfulness of an explanation. A faithful explanation should see model performance degrade rapidly as top features are removed.
• Process: 1. Generate an explanation (e.g., SHAP values). 2. Sort features by importance. 3. Iteratively ablate the top K% of features and record the output change.
• Analysis: The resulting curve shows how much predictive power is retained; a steep drop confirms the explanation correctly identified core features.

This mechanism applies meaningful, structured noise to the input rather than simple removal, based on the explanation itself. It formally tests infidelity, a core validation metric.

• Principle: Perturb the input along the direction suggested by the explanation's importance scores. A high-quality explanation should correlate with large output changes when perturbed this way.
• Mathematical Basis: Infidelity is defined as the expected squared difference between the model's output change and the dot product of the explanation and the perturbation vector: 𝔼_I[(I^T φ(f,x) - (f(x) - f(x-I)))^2].
• Use Case: Directly quantifies if the explanation φ accurately reflects the model's local gradient behavior.

This mechanism tests the robustness of the explanation method itself by applying small, semantically-invariant perturbations to the input and observing the variance in the generated explanations.

• Goal: Assess explanation stability. A robust method should produce similar explanations for perceptually similar inputs.
• Perturbation Types: Adding minor image noise, synonym replacement in text, or small affine transformations.
• Evaluation: Measures like Local Lipschitz Continuity or the Stability Score calculate the explanation's sensitivity to input noise. High variance indicates an unreliable explanation method.

This mechanism finds the minimal perturbed input that changes the model's prediction to a specified target class. It is a proactive form of perturbation analysis.

• Purpose: To create contrastive explanations that answer "What minimal changes would flip the prediction?"
• Process: Uses optimization or search to perturb an instance (e.g., x) into a counterfactual (x') such that f(x') = y_target, while minimizing a distance metric d(x, x').
• Validation Role: The characteristics of the found counterfactual (which features changed, by how much) can be compared to post-hoc feature attributions to check for consistency in the model's decision boundary.

This mechanism perturbs the model itself rather than the input, by randomizing model parameters across layers, to serve as a sanity check for explanation methods.

• Procedure: 1. Generate explanations for a trained model. 2. Progressively randomize the model's weights (starting from output layers back to inputs). 3. Re-generate explanations after each randomization step.
• Expected Result: A meaningful explanation method should produce significantly different results when the model's predictive capability is destroyed. If explanations remain similar, the method may not be truly dependent on the model's learned representations.
• Outcome: Validates that the explanation method is sensitive to the model's actual function, not just its architecture or the input structure.

A perturbation technique that systematically occludes (blocks or replaces) different regions of an input—such as patches of an image or spans of text—and measures the resulting change in the model's output score. The magnitude of the output drop indicates the importance of the occluded region.

• Primary Use: Generating visual saliency maps for image classifiers and object detectors.
• Method: A sliding window (e.g., a gray square) is passed over the input. For each position, the model's prediction probability for the target class is recorded.
• Output: A heatmap where 'hotter' regions correspond to areas where occlusion caused the largest prediction decrease, signifying high importance.

A technique that ablates (sets to zero, removes, or replaces with a baseline value) individual input features or feature groups to isolate their contribution. The change in the model's prediction is the direct attribution for that feature.

• Baseline Choice: Critical to the method. Common baselines include the feature's mean, median, a zero vector, or a blurred version for images.
• Granularity: Can be applied at the pixel level, word/token level, or for higher-level feature embeddings.
• Validation Role: Directly tests the sufficiency and necessity of features identified by other explanation methods (e.g., SHAP, LIME). If removing a 'high importance' feature causes no prediction change, the original explanation may lack faithfulness.

A global model-agnostic technique that evaluates feature importance by randomly shuffling the values of a single feature across the entire dataset and measuring the resulting degradation in a model performance metric (e.g., accuracy, AUC-ROC).

• Scope: Provides a global importance score, in contrast to local, instance-specific methods.
• Process: 1. Calculate a baseline performance score on a validation set. 2. For each feature, permute its values, breaking its relationship with the target. 3. Re-evaluate performance. The importance is the drop in score.
• Key Insight: Features that cause a large performance drop when shuffled are considered important because the model relied on their true distribution.

A perturbation technique that finds the minimal change required to an input instance to alter the model's prediction to a desired, contrasting outcome. The difference between the original and the counterfactual input defines an explanation.

• Answers 'What-If?': Explains a prediction by showing, "Your loan was denied. If your income had been $5,000 higher, it would have been approved."
• Constraints: Optimizations search for changes that are minimal (small L1/L2 distance), plausible (lies within the data manifold), and actionable (suggests feasible real-world changes).
• Validation Utility: The proximity and sparsity of the generated counterfactual are key metrics for evaluating the explanation's quality and usability.

A gradient-based attribution method that integrates the model's gradients along a straight-line path from a baseline input (e.g., a black image) to the actual input. The integral approximates the feature's cumulative contribution.

• Theoretical Foundation: Satisfies desirable axioms like completeness, where the attributions sum to the difference between the model's output at the input and the baseline.
• Perturbation Path: The core perturbation is the gradual interpolation of features. The method aggregates sensitivity across many infinitesimal steps, avoiding the noise of single-point gradients.
• Baseline Dependency: The choice of baseline (e.g., zero, blurred, random noise) is critical and should represent an 'absence of signal.' The attributions explain the prediction relative to this baseline.

A perturbation-based estimation approach for explanation methods like SHAP (KernelSHAP). It approximates Shapley values by randomly sampling subsets of features, perturbing the unsampled features to their baseline values, and observing the model's output.

• Underlying Principle: Shapley values from game theory require evaluating the model with every possible coalition of features. Monte Carlo sampling makes this computationally feasible for complex models.
• Perturbation Mechanism: For each sampled feature subset, the 'missing' features are replaced with values from a background dataset (the baseline). The model's prediction on this perturbed input represents the value of that coalition.
• Output: Converges to the Shapley value for each feature, representing its average marginal contribution across all possible coalitions.

A faithfulness score is a quantitative metric that measures how accurately an explanation reflects the true reasoning process or causal factors of the underlying model for a given prediction. It is the core criterion perturbation analysis seeks to validate.

• Direct Measurement: Often calculated by perturbing features deemed important by the explanation and measuring the resulting drop in model confidence or change in prediction.
• Contrast with Plausibility: A faithful explanation is causally linked to the model's mechanics, whereas a plausible explanation may simply be convincing to a human but not reflect the model's actual process.

Infidelity is an explanation metric that quantifies the degree to which an explanation fails to accurately reflect the model's output when the input is perturbed according to the explanation's importance scores. It is a formal, mathematical inverse of faithfulness.

• Calculation: Given an importance score vector and a set of perturbations, infidelity measures the expected squared error between the explanation's predicted importance and the actual change in model output.
• Use Case: A primary quantitative measure used in automated validation suites to rank explanation methods. Lower infidelity scores indicate higher explanation quality.

Sensitivity analysis in explainability is a broader evaluation framework that assesses how small changes in input features affect both the model's prediction and the generated explanation. Perturbation analysis is a specific implementation of sensitivity analysis focused on the prediction.

• Dual Assessment: Evaluates not only if the model output changes (prediction sensitivity) but also if the explanation changes (explanation sensitivity).
• Robustness Indicator: High sensitivity in the explanation to minor, meaningless perturbations can indicate an unstable or unreliable explanation method.

Explanation robustness refers to the property of an explanation method to produce consistent and stable attributions for a given prediction when the input or model is subjected to minor, semantically-preserving perturbations.

• Core Concern: A robust explanation should not change drastically if the input image is slightly rotated or a synonym is used in text.
• Perturbation as a Test: Robustness is empirically tested using perturbation suites that apply small, realistic variations to inputs and measure the variance in the resulting explanations.

Sufficiency and Completeness are complementary metrics for evaluating the scope of an explanation.

• Sufficiency: Measures whether the subset of features identified as most important by an explanation is, by itself, sufficient for the model to make its original prediction. Tested by providing only the top-K features to the model.
• Completeness: Evaluates whether an explanation accounts for all features that contributed significantly to the prediction. The sum of importance scores for all features should approximate the model's output deviation from a baseline. Perturbation analysis is used to operationalize both tests.

Local fidelity is a property of a post-hoc explanation that measures how well the explanation approximates the behavior of the complex model in the immediate vicinity of a specific input instance. It is a prerequisite for a faithful explanation.

• Local Surrogate Models: Methods like LIME explicitly optimize for local fidelity by training a simple, interpretable model (e.g., linear regression) on perturbed samples around the instance to be explained.
• Perturbation Boundary: Fidelity is typically high only within a small region around the original input. The validity of an explanation diminishes as perturbations move further from the original instance.