Faithfulness Metric

The foundational logic of faithfulness relies on a causal intervention: if the highlighted features are truly important, removing or altering them must break the prediction.

• Positive Perturbation (Insertion): Starting from a neutral baseline (e.g., a blurred image), features are added in order of attributed importance. A faithful map shows a steep, monotonic increase in model confidence.
• Negative Perturbation (Deletion): Starting from the original input, the most salient features are progressively removed or obfuscated. A faithful map exhibits a sharp, rapid drop in the target class probability.
• Baseline Selection: The choice of replacement value (zero, mean, Gaussian noise, or a learned inpainting mask) critically impacts the metric's validity and must be carefully calibrated to avoid out-of-distribution artifacts.

These metrics generate comprehensive area-under-the-curve (AUC) scores rather than single-point estimates, providing a holistic view of attribution quality.

• Deletion Metric: Measures the drop in probability as the top k most important pixels are sequentially removed. A lower AUC indicates a more faithful explanation, as the model's confidence collapses quickly.
• Insertion Metric: Measures the recovery of probability as the top k pixels are added back to a blurred baseline. A higher AUC signifies a more faithful explanation, as the model rapidly reconstructs its decision.
• Comprehensive Evaluation (F1-Score): Often, the harmonic mean of the normalized deletion and insertion scores is used to balance the assessment of both foreground and background feature importance.

ROAR is a rigorous, computationally intensive protocol designed to expose the information leakage of an attribution method. It tests if the map truly captures the features the model uses rather than just the features it sees.

• Procedure: A fraction (e.g., 10-30%) of the most important pixels, as ranked by the attribution map, are replaced with a constant value across the entire dataset. The model is then fully retrained from scratch on this modified data.
• Interpretation: If the map is faithful, the model retrained on the perturbed data should show a significant drop in accuracy compared to a model retrained on randomly perturbed data. If the accuracy remains high, the original model was likely using alternative, unhighlighted features.
• Iterative Application: The process is repeated iteratively, removing more features each round, to track the degradation curve over multiple retraining cycles.

These metrics evaluate faithfulness by analyzing the mathematical relationship between perturbations and output changes, often without requiring a full model retraining.

• Sensitivity-N: A sample-based metric that measures the maximum change in the model's output when a random subset of N input features is perturbed. It correlates the magnitude of the output change with the sum of the attribution scores of the perturbed features.
• Infidelity: A formal measure of the expected prediction error between the original output and a first-order Taylor expansion of the model around a significant perturbation. It quantifies how well the attribution map serves as a local linear explanation.
• Max-Sensitivity: A robustness check that calculates the maximum change in the explanation itself when the input is subjected to small, imperceptible perturbations, ensuring the attribution map is stable.

A mechanistic interpretability technique that tests a hypothesized circuit (a subgraph of the model) by surgically altering activations. It is the gold standard for verifying if a specific set of neurons or attention heads is both necessary and sufficient for a behavior.

• Hypothesis Testing: An interpretability researcher identifies a specific path of weights (e.g., an attention head that moves a name to a specific residual stream position).
• Activation Resampling: The activations of the hypothesized circuit are replaced with activations from a 'corrupted' forward pass (e.g., one where the subject is changed). If the circuit is correct, the model's output should fully reflect the corrupted input, and the loss should recover to the level of the corrupted baseline.
• Faithfulness Check: The metric is the degree to which the scrubbed model's loss matches the loss of the fully corrupted model, confirming that no other components are secretly computing the same function.

While not a direct measure of model-internal faithfulness, alignment with human expert annotations serves as a crucial external validity check, especially in high-stakes domains like medical imaging.

• Pointing Game: A binary metric that checks if the most attributed pixel falls within a human-annotated bounding box of the target object. It measures localization accuracy but not the completeness of the explanation.
• Intersection over Union (IoU): Measures the overlap between a thresholded attribution heatmap and a human segmentation mask. It penalizes both over-attribution to background and under-attribution to the object.
• Plausibility vs. Faithfulness: A critical distinction. A map can be highly plausible to a human (highlighting the object) but unfaithful to the model (which might be using a spurious background correlation). Human correlation is a proxy, not a direct measure, of true faithfulness.

A core faithfulness evaluation protocol that measures the drop in a model's prediction probability as the most important pixels—as ranked by an attribution map—are iteratively removed and replaced with a baseline value.

• Sharp drop in probability indicates a faithful attribution map.
• Gradual decay suggests the map is not identifying the truly critical features.
• Often paired with the Insertion Metric for a comprehensive evaluation.

The inverse of the Deletion Metric. It starts with a heavily blurred or baseline image and incrementally adds pixels from most to least important according to the attribution map.

• A rapid increase in the model's probability signifies a high-quality explanation.
• Measures how efficiently an explanation captures the signal needed for a correct classification.
• Evaluates the sufficiency of the identified features.

A rigorous evaluation protocol that goes beyond simple input perturbation. After ranking features by importance, a fraction of the most important features are physically removed from the dataset, and the model is retrained from scratch on this modified data.

• Tests if the model relies on the identified features for its fundamental mapping.
• If accuracy degrades sharply, the attribution method is validated.
• Prevents the model from simply relying on secondary, unmodified features.

A mechanistic interpretability technique used to test a hypothesized circuit within a neural network. It works by resampling activations from a corrupted dataset and verifying that the model's behavior is fully mediated by the identified subgraph.

• Replaces specific node activations with values from a counterfactual input.
• If the model's output changes exactly as predicted by the circuit hypothesis, the explanation is causally sound.
• Provides a formal test of causal faithfulness.

A theoretical framework that defines desirable mathematical properties a feature attribution method must satisfy to be considered reliable. Key axioms include:

• Completeness: The sum of feature attributions equals the model's output difference from a baseline.
• Sensitivity: If an input changes and the output changes, the differing feature must get a non-zero attribution.
• Implementation Invariance: Two functionally equivalent models must produce identical attributions.

A metric that evaluates attribution quality by measuring the correlation between the number of features perturbed and the resulting change in the model's output. It randomly masks subsets of the n most important features and checks if the prediction changes monotonically.

• A faithful map will show a strong negative correlation between the number of top features removed and the model's confidence.
• Less sensitive to baseline choice than simple deletion metrics.