ROAR

ROAR evaluates attributions by creating a progressive masking curve. In each iteration:

• A fixed fraction (e.g., 10%) of the most important features is removed according to an attribution map
• The model is fully retrained from scratch on the modified dataset
• Performance is measured on the original, unmodified test set This process repeats until all features are removed, generating a degradation curve that reveals how well the attribution method identifies truly critical features.

ROAR serves as a ground-truth benchmark for faithfulness metrics. A faithful attribution method should cause a steep, immediate drop in performance when its top-ranked features are removed and the model is retrained. Key evaluation patterns:

• Steep initial decay: Indicates the attribution correctly identified indispensable features
• Gradual linear decay: Suggests the attribution is no better than random ranking
• Flat curve with late drop: Implies the model relies on features the attribution method missed The area under the degradation curve provides a single quantitative score for comparing attribution methods.

The mandatory retraining step distinguishes ROAR from simpler perturbation-based evaluations like the deletion metric. Without retraining:

• Models encounter out-of-distribution inputs they were never trained to handle
• Performance drops may reflect model fragility rather than true feature importance
• Models may rely on spurious correlations in the remaining features By retraining, ROAR forces the model to adapt to the modified feature space, revealing whether the removed features were genuinely irreplaceable or if the model can recover using alternative signals.

ROAR addresses critical limitations of standard deletion and insertion metrics:

• Deletion metric: Measures prediction drop when features are removed from a single frozen model; confounds model robustness with feature importance
• Insertion metric: Measures recovery when features are added back; assumes a blurred baseline that may be unrealistic
• ROAR advantage: Eliminates the out-of-distribution confound by allowing the model to relearn from the remaining features after each removal step This makes ROAR the gold standard for evaluating whether attributions reflect the model's true reliance structure rather than coincidental activation patterns.

Two primary variants of the ROAR framework exist:

• Standard ROAR: Features are physically removed from the dataset (set to zero or mean), and the model is retrained on the impoverished input
• ROAR-KL: Instead of removing features, the model is retrained with a KL divergence penalty that discourages it from using the top-ranked features while still having access to them ROAR-KL tests whether the model can achieve similar performance without the flagged features, providing a more nuanced measure of whether those features are truly necessary or merely convenient.

ROAR's primary drawback is its extreme computational expense:

• Requires retraining the model from scratch at each masking threshold (typically 10 iterations)
• For large models, this can mean 10x the training budget of a single experiment
• Results can be sensitive to the choice of imputation strategy for removed features (zero-filling, mean imputation, or learned inpainting)
• The retraining process may introduce variance from different random initializations Despite these costs, ROAR remains the most principled empirical test of attribution faithfulness available.

A quantitative evaluation criterion that measures how accurately an attribution map reflects the true importance of features. The core principle: if you perturb the inputs an explainer identifies as most important, the model's prediction should change significantly.

• Directly operationalizes the ROAR protocol's logic
• Measures correlation between attribution rank and prediction impact
• Essential for comparing SHAP, LIME, and gradient-based methods

An evaluation metric that measures attribution quality by incrementally removing pixels from most to least important and recording the decay in prediction probability. A steep, early drop indicates a faithful explanation.

• Complements the Insertion Metric
• Used in ROAR to quantify performance degradation
• Lower Area Under the Curve (AUC) = better attribution

An evaluation metric that measures attribution quality by incrementally adding pixels from most to least important into a blurred baseline and recording the increase in model probability. A sharp, early rise signals accurate feature importance.

• Inverse of the Deletion Metric
• Tests if highlighted features are sufficient for classification
• Higher AUC = better attribution map

A theoretical framework defining desirable mathematical properties a feature attribution method must satisfy to be considered reliable. Key axioms include:

• Completeness: Attributions sum to the prediction difference from baseline
• Sensitivity: Zero-difference features get zero attribution
• Implementation Invariance: Identical functions get identical attributions
• Integrated Gradients uniquely satisfies all axioms

A model-agnostic global explanation method that measures the drop in model performance after randomly shuffling a single feature's values. Breaking the association between feature and target reveals its importance.

• Computed on held-out validation data
• Repeated shuffles reduce variance
• Captures both main effects and interactions
• ROAR extends this logic with iterative retraining

A mechanistic interpretability technique that tests a hypothesized circuit by resampling activations from a corrupted dataset. If the model's behavior is fully mediated by the identified subgraph, the scrubbed activations should produce identical outputs.

• Validates whether identified features are necessary and sufficient
• Complements ROAR's behavioral testing with structural analysis
• Key technique in transformer mechanistic interpretability