Attribution Attack

The attacker simultaneously optimizes for two conflicting goals: preserving the original classification and manipulating the explanation. This is typically formulated as a loss function with two components:

• A classification loss that penalizes any change to the model's top-1 prediction
• An attribution loss that guides the saliency map toward a target explanation chosen by the attacker

The result is a perturbed input that looks nearly identical to the original but redirects the explanation to a benign or misleading region.

Unlike standard adversarial examples that cause misclassification, attribution attacks are designed to evade accuracy-based monitoring systems. Key properties:

• The model's predicted class and confidence score remain statistically unchanged
• Standard performance metrics show no degradation
• The attack bypasses anomaly detection that flags inputs causing prediction shifts
• Only the explanation diverges, making detection require specialized explanation auditing

This stealth property makes attribution attacks particularly dangerous in regulated domains where explanations are used for compliance.

The attacker specifies a target saliency map that the manipulated input should produce. Common attack objectives include:

• Hiding sensitive regions: Making the model appear to ignore a tumor while still correctly classifying the scan as malignant
• Redirecting to confounders: Forcing the explanation to highlight irrelevant background features or artifacts
• Forging clinical plausibility: Creating explanations that point to anatomically incorrect but seemingly reasonable regions
• Undermining audit trails: Generating explanations that would mislead a clinician-in-the-loop reviewing the AI's reasoning

Most attribution attacks leverage gradient information from the explanation method itself. Common approaches:

• White-box attacks: Assume full access to model weights and gradients, enabling precise optimization against specific attribution methods like Grad-CAM or Integrated Gradients
• Universal perturbation attacks: Generate a single noise pattern that manipulates explanations across many inputs
• Adaptive attacks: Dynamically adjust the perturbation strategy based on the specific explanation method being targeted

The attacker computes gradients of the explanation output with respect to the input, then applies projected gradient descent to iteratively refine the perturbation.

Attribution attacks pose a direct threat to regulatory explainability requirements under frameworks like the FDA's SaMD guidance and the EU MDR. Consequences include:

• Audit trail contamination: Manipulated explanations become part of the permanent SaMD audit trail, corrupting post-market surveillance
• Trust calibration failure: Clinicians may develop misplaced trust in the system if explanations appear consistently plausible but are adversarially forged
• Liability gaps: When a misdiagnosis occurs, the manipulated explanation may point to the wrong region, obscuring the true cause of the error
• Compliance violations: Regulators require that explanations faithfully reflect model reasoning; manipulated explanations violate this requirement

Defending against attribution attacks requires explanation-specific robustness measures beyond standard adversarial training:

• Explanation consistency checks: Comparing saliency maps from multiple attribution methods on the same input to detect discrepancies
• Smoothness regularization: Training with explanation regularization penalties that enforce Lipschitz continuity in the explanation function
• Randomized smoothing: Adding noise during explanation generation to certify robustness within a bounded perturbation radius
• Faithfulness monitoring: Continuously measuring the faithfulness score of explanations in production to detect degradation
• Adversarial training for explanations: Augmenting training data with attribution-attacked examples

The general class of methods that assign a relevance or importance score to each input feature of a model, quantifying its contribution to the model's specific output prediction. In medical imaging, this typically means identifying which pixels or voxels in a scan influenced a diagnostic classification.

• Post-hoc explainability applies these methods after training without modifying the model
• Forms the foundation for saliency maps, Grad-CAM, and Integrated Gradients
• Attribution attacks directly manipulate these importance scores while preserving the original classification

A visualization that highlights the pixels or regions of an input image that most strongly influence a model's classification decision. Typically computed by taking the gradient of the class score with respect to the input image.

• 2D saliency maps highlight influential pixels in X-rays and mammograms
• 3D saliency maps extend this to volumetric CT and MRI voxel data
• An attribution attack can produce a saliency map pointing to healthy tissue while the model correctly identifies a lesion, destroying clinical trust

A quantitative metric that evaluates the accuracy of an explanation by measuring how well the attributed importance scores correlate with the actual change in model output when the corresponding features are perturbed or removed.

• Perturbation-based evaluation removes highly attributed pixels and checks if the prediction changes
• Deletion and insertion metrics track the curve of probability change as features are removed or added
• Low faithfulness scores are a primary indicator that an attribution attack may have succeeded

The property of a machine learning model to maintain its prediction accuracy when presented with adversarially perturbed inputs. These inputs are intentionally modified with small, often imperceptible, noise to cause misclassification.

• Standard adversarial attacks target the classification output directly
• Attribution attacks are a stealthier variant that preserves correct classification while corrupting the explanation
• Robustness training against both attack types is essential for regulatory-grade diagnostic AI

The false sense of security or understanding that can arise from viewing a plausible-looking but ultimately unfaithful or misleading explanation. The saliency map appears clinically reasonable but does not accurately reflect the model's true reasoning process.

• Attribution attacks are engineered to create precisely this illusion
• A radiologist may trust a corrupted saliency map that highlights anatomically plausible but incorrect regions
• Trust calibration through rigorous faithfulness testing is the primary defense against this cognitive vulnerability