Occlusion Sensitivity

Occlusion sensitivity operates by systematically occluding (e.g., masking with a gray patch) different contiguous regions of an input, such as an image. For each occluded variant, the model makes a new prediction. The core output is a saliency map where the importance of each region is quantified by the resulting drop in the model's prediction score (e.g., logit or probability) for the original class. This directly measures the causal impact of local information on the output.

• Example: In an image classifier for 'dog', occluding the dog's face causes a large prediction drop, highlighting that region as important.
• Key Insight: It is a model-agnostic method; it treats the model as a black box, requiring only forward passes.

The primary strength of occlusion sensitivity is its strong theoretical link to causal faithfulness. By physically removing information and observing the effect, it provides a more direct measure of feature importance than gradient-based methods, which can be sensitive to saturation and noise. This makes it a common ground truth for validating other explanation methods like Grad-CAM or Integrated Gradients.

• Validation Role: A high faithfulness score for another method means its importance map correlates highly with the prediction drops measured by occlusion.
• Limitation: The causal interpretation is strongest when the occluding patch (e.g., gray) is truly uninformative and doesn't introduce new, confounding signals.

The generated saliency map is highly sensitive to several key hyperparameters, which must be carefully controlled during evaluation:

• Occlusion Patch Size: A large patch may obscure too much, causing global prediction collapse; a small patch may not remove enough semantic information, leading to noisy maps.
• Patch Stride: The step size for sliding the occlusion window. A stride of 1 gives a dense map but is computationally expensive; a larger stride is faster but produces coarser, lower-resolution maps.
• Occlusion Value: The pixel intensity or token used to replace the occluded region (e.g., mean image intensity, black, gray, or noise). This must be chosen to minimize introducing out-of-distribution artifacts that the model might react to anomalously.

A naive implementation requires N forward passes, where N is the number of occluded regions (e.g., patches in an image). This is prohibitively expensive for large inputs or models, making it often unsuitable for real-time use.

• Common Mitigations: Use a larger stride or random sampling of patches to approximate the full map.
• Benchmarking Use: Its high cost often relegates it to an offline validation tool rather than a production explanation method. It is a benchmark for evaluating faster, amortized methods.

Occlusion sensitivity maps are used to compute objective scores for evaluating other explanation methods. Key metrics derived from it include:

• Infidelity: Measures if an explanation's importance scores correctly rank the impact of perturbations. High infidelity means the explanation fails to predict which occlusions cause the biggest prediction drop.
• Sufficiency: Checks if the top-K most important features (per another method) are sufficient for the model's prediction. If occluding everything except those top-K features causes a large prediction drop, the explanation is insufficient.
• Completeness: Ensures an explanation accounts for all important features. If occluding only the features deemed important by an explanation causes the full prediction drop, the explanation is complete.

While conceptually clear, occlusion sensitivity has notable limitations that affect its interpretation:

• Edge Artifacts: Occlusion patches create hard edges, which convolutional neural networks are particularly sensitive to, potentially attributing importance to edges rather than semantic content.
• Context Destruction: Occluding a region destroys not just the object but also its spatial relationships and context, which may be crucial for the prediction.
• Baseline Problem: The choice of occlusion value acts as a baseline. An inappropriate baseline (e.g., bright white for a medical X-ray) can put the model far out of distribution, making the prediction drop meaningless.
• Lack of Positive Attribution: It primarily identifies features whose absence hurts the prediction, not features whose presence strongly supports it (though this can be inferred).

Occlusion sensitivity is a primary tool for diagnosing why a convolutional neural network (CNN) makes a specific classification. By sliding a gray or black patch (an occluder) across an image and plotting the resulting drop in predicted probability, engineers create a heatmap. This visually identifies if the model is focusing on semantically correct regions (e.g., a dog's face for a 'dog' class) or spurious correlations (e.g., a watermark or background texture).

• Key Benefit: Provides an intuitive, visual failure mode analysis.
• Example: Revealing that a 'wolf' classifier relies on the presence of snow in the background, rather than animal morphology, indicating a dataset bias.

Occlusion serves as a ground-truth perturbation to benchmark the faithfulness of gradient-based or surrogate explanation methods like Grad-CAM, Integrated Gradients, or LIME. Since occlusion directly measures the causal impact of removing features, it provides a robust reference.

• Process: Correlate the importance scores from a faster attribution method with the actual prediction drop observed during occlusion.
• Outcome: Low correlation suggests the faster method may be generating misleading or unfaithful saliency maps, guiding engineers toward more reliable techniques.

In domains like radiology, where model decisions directly impact patient care, occlusion sensitivity is crucial for auditing model focus. It answers the critical question: "Is the AI looking at the correct anatomical structure?"

• Application: Validating that a model detecting pulmonary nodules focuses on lung tissue, not surrounding ribs or imaging artifacts.
• Regulatory Value: Provides evidence for algorithmic explainability required by frameworks like the EU AI Act, demonstrating that the model's reasoning aligns with medical expertise.

Occlusion sensitivity helps dissect adversarial attacks—inputs subtly perturbed to cause misclassification. By occluding parts of an adversarial image, researchers can determine if the adversarial noise is localized or diffuse and understand which regions the attack has made critically influential.

• Insight Generated: Shows whether a small, perturbed patch is solely responsible for the incorrect prediction, informing the design of more robust defenses.
• Link to Robustness: A model whose saliency map shifts dramatically under a tiny adversarial perturbation is exhibiting a lack of explanation robustness.

Systematically applying occlusion sensitivity across a dataset can uncover systematic biases. If models consistently attribute high importance to non-causal background features (e.g., copyright tags, specific lighting), it signals a flaw in the training data distribution.

• Actionable Output: Guides data curation and augmentation efforts to reduce spurious correlations.
• Quantitative Measure: The aggregate shift in prediction when occluding a suspected bias feature provides a metric for bias severity.

Beyond input features, occlusion can be applied to intermediate feature maps or model layers. By occluding specific channels in a convolutional layer, researchers can probe the function of learned features.

• Use Case: Determining if certain filters consistently respond to specific shapes, textures, or higher-level concepts.
• Outcome: Informs network pruning decisions (removing unimportant filters) and provides insights for neural architecture search by evaluating feature utility.

Perturbation analysis is the general family of explanation validation techniques to which occlusion sensitivity belongs. It involves systematically modifying or removing parts of an input and observing the resulting change in the model's output to infer feature importance.

• Core Principle: If perturbing a feature causes a large change in the prediction, that feature is deemed important.
• Methods include: Occlusion, feature ablation, and input masking.
• Key Use: Provides a model-agnostic way to test causal relationships between inputs and outputs.

A saliency map is a visual heatmap that highlights the regions of an input (most commonly an image) that most influenced a model's prediction. Occlusion sensitivity is one algorithm for generating such maps.

• Visual Output: Produces a pixel-wise importance score overlay.
• Comparison: Unlike gradient-based methods (e.g., Grad-CAM), occlusion-based saliency maps are computed via forward passes with masked inputs, making them computationally more expensive but intuitive.
• Application: Critical in computer vision for debugging model focus, such as verifying a medical imaging model is looking at the correct anatomical region.

The faithfulness score is a quantitative metric that evaluates how accurately an explanation reflects the true reasoning process of the underlying model. Occlusion sensitivity maps are often evaluated using faithfulness metrics.

• Measurement: Typically involves incrementally removing features deemed important by the explanation and measuring the correlation between the drop in model prediction score and the explanation's importance scores.
• Goal: A high faithfulness score indicates the explanation correctly identifies features the model actually uses, not just correlated signals.
• Example Metric: Infidelity measures the expected error between explanation-based perturbations and actual model output changes.

In explainability, sensitivity analysis refers to evaluating how small, meaningful changes to input features affect both the model's prediction and the generated explanation. It tests the robustness and stability of explanations like those from occlusion.

• Process: Applies controlled perturbations (e.g., slight rotations, brightness changes in images) and observes if the explanation changes erratically.
• Output: A stability score can be derived, measuring explanation consistency under perturbation.
• Importance: An explanation method that is highly sensitive to insignificant input noise is less reliable for human trust and debugging.

The model randomization test (or randomization test) is a sanity check for feature attribution methods like occlusion sensitivity. It verifies if the explanation method produces meaningfully different results for a trained model versus a randomly initialized one.

• Procedure: 1. Generate an explanation for a trained model. 2. Randomize the model's weights (destroying its knowledge). 3. Generate an explanation for the randomized model. 4. Compare the two explanations.
• Valid Result: Explanations for the trained and randomized models should be significantly different. If they are similar, the explanation method may not be detecting learned patterns but merely architectural artifacts.

Local fidelity is a desired property of a post-hoc explanation, measuring how well it approximates the complex model's behavior in the immediate vicinity of a specific input instance. Occlusion sensitivity is evaluated on this criterion.

• Definition: An explanation with high local fidelity accurately predicts how the model would behave if the input were slightly perturbed.
• Contrast with Global Fidelity: It does not claim to explain the model everywhere, only locally around the example.
• Connection to LIME: Methods like LIME explicitly train a simple, interpretable model (e.g., linear regression) locally to achieve high local fidelity.