Saliency Map

The most common saliency technique computes the gradient of the agent's chosen action-value (Q-value) or policy logit with respect to the input pixels. A high-magnitude gradient indicates that a small change in that pixel would drastically alter the decision, marking it as salient.

• Vanilla Gradient: Simple backpropagation to the input layer.
• Integrated Gradients: Accumulates gradients along a path from a baseline to the input, satisfying the completeness axiom.
• SmoothGrad: Averages gradients over noisy copies of the input to reduce visual noise.

This causal approach systematically occludes or modifies regions of the input state and measures the resulting drop in the agent's policy probability or Q-value. A significant performance drop indicates that the occluded region was critical for the decision.

• Occlusion Sensitivity: Slides a gray square across the input, plotting the probability of the chosen action.
• RISE (Randomized Input Sampling): Generates random masks to probe the input space efficiently.
• Meaningful Perturbations: Learns a minimal mask that maximally suppresses the target class score.

CAM-based techniques leverage the global average pooling layer in convolutional architectures to project the weights of the output layer back onto the final convolutional feature maps. This produces a coarse, class-specific localization map without requiring backward gradients.

• Grad-CAM: Uses the average gradient of the target score flowing into the final conv layer to weight feature maps, making it applicable to any CNN architecture.
• Grad-CAM++: Adds pixel-wise weighting for better localization of multiple instances of an object.
• Score-CAM: Eliminates gradient dependency by using the increase in confidence as the weighting mechanism.

In cooperative multi-agent reinforcement learning (MARL), saliency maps are extended to attribute credit not just to spatial features, but to specific agents. This explains which teammate's state or action most influenced a centralized critic's value estimate.

• Agent-Specific Saliency: Computes the gradient of the joint Q-value with respect to a specific agent's observation.
• Interaction Maps: Visualizes the attention weights between agents in graph-based or transformer-based MARL architectures.
• Shapley Q-Values: Decomposes the team reward into marginal contributions, visualized as a bar chart of agent importance.

For sequential decision-making, temporal saliency identifies which past time steps or frames in a trajectory are most responsible for the current action. This explains the agent's memory horizon and temporal dependencies.

• Attention Rollout: Propagates attention weights through transformer layers to estimate the flow of information from past states to the current action.
• Input Perturbation Across Time: Systematically drops past frames to measure the impact on the current policy output.
• RNN Gate Analysis: Examines the forget and input gates of recurrent networks to see which historical states are retained.

The faithfulness of a saliency map is quantified by how well it correlates with the model's true reasoning process. Common evaluation protocols involve measuring the drop in performance when inputs are removed in order of attributed importance.

• Deletion Metric: Measures the area under the probability curve as pixels are removed from most to least salient. A steep drop indicates a faithful map.
• Insertion Metric: Measures the probability increase as pixels are added back from most to least salient.
• Pointing Game: Checks if the most salient pixel falls within a human-annotated bounding box of the target object.