Attention Map

An attention map is fundamentally a heatmap matrix where each cell (i, j) represents the attention weight or score between element i in one sequence (e.g., the query) and element j in another (e.g., the key). This visualizes which parts of the input the model's "focus" flows between during processing.

• Rows typically correspond to output positions or query tokens.
• Columns correspond to input positions or key/value tokens.
• The intensity of each cell indicates the strength of the connection, revealing the model's internal alignment between concepts.

Transformer models use multi-head attention, where multiple independent attention mechanisms operate in parallel. This creates not one, but many attention maps per layer.

• Each attention head can learn to focus on different types of relationships (e.g., syntactic, semantic, long-range).
• Layer-wise analysis shows how attention patterns evolve: lower layers often capture local syntax, while higher layers build complex, task-specific representations.
• For traceability, comparing maps across heads and layers is essential to understand the compositional reasoning of an agent.

Attention weights provide a form of feature attribution, quantifying the contribution of each input element to a given output. This is critical for algorithmic explainability.

• High attention to a specific token provides a saliency trace for a model's decision.
• Techniques like attention rollout or attention flow aggregate weights across layers to estimate the total influence of input tokens on final predictions.
• This quantitative data forms the basis for provenance chains, linking outputs back to specific inputs.

Analyzing patterns within attention maps uncovers the model's implicit reasoning strategies. Common patterns include:

• Diagonal Attention: Common in auto-regressive decoding, where a token attends to its immediate predecessor.
• Vertical Attention: A query token attending broadly to all keys, indicating aggregation of global context.
• Sparse, Focused Attention: Sharp focus on a few key tokens, suggesting precise information retrieval.
• Cyclic or Skip Connections: Attention to non-adjacent tokens, revealing latent reasoning paths and long-range dependencies.

Attention maps can be analyzed in two key modes for observability:

• Static Analysis: Examining maps from a single forward pass to debug a specific agent decision. This is used for post-hoc auditing of a reasoning trace.
• Dynamic Analysis: Monitoring how attention patterns shift across different inputs, agent states, or during reflection cycles. Changes in attention can signal a shift in reasoning strategy or a world model update.
• Dynamic analysis is crucial for agentic anomaly detection, identifying when attention deviates from established, trustworthy patterns.

While invaluable, attention maps are not a perfect proxy for model reasoning. Key limitations include:

• Attention is not explanation: High attention weight does not always equate to causal importance for the output; it can be a side effect of the architecture.
• Averaging can mislead: Mean attention across heads or layers may obscure head-specific, specialized roles.
• Non-attention pathways: Information also flows through feed-forward networks and residual connections, which are not captured.
• Therefore, attention maps are one component of a comprehensive agent reasoning traceability system, best used alongside techniques like saliency traces and provenance chains.

A saliency trace is an observability record that highlights which specific parts of the input data were most influential for a model's decision. Unlike an attention map that shows pairwise weights across a sequence, a saliency trace often uses gradient-based methods (like Grad-CAM or Integrated Gradients) to attribute importance scores to individual input features (e.g., pixels in an image or tokens in text).

• Purpose: Answers "Why did the model focus here?" for debugging and feature attribution.
• Key Difference: Attention maps are intrinsic to transformer forward passes; saliency traces are often post-hoc analyses applied to any model.
• Use Case: Explaining why a vision agent classified an object by highlighting the relevant pixels in the input image.

Stepwise rationale is the sequential, human-readable log of an AI agent's internal reasoning process. It documents each logical inference, assumption, and deduction in natural language, creating a narrative of the agent's "thought process."

• Format: Typically plain text or structured JSON logs.
• Relation to Attention: While an attention map shows where the model looked, a stepwise rationale explains what it concluded at each step.
• Primary Use: Critical for Chain-of-Thought (CoT) prompting, where the model is instructed to "think aloud" to improve answer accuracy and provide transparency.

Internal monologue refers to the stream-of-consciousness, natural language reasoning that an AI agent generates for its own intermediate computation. This trace is used for planning and self-correction but is typically hidden from the final user output.

• Key Characteristic: It is a raw, unfiltered reasoning trace not intended for end-user consumption.
• Distinction from Output: The agent's final answer is a polished summary; the internal monologue contains the messy, exploratory steps.
• Observability Value: Accessing the internal monologue provides the deepest view into the agent's cognitive process, revealing dead ends, uncertainties, and refinement cycles.

A cognitive trajectory is the chronological sequence of an agent's internal states or reasoning steps, plotted through a conceptual or latent space. It visualizes the "path" the agent's reasoning took from problem to solution.

• Representation: Often visualized as a line or path through a 2D/3D projection of a high-dimensional thought vector space.
• Analogy: Like a flight path on a map, but for reasoning.
• Insight Provided: Helps identify reasoning loops, divergences, or inefficient meandering. It connects discrete steps (like those in a stepwise rationale) into a continuous journey.

A latent reasoning path is the sequence of transformations within a neural network's hidden layer representations that corresponds to the model's internal, non-observable processing steps. It is the most fundamental, sub-symbolic trace of computation.

• Granularity: Operates at the level of activation vectors and weight matrices, not human-interpretable concepts.
• Relation to Attention: The attention map is a specific, interpretable slice of the broader latent reasoning path, showing interactions between token representations.
• Analysis Challenge: Requires sophisticated techniques (like probing or representation analysis) to map these latent states to human-understandable concepts.

A retrieval trace is an observability record that logs when, why, and what information an agent fetched from an external knowledge source. It is crucial for understanding the grounding of an agent's reasoning in Retrieval-Augmented Generation (RAG) architectures.

• Records: The query issued, the source (e.g., vector database chunk ID, search API result), the retrieved content, and the relevance score.
• Purpose: Provides attribution for generated facts and allows auditing of whether the agent used correct, up-to-date information.
• Connection to Attention: In a RAG model, an attention map may show how the model attends to both the original prompt and the retrieved context passages.