Chain-of-Thought (CoT) Evaluation

This dimension evaluates the internal soundness of the reasoning sequence. A valid trace must avoid contradictions and maintain a clear, logical flow between steps.

• Logical Consistency Check: Verification that no step contradicts a previous assertion or inferred fact.
• Stepwise Coherence Score: A metric quantifying the semantic and logical connectedness between consecutive reasoning steps.
• Causal Link Verification: Ensures stated cause-and-effect relationships are logically justified, not merely correlative.

Example: In a math problem, a step claiming x > 5 must not be followed by a step that uses x = 3 without a valid intervening operation that changes x's value.

This assesses the factual accuracy of information within the reasoning trace itself, not just the final answer. It identifies unsupported or invented content in the model's internal monologue.

• Hallucination Detection in Trace: Flags statements that are factually incorrect or lack evidential support within the provided context.
• Multi-Hop Reasoning Validation: Verifies the model correctly integrates and synthesizes facts across multiple steps or knowledge sources.
• Error Propagation Tracing: Forensic analysis to locate the initial incorrect step and map how its error cascaded to the final output.

This is critical for high-stakes domains like healthcare or finance, where flawed intermediate reasoning can lead to dangerous conclusions.

Evaluation of how well the reasoning adheres to predefined rules, domain constraints, and desired problem-solving strategies. It measures alignment with procedural correctness.

• Specification Compliance Score: Measures adherence to formal rules, safety properties, or operational constraints (e.g., "always verify the user's age before proceeding").
• Tool-Use Rationale Evaluation: Assesses the justification for calling an external tool/API, including selection appropriateness and correctness of expected outcome.
• Formal Verification of Trace: Application of mathematical logic and automated theorem proving to prove the trace satisfies a given property.

This dimension is key for ensuring agents operate within safe, deterministic boundaries.

This evaluates the model's capacity to monitor and regulate its own reasoning process, demonstrating advanced, reflective problem-solving.

• Meta-Cognition Assessment: Measures evidence of reflection, confidence estimation, and strategic planning within the trace.
• Self-Correction Loop Score: Evaluates the effectiveness of mechanisms for detecting internal errors and initiating revisions.
• Counterfactual Trace Analysis: Examining 'what-if' reasoning traces to understand how the model's logic changes with altered premises.

Agents scoring high here are more resilient and reliable, as they can identify and fix their own mistakes.

Measures how interpretable, justifiable, and aligned with human expert reasoning the trace is. This is crucial for trust, debugging, and compliance.

• Explainability Trace Generation: The process of producing a human-interpretable reasoning trace explicitly to justify a final decision.
• Gold Standard Trace Alignment: Comparison against a human-expert canonical trace using metrics like step overlap or edit distance.
• Inter-Annotator Agreement (IAA): A statistical measure of consistency among human evaluators scoring the same trace, establishing evaluation reliability.

A high-quality explainability trace acts as an audit trail for agents, providing accountability.

The application of automated, scalable methods to assign quantitative scores to reasoning traces, enabling comparison and benchmarking at scale.

• Process Reward Model (PRM): A trained model that assigns a reward to individual steps or the entire sequence based on desired properties like correctness.
• Verifier Model Scoring: A separate model evaluates the correctness of a trace or its conclusion, often used for proof checking.
• Self-Consistency Scoring: Generates multiple reasoning paths for the same problem; the final answer is selected via majority vote, with the score reflecting path agreement.
• Trace Embedding Similarity: Quantifies semantic resemblance between traces by comparing their vector representations.

A reasoning trace is the sequential, step-by-step log of an AI agent's internal cognitive process as it solves a problem. It is the primary object of analysis in CoT evaluation.

• Core Artifact: Serves as the detailed record of intermediate thoughts, deductions, and decisions.
• Evaluation Substrate: Metrics like logical consistency and stepwise coherence are applied directly to the trace.
• Audit Requirement: Essential for debugging, compliance, and understanding model behavior beyond the final output.

A logical consistency check is a verification process that scans a reasoning trace to identify contradictory statements or invalid inferences within the sequence of steps.

• Contradiction Detection: Flags instances where a later step negates or conflicts with an earlier assertion.
• Rule Violation: Identifies steps that break predefined logical or domain-specific rules.
• Foundation for Validity: A trace failing basic consistency checks is fundamentally flawed, regardless of its final answer.

A stepwise coherence score is a quantitative metric that measures the semantic and logical connectedness between consecutive steps in a reasoning trace.

• Transition Quality: Assesses whether each step naturally follows from its predecessor. A jump from "identify the problem" to "therefore the answer is 42" would score poorly.
• Semantic Flow: Often calculated using embedding similarity or trained classifiers to evaluate narrative flow.
• Granular Diagnostic: Pinpoints where reasoning becomes disjointed or leaps unsupported.

Trace validity is a holistic assessment determining if a reasoning trace correctly applies logical rules, adheres to domain constraints, and justifiably leads to its conclusion.

• Holistic Judgment: Goes beyond step-by-step checks to evaluate the entire argument's soundness.
• Constraint Adherence: Ensures the reasoning operates within the problem's boundaries (e.g., physical laws, game rules).
• Justified Conclusion: Verifies that the final answer is a necessary or well-supported outcome of the preceding steps.

A Process Reward Model (PRM) is a machine learning model trained to assign a quality score or reward signal to an entire reasoning trace or its individual steps.

• Learned Evaluation: Trained on human judgments of good/bad reasoning, learning to reward desirable properties like clarity and correctness.
• Reinforcement Learning Tool: Used to provide fine-grained feedback for training AI agents via methods like stepwise reward assignment.
• Automated Scoring: Enables scalable evaluation of reasoning traces without constant human intervention.

Self-consistency scoring is an evaluation method where an AI agent's reasoning is sampled multiple times (generating multiple traces), and the final answer is selected via majority vote.

• Robustness Metric: The agreement rate among different reasoning paths for the same problem serves as a confidence score.
• Consensus over Single Trace: Mitigates the risk of evaluating a single, potentially flawed reasoning sequence.
• Implied Correctness: High self-consistency often correlates with higher answer accuracy, as random errors are unlikely to repeat.