Trace Validity

This dimension verifies that no contradictory statements or inferences are made within the reasoning sequence. A valid trace must be free of internal logical conflicts.

• Key Process: Scanning the trace for assertions that directly negate each other (e.g., "The value is X" followed later by "The value is not X" without retraction).
• Formal Methods: Often employs automated theorem provers or satisfiability modulo theories (SMT) solvers to check for contradictions against a formal knowledge base.
• Example: In a financial reasoning task, a trace stating "The client's risk profile is 'Conservative'" and later recommending a "highly speculative investment" without revising the risk assessment would fail this check.

This metric measures the semantic and logical connectedness between consecutive steps in the reasoning trace. It assesses whether each step naturally follows from the previous one.

• Evaluation Method: Uses semantic similarity models (e.g., sentence transformers) to compute the relatedness of consecutive statements. A sharp drop in similarity may indicate a non-sequitur or hallucination.
• Quantitative Output: Produces a score (e.g., 0-1) for the entire trace, often highlighting low-coherence segments for review.
• Contrast with Consistency: While consistency checks for contradictions, coherence evaluates flow. A trace can be consistent but incoherent if steps are randomly ordered.

This process examines the trace to confirm that stated cause-and-effect relationships are logically sound and not merely correlative or assumed.

• Core Question: Does Step B necessarily follow from Step A, given domain knowledge?
• Techniques: Leverages causal graphs or knowledge bases to validate inferred relationships. Challenges the agent's implicit causal assumptions.
• Example: In a diagnostic trace: "The server latency increased (Step A). Therefore, the database index is corrupted (Step B)." Verification would check if increased latency is a definitive symptom of index corruption or if other causes are more likely.

This score measures the degree to which the agent's reasoning and actions adhere to a predefined set of formal rules, safety properties, or operational constraints.

• Foundation: Relies on a machine-readable specification (e.g., temporal logic rules, guardrails) that defines valid operations and states.
• Application: Critical for agents operating in regulated domains (finance, healthcare) or using tools with strict APIs. The trace is checked against the spec for violations.
• Output: Often a binary pass/fail or a score representing the severity of any deviations (e.g., minor parameter misformatting vs. executing a prohibited action).

This validation confirms that the agent correctly integrates and synthesizes information across multiple discrete steps or knowledge sources to arrive at its final conclusion.

• Challenge: Ensures information is not "lost" or misrepresented as it flows through the chain of reasoning.
• Method: Tracks the provenance of key facts through the trace. Validates that intermediate conclusions used in later steps are correctly derived from earlier ones.
• Example: In a legal research trace, the agent must correctly synthesize a ruling from Case A with a statute from Document B to form an intermediate principle, which is then correctly applied to the facts of Case C. Validation checks each synthesis point.

This dimension assesses the justification within the trace for why a specific external tool or API was called, including the appropriateness of the selection and the correctness of its expected outcome.

• Critical Components:
  • Selection Validity: Was the chosen tool the correct one for the subtask? (e.g., using a calculator for arithmetic vs. a search API for factual lookup).
  • Parameter Validity: Were the inputs to the tool correctly derived from the context?
  • Output Interpretation: Did the agent correctly parse and integrate the tool's result into its ongoing reasoning?
• Failure Mode: A trace might call a search API with a poorly formulated query, receive irrelevant data, and then base subsequent reasoning on that noise.

The systematic assessment of the logical coherence, correctness, and completeness of step-by-step reasoning sequences. It moves beyond final-answer checking to validate the intermediate derivations.

• Focus: Verifies that each step follows logically from the previous one.
• Method: Often uses rubric-based scoring or automated verifier models.
• Goal: Ensures the model's 'show your work' is not just plausible but valid.

A verification process that scans a reasoning trace to ensure no contradictory statements or inferences are made. It is a foundational validity test.

• Identifies: Direct contradictions (e.g., 'A is true' followed by 'A is false') and logical fallacies.
• Implementation: Can use rule-based systems or entailment models.
• Critical For: Complex, multi-step reasoning where early errors can invalidate the entire conclusion.

The identification of factually incorrect or unsupported statements within an agent's internal reasoning steps, not just its final output. This is more challenging than output hallucination detection.

• Scope: Checks intermediate claims against a knowledge source or verifiable facts.
• Importance: A single hallucinated 'fact' within a trace can derail subsequent logical steps, leading to a confidently wrong answer.

Assesses the justification provided within a trace for calling an external tool or API. Validity depends on the appropriateness of the selection and the correctness of its expected outcome.

• Evaluates: Was the right tool chosen for the subtask? Was its input correctly formulated? Does the trace show an understanding of what the tool does?
• Prevents: Arbitrary or 'guessed' tool calls that compromise the reliability of an agentic system.

The application of mathematical logic and automated theorem proving techniques to rigorously prove that a reasoning sequence satisfies a given formal specification. This is the highest standard of validity assurance.

• Process: The trace and the problem's constraints are translated into formal logic statements. A prover checks if the conclusion is entailed by the premises.
• Use Case: Critical for high-assurance domains like aerospace, cybersecurity, and algorithmic trading.

A metric that evaluates the effectiveness of an agent's internal mechanisms for detecting its own reasoning errors and initiating reflective steps to revise its approach. It measures meta-cognitive validity.

• High Score Indicates: The agent can identify dead-ends, spot inconsistencies in its own trace, and pivot strategies.
• Low Score Indicates: The agent persists with flawed reasoning despite internal evidence of problems.