Trace Annotation Schema

The core of a schema is a controlled vocabulary of labels that categorize discrete properties of a reasoning step or the entire trace. These form a taxonomy that ensures consistent classification.

• Logical Operation Labels: e.g., deduction, induction, abduction, contradiction.
• Step Function Labels: e.g., information_retrieval, tool_call:calculator, constraint_check, hypothesis_generation.
• Quality Labels: e.g., correct_inference, logical_fallacy, unsupported_assumption, hallucination.
• Meta-Cognitive Labels: e.g., confidence_expression, error_detection, strategy_shift.

Schemas define quantitative scoring systems to measure trace quality. Each rubric specifies the criteria for each score level to minimize evaluator subjectivity.

• Stepwise Coherence Score: A 1-5 scale rating the logical/semantic flow between consecutive steps.
• Trace Validity Score: A binary or scaled assessment of whether the trace's conclusion is justified by its steps.
• Efficiency Metric: Measures path length or computational steps against an optimal baseline.
• Self-Consistency Score: The agreement rate (e.g., 0.8) between multiple sampled reasoning paths for the same problem.

For deterministic evaluation, schemas can include machine-readable rules that a trace must satisfy. This enables automated checks alongside human annotation.

• Logical Consistency Constraints: Formal rules (e.g., IF step A asserts X, THEN no subsequent step can assert not-X).
• Tool-Use Preconditions/Effects: Specifications that a tool call's rationale must match its documented API contract.
• Domain-Specific Invariants: Rules that must hold throughout the trace (e.g., energy_conservation in a physics problem).
• Temporal Ordering Rules: Constraints on the required sequence of certain reasoning operations.

A schema provides reference materials to anchor evaluations. Gold standard traces are canonical, expert-verified reasoning sequences used for comparison.

• Canonical Solution Traces: Human-expert traces that demonstrate optimal reasoning for benchmark problems.
• Alignment Metrics: Formulas for comparing a generated trace to the gold standard (e.g., ROUGE-L for steps, graph edit distance for non-linear traces, BLEU for rationale similarity).
• Adversarial Example Traces: Pre-annotated traces containing specific failure modes (e.g., subtle logical fallacies) to test evaluator accuracy and model robustness.

This component provides the operational manual for human evaluators, ensuring high Inter-Annotator Agreement (IAA). It turns abstract labels into actionable instructions.

• Label Definitions & Decision Trees: Unambiguous definitions and flowcharts for applying each label.
• Example Traces: Fully annotated positive and negative examples for each scoring category.
• Edge Case Handling: Guidance for ambiguous or novel reasoning patterns.
• IAA Calculation Method: The specific statistical measure (e.g., Cohen's Kappa, Fleiss' Kappa) used to measure and report annotator consistency.

Defines the structured data recorded about the evaluation process itself, crucial for auditability and schema evolution.

• Evaluator ID & Qualifications: Tracks which annotator performed which assessment.
• Annotation Timestamp & Duration: Metrics for measuring evaluation effort and potential fatigue.
• Schema Version: The specific version of the annotation guide used.
• Model & Prompt Context: Identifiers for the AI model and exact prompt that generated the trace being evaluated.
• Disagreement Resolution Log: Records of how conflicts between annotators were adjudicated.