Logical Validity Evaluation Rubric Prompt

This prompt is built for AI evaluation leads and platform engineers who need to automate the assessment of deductive reasoning quality in model outputs. The core job-to-be-done is grading a reasoning trace—the step-by-step logical path a model takes—against a structured rubric that measures premise support, inference rule adherence, and formal fallacy detection. Use it when the validity of the reasoning process itself is the primary evaluation target, not the factual accuracy of the final answer or the stylistic quality of the prose. The ideal user is someone wiring this prompt into an evaluation harness that processes reasoning traces from models like o1, Claude, or Gemini, and needs a consistent, machine-readable validity score with specific rule-violation citations to feed into regression tests, release gates, or model selection decisions.

This prompt requires specific inputs to function correctly. You must provide the full reasoning trace as [REASONING_TRACE], the original question or problem statement as [QUESTION], and any provided source context as [SOURCE_CONTEXT] if the reasoning is supposed to be grounded in evidence. The rubric is designed to operate on traces that explicitly show their work—single-step answers or black-box final outputs will not produce meaningful validity scores. The output contract is a JSON object containing a logical_validity_score (1-5 scale), a dimension_scores breakdown for premise support, inference rule adherence, and fallacy detection, and a violations array with specific step-level citations. Do not use this prompt for grading factual correctness, task completion, helpfulness, or safety unless logical structure is the primary evaluation dimension. It is also not suitable for evaluating creative writing, open-ended brainstorming, or tasks where multiple valid reasoning paths exist and formal deductive structure is not expected.

Before deploying this prompt into a production evaluation pipeline, ensure you have a clear definition of what constitutes a valid reasoning trace for your domain. The rubric's scoring anchors assume formal deductive reasoning standards—if your use case tolerates informal heuristics or abductive leaps, calibrate the anchors accordingly. Wire the prompt into a harness that validates the output JSON schema before accepting scores, logs every evaluation with the trace hash and model version, and flags any evaluation where the judge model's own reasoning appears inconsistent with the assigned score. For high-stakes domains where a missed logical error could have significant downstream impact, always pair automated validity scoring with periodic human audit of a stratified sample of traces, especially those near decision boundaries (scores of 2-3). The next step after reading this section is to review the full prompt template and adapt the rubric dimensions to match your specific logical validity standards.

This template provides a complete, self-contained prompt for an LLM judge to evaluate a reasoning trace against formal logical validity criteria. It is designed to be copied directly into your evaluation harness. The prompt instructs the model to act as a logic evaluator, assess the trace against a defined rubric, and return a structured JSON object with scores and specific citations for any rule violations. The square-bracket placeholders must be replaced with your specific inputs before execution.

textCopy
You are an expert logic evaluator. Your task is to assess the logical validity of the provided reasoning trace. You will evaluate the trace against the following rubric dimensions and return a structured JSON report.

### INPUT DATA

**REASONING_TRACE:**
[REASONING_TRACE]

**PREMISES (if provided):**
[PREMISES]

**CONCLUSION (if provided):**
[CONCLUSION]

### EVALUATION RUBRIC

Score each dimension on a scale of 1 (critically flawed) to 5 (perfectly sound). For any score below 4, you must provide a specific citation from the reasoning trace and a brief explanation of the violation.

1.  **Premise Support:** Does each step logically follow from the provided premises or previously established steps? (Score 1-5)
2.  **Inference Rule Adherence:** Are the inference steps valid (e.g., no affirming the consequent, denying the antecedent, or incorrect quantifier shifts)? (Score 1-5)
3.  **Formal Fallacy Detection:** Is the reasoning free of formal logical fallacies (e.g., circular reasoning, false dilemma, non sequitur)? (Score 1-5)
4.  **Conclusion Alignment:** Does the final conclusion necessarily follow from the chain of reasoning, without logical gaps or unsupported leaps? (Score 1-5)

### OUTPUT FORMAT

Return a single JSON object with the following schema. Do not include any text outside the JSON object.

{
  "overall_validity_score": "number (average of dimension scores, rounded to one decimal)",
  "dimension_scores": {
    "premise_support": {
      "score": "number (1-5)",
      "violations": [
        {
          "step_citation": "string (quote the problematic part of the trace)",
          "explanation": "string (explain why it violates premise support)"
        }
      ]
    },
    "inference_rule_adherence": {
      "score": "number (1-5)",
      "violations": [
        {
          "step_citation": "string",
          "explanation": "string"
        }
      ]
    },
    "formal_fallacy_detection": {
      "score": "number (1-5)",
      "violations": [
        {
          "step_citation": "string",
          "fallacy_type": "string (e.g., circular reasoning, false dilemma)",
          "explanation": "string"
        }
      ]
    },
    "conclusion_alignment": {
      "score": "number (1-5)",
      "violations": [
        {
          "step_citation": "string",
          "explanation": "string"
        }
      ]
    }
  },
  "overall_assessment": "string (a concise summary of the reasoning trace's logical soundness)"
}

To adapt this template, replace the [REASONING_TRACE], [PREMISES], and [CONCLUSION] placeholders with your actual data. The [PREMISES] and [CONCLUSION] fields are optional but highly recommended for a thorough evaluation. The rubric dimensions and scoring anchors are calibrated for deductive reasoning quality; you can adjust the dimension descriptions or add new ones, such as 'Causal Validity' or 'Quantitative Correctness,' to fit your specific domain. The critical constraint is the strict JSON output format, which is essential for automated parsing in an evaluation pipeline. Ensure your application code can handle the violations array, which will be empty for scores of 4 or 5.

Before deploying this prompt into a production evaluation suite, run it against a golden dataset of reasoning traces with known logical flaws. Validate that the JSON output is consistently parseable and that the violation citations are specific and actionable. For high-stakes domains like legal or medical reasoning, the output of this automated judge should be treated as a diagnostic aid, not a final verdict, and must be reviewed by a human expert.

This prompt is designed to operate as a model-graded judge inside an automated evaluation pipeline. It should not be used as a one-off chat prompt. The primary integration point is a structured evaluation harness that feeds reasoning traces into the judge, collects structured validity scores and violation citations, and routes results to dashboards, regression gates, or human review queues. The harness is responsible for enforcing the input contract, validating the output schema, and managing retries when the judge produces malformed or incomplete responses.

Input wiring: The harness must assemble the [REASONING_TRACE] and [RUBRIC_DIMENSIONS] placeholders before invoking the model. The reasoning trace should be a complete, multi-step chain-of-thought output from the system under test. The rubric dimensions should be injected as a structured JSON or YAML block defining each dimension, its scoring anchors (e.g., 1–5 scale with behavioral descriptions), and the specific inference rules or fallacy types to check. If the evaluation context includes source documents or ground-truth premises, inject them as [PREMISE_CONTEXT] to enable premise-support scoring. Output validation: Parse the model response against a strict JSON schema that requires validity_score (numeric), dimension_scores (array of objects with dimension, score, rationale), and violations (array of objects with step_index, rule_violated, description, severity). Reject any response missing these fields or containing scores outside the defined scale. Log schema validation failures and retry with the error message appended to the prompt.

Retry and fallback strategy: If the judge returns invalid JSON after two retries, escalate the trace to a human review queue rather than silently accepting a partial score. For high-stakes domains (legal, medical, safety-critical reasoning), always route traces flagged with severity: critical violations to human review regardless of schema validity. Model selection: Use a model with strong instruction-following and structured output capabilities (e.g., GPT-4, Claude 3.5 Sonnet, or equivalent). Avoid smaller models that may struggle with multi-dimensional rubric application and step-level citation. CI/CD integration: In a regression testing pipeline, run this judge against a golden dataset of reasoning traces with known validity labels. Compare judge scores against human-annotated ground truth to measure judge alignment. Set a minimum alignment threshold (e.g., Cohen's kappa > 0.7) before trusting the judge for automated gating. If alignment drops, recalibrate the rubric anchors or retrain human annotators on edge cases.

Observability: Log every evaluation run with the trace ID, judge model version, prompt template version, raw scores, violation counts, and retry attempts. This audit trail is essential for debugging score drift, judge bias, or rubric misinterpretation over time. What to avoid: Do not use this prompt to evaluate final answers alone—it is designed for reasoning trace quality. Do not skip output validation; malformed judge outputs can silently pass incorrect scores into downstream dashboards. Do not treat the judge's scores as objective truth without periodic human calibration checks, especially when the reasoning domain shifts or new fallacy patterns emerge.