Contradiction Detection Against Known-Correct Answer Prompt Template

This prompt is designed for safety-critical evaluation pipelines where a model-generated output must be audited against a known-correct reference answer. Unlike general factual consistency checks, this prompt focuses specifically on direct contradictions: statements in the candidate output that assert the opposite of what the reference states. In domains like healthcare, legal review, and compliance, a contradiction is more dangerous than an omission. An incomplete answer can be escalated; a confidently wrong answer can cause harm. Use this prompt when your evaluation system needs to produce a structured contradiction report with severity levels, evidence spans, and actionable flags for human review.

The ideal user is an evaluation engineer or AI safety lead building automated regression gates, content verification pipelines, or model-output auditing systems. You need a known-correct reference answer—this prompt is not for open-ended generation where no ground truth exists. The reference must be authoritative and span the claims the candidate output attempts to make. Required context includes: the candidate output to evaluate, the reference answer, a severity taxonomy (e.g., critical, major, minor), and an output schema for the contradiction report. The prompt works best when contradictions are semantically clear rather than matters of interpretation; ambiguous cases should be flagged for human review rather than auto-classified.

Do not use this prompt when the reference answer is incomplete, when the candidate output covers topics outside the reference scope, or when you need to measure omissions rather than contradictions. This prompt will not detect missing information—it only flags statements that directly conflict with the reference. For coverage gaps, use a separate completeness evaluation prompt. Do not use this prompt as a standalone safety gate in regulated domains; always pair it with human review for critical-severity contradictions and maintain an audit trail of contradiction reports over time. If your evaluation needs include hallucination detection without a reference, use a groundedness evaluation prompt instead.

Before deploying, calibrate the severity taxonomy against your domain's risk tolerance. A contradiction about dosage in a clinical summary is critical; a contradiction about formatting preferences is minor. Run this prompt against a labeled dataset of known contradictions and non-contradictions to measure precision and recall before wiring it into a production pipeline. Expect false positives on statements that appear contradictory but are actually paraphrases or context-dependent clarifications—build a human review queue for medium-confidence flags.

This prompt template is designed to detect contradictions between a model-generated output and a known-correct reference answer. It is built for safety-critical domains—such as healthcare, legal review, finance, and compliance—where a confident contradiction is more dangerous than an omission. The prompt instructs the model to perform a claim-by-claim comparison, flag only direct conflicts (not differences in style or omission), assign severity levels, and provide evidence spans from both the output and the reference. Copy the template below into your evaluation harness, replace each square-bracket placeholder with production data, and run it as part of your regression or pre-release gate suite.

textCopy
You are a contradiction detection evaluator. Your task is to compare a model-generated [OUTPUT] against a known-correct [REFERENCE_ANSWER] and identify statements in the output that directly conflict with the reference.

## INPUT
[OUTPUT]:
[OUTPUT_TEXT]

[REFERENCE_ANSWER]:
[REFERENCE_TEXT]

## CONSTRAINTS
[CONSTRAINTS]
- Only flag direct factual contradictions. Do not flag differences in wording, style, level of detail, or information present in the reference but absent from the output (omissions are not contradictions).
- For each contradiction, provide an evidence span from both the output and the reference that demonstrates the conflict.
- Assign a severity level to each contradiction: CRITICAL (safety, legal, or compliance impact), HIGH (material factual error), MEDIUM (misleading but not dangerous), LOW (minor inaccuracy).
- If no contradictions are found, return an empty contradictions array and set has_contradictions to false.
- Do not evaluate the quality of the output beyond contradiction detection.

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
{
  "has_contradictions": boolean,
  "contradictions": [
    {
      "severity": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
      "output_claim": "The contradictory statement from the output.",
      "output_span": "Exact text from [OUTPUT] that contains the contradiction.",
      "reference_claim": "The correct statement from the reference.",
      "reference_span": "Exact text from [REFERENCE_ANSWER] that contradicts the output.",
      "explanation": "Why these statements conflict."
    }
  ],
  "summary": "Brief summary of findings."
}

## EXAMPLES
[EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

## INSTRUCTIONS
1. Parse the [OUTPUT] and [REFERENCE_ANSWER] into individual claims.
2. For each claim in the output, check if it directly contradicts any claim in the reference.
3. For each contradiction found, extract the exact evidence spans and assign a severity level.
4. Return the results in the specified JSON schema.

Adapting the template: Replace [OUTPUT_TEXT] with the model-generated text under evaluation and [REFERENCE_TEXT] with the known-correct answer. Use [CONSTRAINTS] to add domain-specific rules—for example, in healthcare you might add 'Flag contradictions involving medication dosage, allergy status, or critical lab values as CRITICAL.' Use [EXAMPLES] to provide few-shot demonstrations of contradictions at each severity level, which significantly improves calibration. Set [RISK_LEVEL] to HIGH for regulated domains to trigger downstream human review of all CRITICAL and HIGH findings. For production pipelines, validate the output JSON against the schema before accepting results, and log any parse failures for retry or escalation.

What to do next: After copying the template, wire it into your evaluation harness with schema validation and retry logic. Run it against a golden dataset of known contradiction and non-contradiction pairs to calibrate severity thresholds. Monitor for false positives—the model may flag stylistic differences as contradictions if the instructions are too loose. If you observe over-flagging, tighten the constraints to emphasize that only factual conflicts count. For high-stakes domains, always route CRITICAL contradictions to human review before taking action on the findings.

The contradiction detection prompt is not a standalone tool—it's a component that must be wired into a broader evaluation harness. The prompt expects a [MODEL_OUTPUT] and a [REFERENCE_ANSWER] and returns a structured contradiction report. In production, these inputs typically come from an upstream generation step and a golden dataset or knowledge base. The harness is responsible for supplying both inputs, parsing the JSON response, validating the schema, and routing the result to the appropriate downstream action: pass, retry, or escalate for human review.

Schema validation is the first gate. The prompt instructs the model to return JSON with a specific structure: a contradictions array containing objects with severity, claim_from_output, conflicting_reference, evidence_spans, and explanation fields. Before trusting the output, validate that the response parses as valid JSON and that all required fields are present. Use a lightweight validator—Pydantic in Python, Zod in TypeScript, or a JSON Schema assertion in your test runner—to catch malformed responses. If validation fails, retry once with an explicit error message appended to the prompt: Your previous response was not valid JSON or was missing required fields. Return ONLY the contradiction report in the exact schema specified. If the second attempt also fails, log the failure and escalate. Do not silently accept malformed contradiction reports in safety-critical domains.

Severity routing determines what happens next. The prompt defines three severity levels: direct_contradiction, partial_contradiction, and potential_conflict. Your harness should treat these differently. direct_contradiction findings should block deployment or trigger immediate review—the model output directly conflicts with a known-correct answer on a material fact. partial_contradiction findings may be acceptable depending on your tolerance; consider threshold-based gating where more than N partial contradictions triggers review. potential_conflict findings are informational and can be logged for trend analysis without blocking the pipeline. Implement this routing in application code, not in the prompt, so the logic is auditable and adjustable without changing the model's instructions.

Human review integration is mandatory for high-risk domains. If you're using this prompt in healthcare, legal, finance, or safety contexts, every direct_contradiction finding must be reviewed by a qualified human before any action is taken. The harness should queue flagged outputs with the full contradiction report, the original model output, and the reference answer. Present evidence spans side-by-side so reviewers can quickly verify whether the contradiction is real or a false positive. Track review outcomes to measure the judge's precision and calibrate severity thresholds over time.

Model choice matters for judge reliability. This prompt requires careful reading and precise span extraction. Use a model with strong instruction-following and JSON output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 2.0 Flash are reasonable defaults as of early 2025. Avoid using smaller or older models as judges for contradiction detection unless you've calibrated them against human judgments and confirmed acceptable false-positive and false-negative rates. If cost is a concern, consider using a stronger model for direct_contradiction detection and a lighter model for potential_conflict screening, with the stronger model as the final arbiter on escalated cases.

Log everything for observability and calibration. For every contradiction check, log: the prompt version, model version, input lengths, raw response, validation status, contradiction counts by severity, and the routing decision. This data is essential for detecting judge drift, calibrating severity thresholds, and debugging false positives. If you're running this prompt at scale, aggregate contradiction rates by source, topic, or model to identify systemic issues. A sudden spike in direct_contradiction findings may indicate a problem with the upstream generation model, not the judge.

What to avoid: Do not use this prompt as a real-time guardrail in user-facing chat without human review for direct_contradiction findings. Do not treat the contradiction report as ground truth—it is a model's judgment and can be wrong. Do not skip schema validation; malformed JSON from the judge is a common failure mode. Do not use this prompt to compare two model outputs without a known-correct reference; it is designed for reference-guided evaluation, not pairwise comparison. For pairwise comparison, use a dedicated preference prompt from the Pairwise Comparison and Preference Prompts content group.