Conversation Contradiction Detection Prompt Template

This prompt is a batch audit tool for QA engineers and evaluation pipeline owners who need to detect self-contradiction in multi-turn chat logs. Its primary job is to produce a structured contradiction map that links conflicting statement pairs, identifies the turn indices where each statement occurred, labels the type of contradiction (e.g., factual, logical, policy), and assigns a severity score. Use it when single-turn evaluation passes each response in isolation but misses cross-turn inconsistencies that degrade user trust—such as an assistant giving contradictory instructions in turns 2 and 7, or reversing a stated policy without acknowledgment. The ideal user is someone responsible for offline quality gates, regression testing, or pre-release evaluation of conversational AI systems.

You should not use this prompt as a real-time guardrail in a production chat pipeline. It is designed for thorough, comparative analysis across a full session log, which makes it too slow and token-intensive for synchronous interception. It is also not a replacement for a general factual accuracy check; it specifically targets internal consistency within the assistant's own responses, not alignment with external ground truth. The prompt requires the full conversation transcript as input, along with a defined output schema and contradiction taxonomy. The output is a JSON object containing a list of contradiction pairs, each with turn_a, turn_b, statement_a, statement_b, contradiction_type, severity, and a brief explanation. This structured output is designed to be consumed by an automated evaluation pipeline, not read as a standalone narrative.

Before using this prompt, ensure you have a clear definition of what constitutes a contradiction in your product context. A legitimate topic shift or a clarification that refines a previous statement is not a contradiction. The prompt includes instructions to distinguish these cases, but you should calibrate it with your own labeled examples to control the false-positive rate. After running the audit, always review a sample of flagged contradictions to confirm they represent genuine user-facing problems. The next step is to integrate this prompt into your offline evaluation harness, where it can be run against golden datasets and used to gate releases when contradiction scores exceed a defined threshold.

This prompt template is designed to be dropped directly into your evaluation pipeline. It expects a full conversation transcript and a configuration block that defines your contradiction taxonomy and severity scale. Replace every square-bracket placeholder with your actual data before running the prompt. The template is self-contained: it includes the role definition, output schema, scoring rules, and an explicit request for structured JSON output that your application can parse and log.

textCopy
You are a conversation quality auditor. Your task is to analyze the provided conversation transcript and identify every pair of statements made by the assistant that contradict each other across different turns.

[CONVERSATION_TRANSCRIPT]

[CONTRADICTION_TAXONOMY]

[SEVERITY_SCALE]

For each contradiction found, produce a JSON object with the following fields:
- contradiction_id: a unique string identifier for this contradiction pair (e.g., "C001")
- statement_a: the exact text of the first contradictory statement
- statement_a_turn: the turn index where statement_a appears
- statement_b: the exact text of the second contradictory statement
- statement_b_turn: the turn index where statement_b appears
- contradiction_type: one of the types defined in [CONTRADICTION_TAXONOMY]
- severity: one of the levels defined in [SEVERITY_SCALE]
- explanation: a one-sentence explanation of why these statements conflict

Return a single JSON object with a "contradictions" array containing all detected contradictions. If no contradictions are found, return an empty array.

[OUTPUT_SCHEMA]

Do not flag legitimate topic shifts, user corrections that the assistant acknowledges, or clarifications that resolve apparent conflicts. Only flag unresolved factual or logical contradictions.

To adapt this template, start by defining your [CONTRADICTION_TAXONOMY]. Common types include factual_inconsistency (two statements assert different facts), policy_reversal (the assistant changes a rule or constraint mid-conversation), logical_contradiction (one statement negates the logical consequence of another), and identity_mismatch (the assistant claims different capabilities or attributes). Your [SEVERITY_SCALE] should map to your team's risk tolerance—a typical scale is critical (the contradiction would cause user harm or system failure), major (the contradiction undermines trust or task completion), minor (the contradiction is noticeable but low-impact), and cosmetic (stylistic or phrasing inconsistency with no factual weight). The [OUTPUT_SCHEMA] placeholder should contain the full JSON Schema you expect, including required fields, types, and enum constraints. Wire this schema into your post-processing validator so malformed outputs are caught before they reach your dashboard. If you are evaluating conversations in a regulated domain, add a [RISK_LEVEL] field and route critical contradictions for human review before acting on the results.

The contradiction detection prompt is designed to operate as a batch evaluation step within a QA pipeline, not as a real-time chat interceptor. The typical integration pattern involves pulling completed conversation logs from your data warehouse or logging system, formatting each session into the [CONVERSATION_LOG] placeholder, and submitting batches through an async evaluation job. Because the prompt produces structured JSON with turn indices, contradiction types, and severity scores, the output is immediately ingestible by downstream dashboards, alerting systems, or regression test suites. The primary integration surface is the contradiction map schema—every consuming system should validate against this schema before trusting the results.

Validation and retry logic must be layered around this prompt because LLM outputs can drift in structure or miss required fields. Implement a JSON schema validator that checks for the required top-level keys: contradictions (array), each containing turn_pair, statement_a, statement_b, contradiction_type, severity, and explanation. If validation fails, retry up to two times with the same input and an explicit format correction instruction appended to the prompt. If the third attempt still fails validation, log the raw output and flag the session for human review rather than silently dropping it. For severity scores, enforce that values fall within the expected range (1-5) and that contradiction_type matches the allowed enum values defined in your rubric. False positive mitigation is critical: implement a secondary lightweight check that compares flagged contradiction pairs against a list of known legitimate topic shifts or clarifications. If a flagged pair matches a known safe pattern, downgrade the severity or suppress the alert. This two-stage approach—structural validation followed by semantic false-positive filtering—catches most production failures before they reach a human reviewer.

Model choice and cost considerations matter here because contradiction detection across long conversations requires strong reasoning and large context windows. Use a model with at least 128K context and strong instruction-following capabilities. For high-volume QA pipelines processing thousands of sessions daily, consider routing low-risk sessions (short conversations, low-severity historical patterns) through a faster, cheaper model and reserving the full-capability model for sessions exceeding a turn-count threshold or flagged by upstream risk classifiers. Logging and observability should capture the prompt version, model identifier, input conversation ID, output contradiction count, validation status, retry count, and latency per session. This telemetry lets you track prompt drift over time and detect when model behavior changes introduce new false positive patterns. Human review integration is essential for high-severity contradictions (severity >= 4) and for any session where the contradiction map contains more than three flagged pairs. Route these to a review queue where QA engineers can confirm or reject each flagged contradiction, providing ground-truth labels that feed back into your evaluation calibration process.

What to avoid: Do not deploy this prompt as a real-time guardrail that blocks assistant responses mid-conversation. The latency and cost profile is wrong for synchronous use, and the false positive rate—while manageable in batch QA—would create unacceptable user friction if used to interrupt live chats. Do not skip the false-positive filtering step; legitimate topic shifts, user corrections that the assistant acknowledges, and clarifications of earlier ambiguous statements will otherwise generate noise that erodes trust in the pipeline. Finally, do not treat the contradiction map as the final word without human calibration. Run a monthly alignment check where you sample 50-100 flagged sessions, have human reviewers label them independently, and compare against the prompt's output to detect drift in contradiction sensitivity or type classification accuracy.