Answer Abstention on Conflicting Evidence Prompt

This prompt is designed for high-stakes RAG systems where providing a wrong or misleading answer is more damaging than providing no answer at all. The primary job-to-be-done is to act as a safety gate: when retrieved evidence contains irreconcilable conflicts, the system must abstain from synthesizing a definitive answer and instead surface the disagreement to the user or escalate for human review. The ideal user is an AI engineer or product team building customer-facing Q&A in regulated domains—healthcare, legal, finance, compliance—where factual accuracy and source fidelity are non-negotiable. Required context includes the user's question, a set of retrieved passages, and optionally source metadata such as publication dates, authoritativeness scores, or document types. Without this context, the prompt cannot assess whether evidence conflicts exist.

Do not use this prompt when the system's primary goal is to always provide a fluent answer regardless of evidence quality, or when the cost of abstention is higher than the cost of a potential error. It is also inappropriate for low-stakes applications like general knowledge chatbots, creative writing assistants, or entertainment use cases where users do not expect factual guarantees. The prompt assumes that retrieved evidence has already been filtered for relevance; it does not perform retrieval quality assessment or deduplication. If your pipeline lacks source credibility metadata or conflict detection upstream, pair this prompt with the Multi-Source Conflict Detection Prompt Template or Source Credibility Weighting Prompt before routing to abstention logic.

The abstention decision is binary but nuanced: the prompt must distinguish between resolvable conflicts (where one source is clearly more authoritative or recent) and genuine deadlocks (where equally credible sources contradict each other on material facts). In resolvable cases, the system should provide a weighted answer with explicit caveats. In deadlock cases, it should refuse to answer and produce a structured conflict report. Implementers should define a confidence threshold below which abstention triggers automatically. For regulated workflows, always route abstention decisions through a human review queue with the full evidence set and conflict analysis attached. The next section provides the copy-ready prompt template you can adapt to your domain's risk tolerance and evidence schema.

This template implements a structured abstention decision for high-stakes RAG systems. It forces the model to classify the answerability state before generating any response, then routes to the appropriate output format—full answer, partial answer with caveats, or refusal with escalation. The template uses explicit confidence thresholds, uncertainty language templates, and source conflict severity rules that you can tune for your domain's risk tolerance. Copy this template and replace the square-bracket placeholders with your application's specific inputs, schemas, and constraints.

textCopy
You are an answer-abstention classifier for a [DOMAIN] knowledge system. Your job is to decide whether a question can be answered given the retrieved evidence, and if so, how to respond.

## INPUTS
- Question: [QUESTION]
- Retrieved Evidence: [RETRIEVED_DOCUMENTS]
- Source Metadata: [SOURCE_METADATA]
- Risk Level: [RISK_LEVEL]

## CLASSIFICATION RULES
First, classify the answerability state into exactly one of:
1. **ANSWERABLE**: Evidence is sufficient, consistent, and meets the confidence threshold for this risk level.
2. **PARTIAL_ANSWER**: Evidence supports some claims but has gaps or minor conflicts on non-critical details.
3. **CONFLICT_UNRESOLVED**: Sources directly contradict on material claims and no clear resolution exists.
4. **INSUFFICIENT_EVIDENCE**: Retrieved context lacks the information needed to answer.
5. **OUT_OF_DOMAIN**: Question falls outside the system's knowledge boundaries.

## CONFLICT DETECTION
When comparing sources, flag contradictions if:
- Two or more sources make mutually exclusive factual claims
- Numerical values differ beyond [TOLERANCE_BAND]
- Temporal claims conflict (e.g., effective dates, version numbers)
- Definitions or classifications disagree

## ABSTENTION THRESHOLDS
For risk level [RISK_LEVEL]:
- Require at least [MIN_SOURCE_COUNT] supporting sources
- Require source agreement score >= [AGREEMENT_THRESHOLD]
- If any source has credibility score < [CREDIBILITY_FLOOR], flag for review
- If conflict severity is [CRITICAL_SEVERITY_LEVEL] or higher, abstain

## OUTPUT FORMAT
Return a JSON object with this schema:
{
  "answerability": "ANSWERABLE|PARTIAL_ANSWER|CONFLICT_UNRESOLVED|INSUFFICIENT_EVIDENCE|OUT_OF_DOMAIN",
  "confidence_score": 0.0-1.0,
  "answer": "string or null",
  "caveats": ["string"],
  "conflicts_detected": [
    {
      "claim_a": "string",
      "source_a": "string",
      "claim_b": "string",
      "source_b": "string",
      "conflict_type": "FACTUAL|NUMERICAL|TEMPORAL|DEFINITIONAL",
      "severity": "CRITICAL|MAJOR|MINOR|INFORMATIONAL"
    }
  ],
  "citations": [
    {
      "source_id": "string",
      "quote": "string",
      "supports": "SUPPORTS|REFUTES|NEUTRAL"
    }
  ],
  "escalation_recommended": true|false,
  "escalation_reason": "string or null",
  "uncertainty_statement": "string"
}

## UNCERTAINTY LANGUAGE TEMPLATES
- For PARTIAL_ANSWER: "Based on available evidence, [CONFIDENT_CLAIMS]. However, [GAPS_OR_CONFLICTS]."
- For CONFLICT_UNRESOLVED: "Sources disagree on [DISPUTED_CLAIMS]. Source [A] states [CLAIM_A], while Source [B] states [CLAIM_B]. I cannot resolve this conflict."
- For INSUFFICIENT_EVIDENCE: "The retrieved documents do not contain sufficient information to answer this question. [WHAT_IS_MISSING]."

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

Adapt this template by replacing the domain, risk level, and threshold placeholders with your application's specific values. For healthcare or legal use cases, set [CREDIBILITY_FLOOR] high and [AGREEMENT_THRESHOLD] above 0.8. For internal knowledge bases with lower stakes, you can relax these thresholds. The [CONSTRAINTS] placeholder should include domain-specific rules like regulatory requirements, prohibited claim types, or mandatory disclaimers. The [EXAMPLES] placeholder should contain 2-4 few-shot examples showing correct abstention decisions for your domain's common conflict patterns. After adapting, validate the output schema against your downstream application's expectations and run the eval checks described in the implementation harness section before deploying.

This prompt is not a standalone chat instruction—it is a decision node in a RAG pipeline. Wire it after retrieval and evidence ranking, but before answer generation. The prompt receives a user query, a set of ranked evidence passages with source metadata, and a risk classification for the domain. Its job is to produce a structured abstention decision: answer, partial answer with caveats, or full refusal. The output must be machine-readable so downstream code can route the response without parsing free text. Use a JSON schema constraint in your API call or a structured output mode (e.g., response_format in OpenAI, tool_use in Claude) to enforce the decision object shape.

Integration flow: 1) Retrieve and deduplicate evidence. 2) Run a conflict detection prompt to identify contradictory claims. 3) If conflicts exist, invoke this abstention prompt with the conflicting evidence set, the original query, and a [RISK_LEVEL] parameter (e.g., low, medium, high, critical). 4) Parse the output decision. If decision: "answer", pass the evidence to your synthesis prompt. If decision: "partial", route to a caveat-aware answer generator and append the caveats array to the final response. If decision: "abstain", return the abstention_message directly to the user and optionally create a human review ticket. 5) Log the full prompt, response, and decision for audit. In regulated domains, store the evidence set and abstention rationale as part of the compliance record.

Validation and retries: Validate the output against a strict schema before acting on it. Required fields: decision (enum: answer, partial, abstain), confidence_score (float 0-1), rationale (string), and conditional fields: caveats (array of strings, required if partial), abstention_message (string, required if abstain). If validation fails, retry once with the validation error appended to the prompt as a correction hint. If the retry also fails, escalate to a human reviewer and log the failure. Do not loop more than twice—a malformed abstention decision is itself a signal that the system should stop and hand off. For high-risk domains, add a human approval step before any partial decision is shown to end users. Route partial decisions with confidence below 0.7 to a review queue. Use a model with strong instruction-following and JSON reliability (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models for this task—abstention logic requires calibrated uncertainty reasoning that weaker models handle poorly.