Evidence Gap vs. Conflict Differentiation Prompt

This prompt is for RAG system builders who need their application to distinguish between two distinct failure modes: genuinely contradictory evidence and simply insufficient evidence. In production, these states require different handling—conflicts might trigger a dissenting-views answer or human review, while gaps should trigger a new retrieval strategy or a clean abstention. The ideal user is an AI engineer or product developer integrating a knowledge base Q&A system where the cost of a wrong answer is high, such as in legal research, clinical decision support, or financial compliance. You should use this prompt when your retrieval pipeline returns multiple documents and you need the model to classify the answerability state before generating a final response, not after.

Do not use this prompt when you have only a single source document, when retrieval quality is so poor that every query returns noise, or when your system already has a dedicated conflict resolution step downstream. This prompt is a classification and differentiation step, not a synthesis or resolution step. It works best when placed after retrieval and evidence ranking but before answer generation. Required context includes the user's original question, the full set of retrieved passages with source identifiers, and any domain-specific definitions of what counts as a conflict versus a gap. Without clear operational definitions of 'conflict' and 'gap' for your domain, the model will default to generic interpretations that may not match your product's risk tolerance.

Before deploying this prompt, define your four target states precisely: answerable (sufficient consistent evidence exists), conflicting evidence (multiple sources make contradictory claims on the same specific point), insufficient evidence (retrieved context is on-topic but lacks the specific information needed), and out-of-domain (the question falls outside the knowledge base's scope entirely). Write these definitions into the [CONSTRAINTS] placeholder. Also decide your tolerance for false positives on conflict detection—flagging a gap as a conflict will trigger unnecessary human review, while missing a real conflict may expose users to contradictory information presented as fact. Test with a golden dataset of at least 50 labeled examples covering all four states, including edge cases where sources partially overlap or use different terminology for the same claim.

This prompt template is the core classification instruction for distinguishing between missing information and contradictory information in a RAG pipeline. It forces the model to reason about whether the retrieved context contains enough evidence to answer, whether that evidence disagrees, or whether the question falls outside the system's knowledge domain entirely. The template uses square-bracket placeholders for all dynamic inputs so you can drop it into a prompt assembly pipeline, a test harness, or an eval framework without modification.

textCopy
You are a precise evidence classifier for a retrieval-augmented generation system. Your job is to determine whether a user's question can be answered from the provided context, and if not, to correctly identify why.

## INPUT

**User Question:**
[USER_QUESTION]

**Retrieved Context:**
[RETRIEVED_CONTEXT]

**Domain Scope:**
[DOMAIN_SCOPE]

## CLASSIFICATION RULES

Analyze the retrieved context against the user question and classify into exactly one of the following states:

### ANSWERABLE
- The retrieved context contains sufficient, consistent evidence to answer the question.
- All relevant facts align across sources.
- No contradictory claims exist on material points.

### CONFLICTING_EVIDENCE
- The retrieved context contains evidence that directly contradicts itself on material facts needed to answer the question.
- Sources disagree on claims, numbers, dates, or conclusions that matter for the answer.
- This is NOT about missing information—it is about information that is present but inconsistent.

### INSUFFICIENT_EVIDENCE
- The retrieved context does not contain enough information to answer the question.
- Evidence is missing, partial, vague, or only tangentially related.
- This is NOT about contradictory information—it is about information that is absent or too weak.

### OUT_OF_DOMAIN
- The question falls outside the system's defined domain scope entirely.
- Even with perfect retrieval, this system should not attempt to answer.
- The question may be about a different domain, a disallowed topic, or outside the knowledge boundary.

## OUTPUT FORMAT

Return a JSON object with exactly these fields:

{
  "classification": "ANSWERABLE | CONFLICTING_EVIDENCE | INSUFFICIENT_EVIDENCE | OUT_OF_DOMAIN",
  "confidence": 0.0-1.0,
  "rationale": "Brief explanation citing specific evidence or gaps",
  "conflicting_sources": ["source_id_1", "source_id_2"] | null,
  "missing_information": ["what specific information is missing"] | null,
  "domain_check": "Why this is or is not within domain scope"
}

## CONSTRAINTS

- Do not answer the user's question. Only classify answerability.
- If evidence both supports and contradicts the question, classify as CONFLICTING_EVIDENCE, not ANSWERABLE.
- If evidence is present but too weak to answer confidently, classify as INSUFFICIENT_EVIDENCE.
- Use the domain scope definition strictly. When in doubt about domain boundaries, lean toward OUT_OF_DOMAIN.
- Cite specific source identifiers when flagging conflicts.
- Confidence should reflect how certain you are about the classification, not about the answer itself.

Adaptation guidance: Replace [USER_QUESTION] with the raw user input. Populate [RETRIEVED_CONTEXT] with your retrieval pipeline's output, including source identifiers and passage text. Define [DOMAIN_SCOPE] concretely—for example, 'This system answers questions about US federal tax code only' or 'This system covers internal product documentation for AcmeCorp widgets.' The more specific the domain scope, the more reliable the OUT_OF_DOMAIN classification becomes. If your retrieval system does not return source identifiers, remove the conflicting_sources field or replace it with passage indices. For high-stakes domains, add a [RISK_LEVEL] placeholder that adjusts the confidence threshold for escalating to human review.

Before deploying this prompt, run it through a golden dataset of at least 50 labeled examples covering all four classification states. Pay special attention to boundary cases: weak evidence that could be misclassified as conflicting, domain-adjacent questions that could leak through as INSUFFICIENT_EVIDENCE instead of OUT_OF_DOMAIN, and genuinely conflicting evidence that the model might smooth over and mislabel as ANSWERABLE. Wire the output JSON into your application's routing logic so that CONFLICTING_EVIDENCE and OUT_OF_DOMAIN trigger different downstream behaviors—never collapse them into a single error path.

This prompt is a classification step that sits between retrieval and answer generation. It should be called after the top-k chunks are retrieved but before the final answer is synthesized. The model's job is to produce a structured classification label—answerable, conflicting_evidence, insufficient_evidence, or out_of_domain—along with a brief rationale. The downstream system uses this label to decide whether to proceed with answer generation, trigger additional retrieval, surface a conflict report, or escalate to a human queue. Treat this as a deterministic gate, not an optional annotation. The prompt must return a strict JSON schema so the application can parse the label programmatically without regex or fuzzy matching.

Wire the prompt into a dedicated classification function that accepts [USER_QUESTION] and [RETRIEVED_CHUNKS] as inputs. Validate the output against a JSON schema that requires classification (enum of four values), rationale (string), and confidence (float between 0.0 and 1.0). If the model returns malformed JSON, retry once with a repair prompt that includes the raw output and the schema. If the retry also fails, default to insufficient_evidence and log the failure for review. For high-stakes domains, add a confidence threshold: if confidence is below 0.7, route to a human reviewer regardless of the classification. Log every classification decision with the question, chunk IDs, classification, rationale, and confidence for audit and eval drift detection.

Model choice matters here. Use a model with strong instruction-following and JSON mode support (e.g., GPT-4o, Claude 3.5 Sonnet, or a fine-tuned Llama variant). Avoid smaller models that conflate gaps with conflicts—this is the most common failure mode. In your eval harness, build a test set with equal representation of all four classes, including edge cases where retrieved chunks are partially relevant but insufficient, and where two chunks genuinely contradict each other. Measure precision and recall per class, not just overall accuracy. A false positive on conflicting_evidence (classifying a gap as a conflict) is more damaging than a false negative because it triggers unnecessary escalation workflows. Set your eval thresholds accordingly and monitor classification distributions in production to catch drift early.