Faithful Synthesis vs. Fabrication Classification Prompt

This prompt is designed for eval pipeline builders and quality assurance engineers who need to distinguish legitimate paraphrasing from hallucinated content in RAG-generated answers. Its core job is to classify each segment of an answer as faithful synthesis (rephrasing grounded in source evidence), unsupported inference (logical leaps beyond the evidence), or fabrication (content with no source basis). Use this prompt as a post-generation guardrail before answers reach users, as a component in automated evaluation pipelines, or to calibrate inter-annotator agreement across human reviewers and LLM judges. It assumes you already have a generated answer and the retrieved context that was provided to the generation model. This is not a generation prompt; it is a classification and verification prompt that operates on existing outputs.

The ideal user has a RAG pipeline in production and needs automated, structured quality signals to gate answers before delivery. You should use this prompt when: (a) you need segment-level granularity rather than a single holistic score, (b) you want a rationale grounded in specific source passages to enable downstream remediation, or (c) you are building an eval harness that requires inter-annotator agreement metrics between human reviewers and LLM judges. The prompt expects two inputs: the full generated answer and the complete retrieved context provided to the generation model. Do not truncate or summarize the context before passing it—this prompt relies on exact source matching to produce accurate classifications.

Do not use this prompt as a substitute for retrieval quality evaluation. It classifies answer fidelity given the provided context, but it cannot detect whether the retrieval step itself returned irrelevant or incomplete documents. For that, pair it with an evidence sufficiency gate or retrieval relevance scorer upstream. Also avoid using this prompt for real-time user-facing latency budgets under 500ms without a fast model; the segment-by-segment analysis is thorough but token-intensive. In regulated environments, treat the output as a screening signal, not a final determination—always maintain a human review path for answers flagged with fabrications or contradictions. Next, copy the prompt template in the following section and adapt the output schema to match your downstream logging or gating system.

This prompt template is designed for eval pipeline builders who need to programmatically distinguish legitimate paraphrasing from hallucinated content in RAG-generated answers. It classifies each segment of a generated answer against the provided source context, producing a structured classification label with a rationale grounded in source evidence. The template uses square-bracket placeholders for all inputs, making it straightforward to integrate into automated evaluation harnesses, CI/CD pipelines, or human review queues.

codeCopy
You are an expert factuality classifier. Your task is to compare each segment of a generated answer against the provided source context and classify it into one of three categories.

## INPUTS

### Source Context
[SOURCE_CONTEXT]

### Generated Answer
[GENERATED_ANSWER]

### Classification Definitions
- **Faithful Synthesis**: The segment accurately paraphrases, summarizes, or restates information explicitly present in the source context. Minor wording changes that preserve meaning are acceptable.
- **Unsupported Inference**: The segment draws a reasonable conclusion or implication from the source context, but the exact claim is not explicitly stated. The inference must be logically derivable from the evidence.
- **Fabrication**: The segment introduces information, claims, facts, or details that are neither stated in nor reasonably inferable from the source context. This includes invented numbers, names, dates, events, or assertions that contradict the source.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "classifications": [
    {
      "segment_text": "The exact text from the generated answer being classified",
      "segment_start_char": 0,
      "segment_end_char": 0,
      "classification": "faithful_synthesis | unsupported_inference | fabrication",
      "rationale": "Explanation grounded in the source context. Cite the specific source passage(s) that support or fail to support this segment.",
      "supporting_source_spans": ["Exact quoted text from the source context that supports the classification, or empty array if none exists"],
      "confidence": "high | medium | low"
    }
  ],
  "summary": {
    "total_segments": 0,
    "faithful_synthesis_count": 0,
    "unsupported_inference_count": 0,
    "fabrication_count": 0,
    "overall_faithfulness_score": 0.0
  }
}

## CONSTRAINTS
- Classify every factual claim in the answer. Do not skip segments.
- If a segment contains multiple claims, split it into separate classification entries.
- For NO_EVIDENCE classifications, the supporting_source_spans array must be empty.
- The overall_faithfulness_score is calculated as: (faithful_synthesis_count + unsupported_inference_count) / total_segments. Fabrications reduce this score.
- If the source context is empty or missing, classify all segments as NO_EVIDENCE with confidence "low".
- Do not use external knowledge. Base all classifications solely on the provided source context.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this prompt for your pipeline, replace the square-bracket placeholders with your actual data. The [SOURCE_CONTEXT] should contain the retrieved passages that were used to generate the answer. The [GENERATED_ANSWER] should contain the full text output from your RAG system. For [OUTPUT_SCHEMA], you can use the provided JSON schema or substitute your own structured format. The [EXAMPLES] placeholder should be populated with 2-3 annotated few-shot examples showing correct classifications for each category, which significantly improves classification accuracy. Set [RISK_LEVEL] to "high" for healthcare, legal, or financial use cases where fabrication is unacceptable, or "medium" for general Q&A systems. For high-risk deployments, always route low-confidence classifications and fabrication detections to a human review queue before taking automated action.

This classification prompt is designed to operate as a post-generation guardrail within an automated evaluation pipeline or a synchronous production filter. Its primary job is to receive a segment of generated text alongside the source evidence and return a structured label—faithful synthesis, unsupported inference, or fabrication—with a grounded rationale. The harness must treat this prompt as a deterministic component: it expects a specific input schema, produces a parseable output, and should be wrapped with validation, retry logic, and logging before any decision is made to block, flag, or release content to a user.

To integrate this prompt, construct a harness function that accepts [ANSWER_SEGMENT] and [SOURCE_EVIDENCE] as inputs. The function should call the model with a low temperature (e.g., 0.0–0.2) to maximize classification stability. Parse the response into a strict schema expecting at minimum classification (an enum of FAITHFUL_SYNTHESIS, UNSUPPORTED_INFERENCE, FABRICATION) and rationale (a string grounded in the provided evidence). Implement a validation layer that rejects any response missing these fields or containing an unrecognized classification value. On validation failure, retry once with the error message appended to the prompt as a correction hint. If the retry also fails, escalate the segment for human review rather than silently passing or blocking. Log every classification result—including the model, prompt version, input hashes, and latency—to enable drift detection and inter-annotator agreement analysis over time.

For high-stakes production use, do not rely on a single classification pass. Run the prompt against the same segment with multiple models or multiple temperature seeds and compare outputs. If classifications diverge, route the segment to a human review queue. This inter-annotator check is essential for catching edge cases where the model is uncertain about the boundary between legitimate paraphrasing and unsupported inference. Additionally, consider batching multiple segments from the same answer into a single classification request to reduce API overhead, but be aware that longer inputs may degrade classification accuracy on individual segments. Always store the source evidence alongside the classification result so that downstream auditors can verify the rationale. The next step after implementing this harness is to run a calibration pass against a golden dataset of human-labeled segments to establish your precision and recall baselines before enabling automated blocking.