RAG Answer Faithfulness Evaluation Rubric Prompt

This prompt is designed for AI product teams who need to automate the quality assessment of their RAG pipeline's outputs at scale. The core job-to-be-done is to replace ad-hoc, manual spot-checking of answers with a structured, repeatable, model-graded evaluation. The ideal user is a quality engineer, an AI product manager, or an MLOps engineer responsible for monitoring a production RAG system. They are not just looking for a binary 'good' or 'bad' label; they need a diagnostic score across multiple dimensions—claim support, citation accuracy, abstention appropriateness, and source conflict handling—that can be trended over time and linked back to specific trace events for root-cause analysis.

To use this prompt effectively, you must already have a production trace that captures the full lifecycle of a single request. This includes the user's query, the list of retrieved context passages with their identifiers, and the final generated answer. The prompt acts as an evaluator, taking this trace as input and applying a detailed rubric to score the answer's faithfulness. The output is a structured JSON object containing a composite score, per-dimension subscores, and trace-anchored justifications for each score. This allows you to log the evaluation result alongside the trace, creating a queryable dataset for monitoring dashboards, alerting on quality regressions, and debugging specific failure modes like a drop in citation accuracy after a retrieval pipeline change.

This prompt is a powerful first-pass automation tool but is not a replacement for human review in high-stakes or safety-critical domains. Do not use it as the sole arbiter of truth for medical, legal, or financial advice generated by your system. It is also not designed for evaluating creative writing, open-ended summarization without a grounding source, or tasks where 'faithfulness' is not the primary success metric. A common failure mode is using this prompt on traces with poorly chunked or irrelevant retrieved context; the rubric will correctly score the answer as unfaithful, but the root cause is a retrieval failure, not a generation failure. Use this evaluation prompt as part of a broader observability suite, pairing it with retrieval quality metrics and human-in-the-loop review for a subset of traces to continuously calibrate the model grader's performance.

This prompt template applies a structured evaluation rubric to a production trace, scoring the faithfulness of a generated answer against the retrieved context. It is designed to be dropped into an automated evaluation pipeline where each trace contains the user query, retrieved passages, and the final generated answer. The rubric dimensions—claim support, citation accuracy, abstention appropriateness, and source conflict handling—force the evaluator to anchor every score in specific trace evidence rather than producing a single opaque number.

textCopy
You are an expert RAG evaluator. Your task is to score the faithfulness of a generated answer against the retrieved context captured in a production trace.

## INPUT
- User Query: [USER_QUERY]
- Retrieved Context (ordered by retrieval rank): [RETRIEVED_CONTEXT]
- Generated Answer: [GENERATED_ANSWER]
- Citations Present in Answer: [CITATIONS]

## EVALUATION RUBRIC
Score the answer on each dimension below. For every score, you must cite the specific retrieved passage span or answer segment that justifies your rating.

### 1. Claim Support (1-5)
- 5: Every factual claim in the answer is directly supported by at least one retrieved passage. No claims are invented or contradicted.
- 4: All major claims are supported; minor details may be imprecise but not false.
- 3: Most claims are supported, but one or two claims lack clear evidence or are partially contradicted.
- 2: Multiple unsupported claims exist, or a central claim is contradicted by the retrieved context.
- 1: The answer is largely fabricated with little to no grounding in the retrieved context.

### 2. Citation Accuracy (1-5)
- 5: Every inline citation points to a real retrieved passage that directly supports the cited claim. No misattributions.
- 4: All citations are to real passages; one citation may be slightly imprecise but still directionally correct.
- 3: One citation points to a passage that does not support the claim, or a supported claim is missing a citation.
- 2: Multiple citation errors, or citations point to irrelevant passages.
- 1: Citations are fabricated, point to non-existent sources, or are systematically misleading.
- N/A: No citations were provided in the answer.

### 3. Abstention Appropriateness (1-5)
- 5: The model correctly answered when context was sufficient and correctly abstained or expressed uncertainty when context was insufficient.
- 4: The model answered appropriately but could have expressed slightly more caution on one edge case.
- 3: The model answered a question that the context could not fully support, or abstained when sufficient context was available.
- 2: The model confidently answered a question with clearly insufficient or contradictory context.
- 1: The model fabricated an answer when it should have abstained, or refused to answer a well-supported question.
- N/A: The trace does not involve an abstention decision.

### 4. Source Conflict Handling (1-5)
- 5: When retrieved passages conflict, the answer acknowledges the disagreement and presents both sides without fabricating a resolution.
- 4: The answer acknowledges conflict but slightly favors one side without justification.
- 3: The answer presents one side of a conflict as fact without acknowledging the other side.
- 2: The answer invents a middle ground or resolution not supported by any source.
- 1: The answer ignores clear conflicts and presents a fabricated consensus.
- N/A: No source conflicts were present in the retrieved context.

## OUTPUT FORMAT
Return a valid JSON object with this exact schema:
{
  "overall_faithfulness_score": <float between 1.0 and 5.0>,
  "dimension_scores": {
    "claim_support": {
      "score": <int 1-5>,
      "justification": "<trace-anchored explanation>",
      "evidence_spans": ["<quoted passage text or answer segment>"]
    },
    "citation_accuracy": {
      "score": <int 1-5 or null if N/A>,
      "justification": "<trace-anchored explanation>",
      "evidence_spans": ["<quoted passage text or answer segment>"]
    },
    "abstention_appropriateness": {
      "score": <int 1-5 or null if N/A>,
      "justification": "<trace-anchored explanation>",
      "evidence_spans": ["<quoted passage text or answer segment>"]
    },
    "source_conflict_handling": {
      "score": <int 1-5 or null if N/A>,
      "justification": "<trace-anchored explanation>",
      "evidence_spans": ["<quoted passage text or answer segment>"]
    }
  },
  "unsupported_claims": [
    {
      "claim": "<the unsupported claim>",
      "closest_retrieved_passage": "<best available passage or null>",
      "severity": "critical|major|minor"
    }
  ],
  "hallucination_flags": [
    {
      "type": "fabrication|contradiction|misattribution|overgeneralization",
      "description": "<what was hallucinated>",
      "trace_reference": "<answer segment or citation ID>"
    }
  ]
}

## CONSTRAINTS
- Do not speculate about what the model might have intended. Score only what is present in the trace.
- If a dimension is not applicable, set its score to null and explain why in the justification.
- The overall_faithfulness_score must be the mean of the applicable dimension scores, rounded to one decimal place.
- Every justification must reference specific text from the retrieved context or generated answer.
- If the retrieved context is empty, score claim_support as 1 and note the retrieval failure.

Adapt this template by adjusting the rubric thresholds to match your product's risk tolerance. For regulated domains, lower the threshold for flagging unsupported claims and require human review when the overall score falls below 4.0. The severity field in unsupported claims should map to your incident response playbook: critical triggers immediate review, major goes to a quality queue, and minor feeds a trend dashboard. Before deploying, run this prompt against a golden set of 50 traces with known faithfulness issues to calibrate whether the model-graded scores correlate with human judgments. If correlation is below 0.8, refine the rubric descriptions or add few-shot examples of borderline cases.

This prompt is designed to run as a batch evaluation step within a CI/CD pipeline or a scheduled monitoring job, not as a real-time user-facing check. The typical integration point is after a RAG system has produced an answer and the full trace—including the user query, retrieved documents, and final output—has been logged. You feed the trace into this prompt, receive a structured rubric score, and write that score to your observability platform (e.g., LangSmith, Braintrust, or a custom metrics store). The prompt expects a JSON object containing [QUERY], [RETRIEVED_CONTEXT] (an array of document objects with id, content, and optional timestamp), and [GENERATED_ANSWER]. The output is a JSON object matching the [OUTPUT_SCHEMA] you define, which should include at minimum a faithfulness_score (0-100), a severity classification, and a justifications array keyed to each rubric dimension.

Validation and retry logic is critical because the prompt's output must be machine-readable for downstream dashboards and alerts. Wrap the model call in a validation layer that parses the JSON response and checks for required fields, correct types, and score ranges. If validation fails, implement a single retry with the error message injected into the prompt as additional context (e.g., 'Your previous output failed JSON schema validation with error: [ERROR]. Please correct and return valid JSON only.'). If the retry also fails, log the raw output and flag the trace for human review. For high-stakes domains (healthcare, legal, finance), always route severity: critical results to a human review queue before the score is accepted. Use a model with strong JSON mode and instruction-following capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate choices. Avoid smaller or older models that may hallucinate the rubric structure or misalign justifications with trace evidence.

Cost and latency management requires batching and caching. Group evaluation requests to process multiple traces in a single batch call when your provider supports it, or use asynchronous workers with concurrency limits to avoid rate limiting. Cache the prompt prefix (the rubric instructions and output schema) to reduce per-request token costs. For production monitoring, sample traces rather than evaluating every response—start with 10% random sampling plus 100% of traces where user feedback signals a problem. Store the rubric scores alongside trace IDs in your observability platform so you can correlate faithfulness regressions with prompt version changes, embedding model updates, or knowledge base modifications. Do not use this prompt as a real-time guardrail that blocks user responses; the latency and cost are too high, and the rubric is designed for offline analysis, not synchronous decision-making.