Conversational RAG Evaluation Rubric Prompt

This prompt is designed for QA engineers and evaluation pipeline builders who need to programmatically assess the quality of a multi-turn Retrieval-Augmented Generation (RAG) assistant. The core job-to-be-done is automated regression testing and production monitoring: you want a structured, repeatable scorecard that tells you if a conversation is breaking down across multiple turns. The ideal user is someone integrating an LLM-as-a-judge into a CI/CD pipeline for prompt changes or a nightly production eval harness. It is not a prompt for grading single-turn factoid Q&A, creative writing, or open-ended chatbot personality. It assumes a complete, structured input: the full conversation transcript, the evidence retrieved at each turn, and the final answers provided by the assistant under test.

To use this effectively, you must provide the prompt with a specific evaluation context. The prompt expects a JSON or structured text block containing the conversation history, a mapping of which evidence chunks were available for each user query, and the assistant's corresponding responses. The output is a structured rubric with five dimensions: turn consistency (does the answer contradict prior turns?), citation accuracy (are claims correctly sourced?), clarification appropriateness (did the assistant ask the right questions when confused?), abstention correctness (did it refuse to answer when evidence was missing?), and evidence faithfulness (is the answer grounded in the provided text?). Each dimension receives a score and a pass/fail flag based on strict criteria, making the output directly consumable by an automated test harness.

Do not use this prompt for single-turn RAG evaluation or for grading open-ended creative generation. It is specifically calibrated for the failure modes of multi-turn systems, such as context drift, stale evidence reuse, and contradictory answers across a session. Before running this at scale, validate the rubric against a golden dataset of 20-30 conversations where you have manually assigned pass/fail grades. If the judge's scores deviate significantly from human consensus, adjust the dimension definitions or add few-shot examples of edge cases. For high-stakes domains like healthcare or legal, always route failing conversations for human review rather than relying solely on the automated score.

This prompt template is designed to be dropped directly into an evaluation harness. It instructs the model to act as a strict evaluator, grading a conversational RAG assistant's final answer against a detailed rubric. The rubric covers four critical dimensions: turn consistency, citation accuracy, clarification appropriateness, and abstention correctness. Each dimension is scored on a defined scale, with clear pass/fail criteria and a requirement for a structured JSON output that your test suite can parse.

textCopy
You are a strict evaluator grading the quality of a conversational RAG assistant's response.

Your task is to evaluate the assistant's final answer based on the provided context, conversation history, and the evaluation rubric below. You must output a single, valid JSON object conforming to the [OUTPUT_SCHEMA].

## EVALUATION INPUTS

**Conversation History:**
[HISTORY]

**Current User Question:**
[QUESTION]

**Retrieved Evidence for Current Turn:**
[EVIDENCE]

**Assistant's Final Answer to Evaluate:**
[ANSWER]

## EVALUATION RUBRIC

Evaluate the answer on the following four dimensions. For each, provide a score and a brief, evidence-based justification.

1.  **Turn Consistency (1-5):** Does the answer contradict any previously established facts or answers from the conversation history? A score of 5 means perfect consistency; 1 means a direct contradiction of a prior, resolved point.
2.  **Citation Accuracy (1-5):** Are the claims in the answer directly supported by the provided [EVIDENCE]? Are citations present and correctly attributed? A score of 5 means every factual claim is perfectly grounded with a correct citation; 1 means a hallucinated claim or fabricated citation.
3.  **Clarification Appropriateness (1-5):** If the question was ambiguous, did the assistant ask a helpful, targeted clarifying question instead of guessing? If the question was clear, this should be scored a 5. A score of 1 means it answered an ambiguous question without clarification, or asked a redundant question.
4.  **Abstention Correctness (Pass/Fail):** If the [EVIDENCE] was insufficient to answer the question, did the assistant correctly abstain or express uncertainty? If the evidence was sufficient, this is a "Pass". A "Fail" means it answered confidently with no evidence, or abstained when a clear answer was present.

**Overall Pass/Fail:** The answer passes if Turn Consistency >= 4, Citation Accuracy >= 4, Clarification Appropriateness >= 4, and Abstention Correctness is "Pass". Otherwise, it fails.

[OUTPUT_SCHEMA]
{
  "turn_consistency": { "score": "int (1-5)", "justification": "string" },
  "citation_accuracy": { "score": "int (1-5)", "justification": "string" },
  "clarification_appropriateness": { "score": "int (1-5)", "justification": "string" },
  "abstention_correctness": { "result": "string (Pass|Fail)", "justification": "string" },
  "overall_pass": "boolean"
}

To adapt this template, replace the square-bracket placeholders with data from your test harness. The [HISTORY] should be a serialized representation of the prior conversation turns. The [EVIDENCE] field must contain the raw text chunks retrieved for the current [QUESTION]. The [ANSWER] is the final string output from your RAG system. The [OUTPUT_SCHEMA] is provided inline as a JSON schema for clarity, but you can move it to your system prompt or a structured output API parameter. For high-stakes domains, ensure a human reviews a sample of failing evaluations to calibrate the rubric and prevent subtle errors from being automated away.

This prompt is designed to be the scoring engine inside an automated evaluation harness, not a one-off manual review tool. The harness should feed the prompt a structured conversation transcript, the retrieved evidence for each turn, and the system's generated answers. The prompt returns a JSON evaluation payload that downstream systems can parse, log, and act on. The primary integration point is a post-generation evaluation step: after your RAG system produces a multi-turn answer, the harness calls this rubric prompt before the answer reaches the user or is logged for offline analysis. For high-stakes domains, insert a human review gate when the overall_score falls below a defined threshold or when any critical dimension (e.g., citation_accuracy, abstention_correctness) is flagged as fail.

Wire the prompt into your application by constructing a test case object with the required fields: [CONVERSATION_HISTORY] as a list of role-content-turn triples, [RETRIEVED_EVIDENCE] as a mapping of turn IDs to ranked passage lists, and [SYSTEM_ANSWERS] as the assistant's responses per turn. The prompt template expects these inputs to be serialized into the [INPUT] placeholder. On the output side, parse the JSON response and validate it against a strict schema: confirm that overall_score is a float between 0.0 and 1.0, that each dimension object contains a score, pass_fail enum (pass, fail, not_applicable), and a non-empty rationale string. If validation fails, retry the prompt once with the validation error message appended to the [CONSTRAINTS] field. If the retry also fails, log the raw output and escalate for manual review rather than silently accepting a malformed evaluation.

Model choice matters for rubric consistency. Use a model with strong instruction-following and JSON output capabilities, such as gpt-4o or claude-3-5-sonnet, and set temperature=0 to maximize scoring determinism. For cost-sensitive pipelines running high-volume evaluations, consider routing simpler pass/fail checks to a smaller model like gpt-4o-mini and reserving the full rubric prompt for borderline cases. Log every evaluation result with the prompt version hash, model identifier, and input transcript ID to maintain an audit trail. When deploying changes to the rubric prompt itself, run a regression suite against a golden dataset of scored conversations to detect drift in scoring behavior before the new version reaches production. The next step is to build a dashboard that surfaces overall_score distributions, per-dimension failure rates, and conversation-length trends so the team can spot systemic RAG quality regressions early.