Multi-Turn Grounding Score Prompt Template

This prompt is for RAG chat evaluation teams who need to measure whether each assistant turn is grounded in provided source material across a full multi-turn session. Single-turn groundedness checks miss the compounding effect of unsupported claims that get repeated, extended, or contradicted in later turns. If your product ships conversational AI backed by retrieval—customer support copilots, research assistants, clinical documentation tools—you need to know not just whether one response was faithful, but whether the entire session stayed anchored to evidence or drifted into fabrication.

Use this prompt when you have access to the full conversation transcript and the source context provided to the model at each turn. The prompt expects turn-indexed source material and produces per-turn grounding scores, extracted unsupported claims with turn indices, and a session-level grounding trend line. It is designed for automated, model-graded evaluation in regression testing pipelines, pre-release QA gates, and ongoing production monitoring. The output schema includes both granular claim-level evidence and aggregate scores, so you can feed results into dashboards, alerting systems, or human review queues.

Do not use this prompt when you lack per-turn source context—if you only have final answers without the retrieval results that informed them, grounding cannot be reliably assessed. Do not use it for single-turn evaluation; simpler groundedness prompts exist for that purpose and will be cheaper and faster. Do not treat the LLM judge's scores as ground truth without calibration against human ratings on a sample of your own data. In regulated domains such as healthcare or finance, always route low-grounding sessions to human review and maintain audit trails of both the scores and the source evidence used to produce them. Start by running this prompt against a golden dataset of conversations where you already know the grounding quality, calibrate the judge's thresholds, and only then promote it to automated gating.

This prompt template is designed to be pasted directly into your LLM judge. It instructs the model to evaluate every assistant turn in a multi-turn conversation for grounding against provided source material. The judge will produce per-turn scores, extract unsupported claims, and generate a session-level trend line. Before using it, ensure you have assembled the full conversation transcript and the corresponding source context for each turn where the assistant referenced external information.

textCopy
You are an expert evaluation judge for Retrieval-Augmented Generation (RAG) chat systems. Your task is to evaluate a multi-turn conversation between a user and an AI assistant. For each assistant turn, you will receive the source context that was provided to the assistant for that turn. Your job is to score how well each assistant response is grounded in its corresponding source material.

## INPUT DATA

### Conversation Transcript
[CONVERSATION_TRANSCRIPT]

### Source Context Per Turn
[SOURCE_CONTEXT_PER_TURN]

## SCORING CRITERIA

For each assistant turn, assign a grounding score from 1 to 5:

- **5 (Fully Grounded):** Every factual claim in the assistant response is directly supported by the provided source context. No information is added beyond the sources.
- **4 (Mostly Grounded):** The vast majority of claims are supported. Minor, non-material details may be inferred but do not contradict sources.
- **3 (Partially Grounded):** Some claims are supported, but the response includes unsupported statements, overgeneralizations, or information not present in the sources.
- **2 (Weakly Grounded):** The response contains significant unsupported claims or misinterprets the source material. Only a small portion is verifiable.
- **1 (Not Grounded):** The response is entirely unsupported, hallucinated, or contradicts the provided source context.

If a turn has no source context provided, score it as "N/A" and note that grounding cannot be assessed.

## OUTPUT FORMAT

Return a valid JSON object with the following structure:

[OUTPUT_SCHEMA]

## CONSTRAINTS

[CONSTRAINTS]

## EXAMPLES

[EXAMPLES]

After pasting this template, replace the square-bracket placeholders with your actual data. The [CONVERSATION_TRANSCRIPT] should contain the full multi-turn dialogue, clearly labeling each user and assistant message with turn indices. The [SOURCE_CONTEXT_PER_TURN] must map each assistant turn index to the specific documents, passages, or chunks retrieved for that turn. The [OUTPUT_SCHEMA] placeholder should be replaced with your exact JSON schema, which must include fields for per_turn_scores (an array of objects with turn_index, score, unsupported_claims, and rationale), session_grounding_trend (a summary of how grounding changes across turns), and overall_session_score. The [CONSTRAINTS] section is where you add any domain-specific rules, such as "if a claim is a direct quote from the source, it is grounded" or "inferred background knowledge is not considered grounded unless explicitly stated." Finally, replace [EXAMPLES] with one or two worked examples showing a turn, its source context, and the expected score with rationale. This calibration is essential for consistent judge behavior.

Before deploying this prompt in a production evaluation pipeline, run it against a golden dataset of 20-50 conversations that have been manually scored by human raters. Measure inter-rater reliability between the LLM judge and your human baseline. Pay particular attention to edge cases: turns where the assistant correctly answers but cites the wrong source, turns where the source context is partially irrelevant, and turns where the assistant abstains due to insufficient context. These cases often reveal scoring inconsistencies that require prompt tuning. If the judge's scores diverge from human ratings by more than one point on average, revisit your [EXAMPLES] and [CONSTRAINTS] to tighten the scoring guidance.

The multi-turn grounding score prompt is designed to run as a post-hoc evaluation pass over a complete conversation log, not as a real-time guardrail during generation. Each turn in the conversation must be paired with its corresponding retrieved source context before the prompt is assembled. The harness is responsible for aligning turn indices, injecting the correct source material per turn, and collecting the structured per-turn scores and unsupported claim extractions that the prompt produces. This is a batch evaluation workflow, not a streaming one.

Input assembly is the most error-prone step. Build a pre-processing function that takes the conversation array and a parallel array of retrieval results per turn. For each assistant turn, inject the source context into the [TURN_SOURCE_CONTEXT] placeholder. If a turn has no retrieved sources, inject an explicit [NO_SOURCES_PROVIDED] marker rather than leaving the field empty—this prevents the judge from hallucinating source presence. The [CONVERSATION_LOG] should include the full turn history up to and including the turn being evaluated, with clear User: and Assistant: prefixes. For session-level trend analysis, run the prompt once per assistant turn (or sample turns for long sessions) and collect results into an array keyed by turn index.

Output parsing must handle the structured JSON the prompt requests. Define a validation schema that checks: turn_index is an integer matching the evaluated turn, grounding_score is a float between 0.0 and 1.0, unsupported_claims is an array of strings (empty if score is 1.0), and claim_source_mapping is an object where each claim string maps to a source citation or null. Reject and retry any response where unsupported_claims contains claims not present in the assistant turn text—this indicates the judge hallucinated a claim. Use a JSON schema validator in your harness, not a loose regex check.

Retry logic should be conservative. If the output fails schema validation, retry once with the same prompt plus the validation error message appended as a [CORRECTION] block. If the second attempt also fails, log the raw output and flag the turn for human review. Do not retry more than twice; a judge that can't produce valid JSON after correction is likely confused by the input and will not self-repair. For high-stakes evaluation pipelines, route all turns with grounding scores below a configurable threshold (e.g., 0.7) to a human review queue before accepting the score.

Session-level aggregation happens after all per-turn scores are collected. Compute the grounding trend line by plotting grounding_score against turn_index. Flag sessions where the trend line shows a downward slope exceeding your threshold—this indicates grounding decay across the conversation, a common failure mode in long RAG chats. Store per-turn scores, unsupported claims, and the trend slope as structured evaluation artifacts. These artifacts feed into dashboards, regression test reports, and model comparison pipelines. Do not rely on a single session-level average score; the trend and the specific unsupported claims are the actionable signals.

Model choice matters for judge reliability. Use a model with strong instruction-following and structured output capabilities. Avoid models known to drift on long-context tasks, since multi-turn evaluation prompts can be token-heavy. If cost is a concern, sample every Nth turn for sessions longer than 20 turns rather than switching to a weaker judge model. Log the model version, prompt template version, and evaluation timestamp alongside every score so that judge drift can be detected over time. When you update the prompt template, re-run it against a golden dataset of scored conversations and compare score distributions before deploying the change.