Rubric for Conversation Coherence Scoring Prompt Template

This prompt is for chat product teams and evaluation engineers who need to automate quality scoring of multi-turn conversations. It produces a structured rubric that an LLM judge can apply to score context retention, contradiction avoidance, topic progression, and turn-taking appropriateness. Use this when single-turn evaluation misses failures like context loss across messages, factual contradictions between turns, or incoherent topic shifts. This prompt is designed for model-graded evaluation pipelines where human spot checks are too slow or expensive to run on every conversation.

The rubric is particularly valuable when you are shipping a chat product where users expect the assistant to remember what was said earlier, stay consistent with its own statements, and navigate topic changes gracefully. Without a multi-turn coherence rubric, your evaluation pipeline will silently pass conversations where the assistant contradicts itself three turns apart or loses track of the user's original request after a digression. The prompt template includes explicit handling for context window boundaries—a common failure mode where older messages fall out of the context window and the model's behavior degrades without the evaluator detecting the root cause.

Do not use this prompt for single-turn evaluation tasks, for grading factual accuracy against external ground truth, or for safety policy adherence scoring. Those require separate, specialized rubrics. This prompt also assumes you already have a working LLM judge pipeline; it does not cover judge deployment, inter-rater reliability measurement, or score calibration against human ratings. If you need those capabilities, pair this rubric with the score calibration and judge alignment playbooks in this pillar. Before deploying, validate the rubric against at least 20 human-scored conversations to confirm that the automated scores correlate with human judgments and that the context-window boundary handling works as expected.

This prompt template is designed to be dropped directly into your evaluation harness. It instructs an LLM judge to assess a multi-turn conversation transcript for coherence, focusing on context retention, contradiction avoidance, topic progression, and turn-taking appropriateness. The template uses square-bracket placeholders for all variable components—replace each one with your specific configuration before sending it to the judge model. The output is structured as a JSON object with dimension scores, a composite score, and a required justification, making it suitable for automated evaluation pipelines.

textCopy
You are an expert conversation quality evaluator. Your task is to score the coherence of a multi-turn dialogue between a user and an AI assistant.

## Evaluation Context
[CONTEXT_DESCRIPTION]

## Rubric Dimensions
Score each dimension on a scale of [SCORE_RANGE]. Use the following definitions:

### 1. Context Retention (CR)
Does the assistant remember and correctly reference information from earlier turns? Penalize forgotten details, repeated questions, or contradictory statements about previously established facts.
- [LEVEL_1_ANCHOR]
- [LEVEL_2_ANCHOR]
- [LEVEL_3_ANCHOR]
- [LEVEL_4_ANCHOR]
- [LEVEL_5_ANCHOR]

### 2. Contradiction Avoidance (CA)
Does the assistant avoid contradicting its own prior statements? Flag direct contradictions, inconsistent advice, or factual reversals across turns.
- [LEVEL_1_ANCHOR]
- [LEVEL_2_ANCHOR]
- [LEVEL_3_ANCHOR]
- [LEVEL_4_ANCHOR]
- [LEVEL_5_ANCHOR]

### 3. Topic Progression (TP)
Does the conversation flow logically from one turn to the next? Assess whether topic shifts are smooth, abrupt, or disjointed. Penalize non-sequiturs and ignored user topic initiations.
- [LEVEL_1_ANCHOR]
- [LEVEL_2_ANCHOR]
- [LEVEL_3_ANCHOR]
- [LEVEL_4_ANCHOR]
- [LEVEL_5_ANCHOR]

### 4. Turn-Taking Appropriateness (TA)
Does the assistant respond appropriately to the user's last message? Assess whether responses acknowledge user input, answer direct questions, and maintain conversational rhythm.
- [LEVEL_1_ANCHOR]
- [LEVEL_2_ANCHOR]
- [LEVEL_3_ANCHOR]
- [LEVEL_4_ANCHOR]
- [LEVEL_5_ANCHOR]

## Context Window Boundary Handling
[CONTEXT_WINDOW_INSTRUCTIONS]

## Conversation Transcript
[CONVERSATION_TRANSCRIPT]

## Output Schema
Return ONLY a valid JSON object with this exact structure:
{
  "scores": {
    "context_retention": <integer>,
    "contradiction_avoidance": <integer>,
    "topic_progression": <integer>,
    "turn_taking": <integer>
  },
  "composite_score": <float>,
  "justification": "<string explaining key observations, edge cases, and reasoning for each score>",
  "uncertainty_flags": ["<list of specific concerns or borderline calls>"]
}

## Constraints
[ADDITIONAL_CONSTRAINTS]

To adapt this template, start by defining your scoring scale in [SCORE_RANGE]—common choices are 1–5 or 1–10. Populate each dimension's [LEVEL_N_ANCHOR] placeholders with concrete behavioral descriptions that distinguish each score point. For example, a Level 1 anchor for Context Retention might read: 'Assistant repeatedly asks for information already provided or contradicts established facts without acknowledgment.' The [CONTEXT_WINDOW_INSTRUCTIONS] placeholder is critical for production use: specify how the judge should handle conversations that exceed the model's context window, such as instructing it to note when earlier context appears truncated and to avoid penalizing the assistant for missing information that may have scrolled out of view. The [CONVERSATION_TRANSCRIPT] should be formatted with clear turn markers (e.g., 'User:', 'Assistant:') and include timestamps if temporal coherence matters for your use case. After populating all placeholders, run the prompt against a golden dataset of conversations with known coherence issues to calibrate the judge's scoring distribution against human ratings before deploying it in an automated pipeline.

The conversation coherence rubric prompt is designed to operate as a model-graded evaluation step within a larger evaluation harness, not as a standalone chat interaction. In production, you will typically call this prompt after a multi-turn dialogue has completed, feeding the full transcript (or a sliding window of recent turns) into the [CONVERSATION_TRANSCRIPT] placeholder. The prompt returns a structured JSON scorecard that your harness can parse, log, and alert on. Treat this prompt as a deterministic measurement instrument: it should produce consistent scores for the same input, so you must control temperature (set to 0 or near-zero), pin the model version, and validate the output schema on every call.

Integration pattern: Wrap the prompt call in a function that (1) assembles the transcript with turn-level speaker labels and timestamps, (2) injects the rubric definition and scoring scale from your configuration store, (3) calls the LLM with response_format set to the expected JSON schema, and (4) validates the returned JSON before accepting the score. If the model returns malformed JSON, a missing required field, or a score outside the defined scale, retry once with an explicit repair instruction appended to the prompt (e.g., 'Your previous response was invalid. Return ONLY valid JSON matching the schema.'). If the retry also fails, log the failure, flag the evaluation as UNSCORABLE, and route it for human review. For high-stakes chat products, implement a calibration check: periodically sample scored transcripts and have a human evaluator independently score them using the same rubric, then compute Cohen's kappa or Pearson correlation to detect judge drift.

Context window handling: For long conversations that exceed the model's context window, implement a sliding window strategy. Score the most recent N turns (where N is calibrated to your model's token limit minus rubric and instruction overhead), and include a [PRIOR_CONTEXT_SUMMARY] placeholder populated by a separate summarization step for earlier turns. The rubric's context retention dimension should then evaluate coherence against both the recent turns and the provided summary. Monitoring: Log every score alongside the transcript hash, model version, timestamp, and any validation errors. Set alerts on score drops below your defined threshold, sudden increases in UNSCORABLE rates, and schema validation failure rates exceeding 2%. Store raw scores in your evaluation database for trend analysis and prompt version comparison.

What to avoid: Do not use this prompt for real-time scoring during live user conversations—latency and cost make it unsuitable for synchronous guardrails. Do not skip schema validation; malformed judge outputs silently corrupt your evaluation metrics. Do not treat the coherence score as a single source of truth without periodic human calibration checks. Finally, do not reuse the same model instance for both generating the conversation and scoring it, as self-evaluation bias inflates coherence scores. Always use a separate judge model or a separate API call with no shared context.