Multi-Turn Conversation Scorecard Generation Prompt

This prompt is designed for evaluation leads and AI engineers who need a structured, multi-dimensional quality report for a full conversation session, not just a single turn. The job-to-be-done is generating a unified scorecard that captures per-dimension scores (e.g., groundedness, goal completion, context adherence), an overall session quality rating, critical failure flags, and a summary justification. It is ideal when you need a holistic view of conversation quality that can be logged, trended, and used as a release gate or a diagnostic artifact. Use this when you have a complete conversation transcript and a defined set of evaluation dimensions that matter for your product.

Do not use this prompt for real-time, per-turn evaluation during a live conversation. It is designed for post-session or offline batch analysis. It is also not a replacement for single-turn scoring prompts when you need granular, turn-level diagnostics; instead, use the sibling Multi-Turn Context Adherence Scoring Prompt or Turn-by-Turn Citation Accuracy Prompt for that level of detail. This prompt requires a well-formed rubric as input—if you haven't defined your quality dimensions and scoring scales, start with the Conversation Evaluation Rubric Designer Prompt first. The output is a structured JSON scorecard, making it suitable for automated pipelines, dashboards, and evaluation databases, but it is not a conversational summary for human readers.

Before wiring this into a production evaluation pipeline, ensure you have a human-validated rubric and a set of calibration examples. The prompt's effectiveness depends entirely on the clarity of the input dimensions and the model's ability to apply them consistently. Always pair this with a judge alignment verification step (see the Multi-Turn Judge Alignment Verification Prompt) to measure how well the automated scores correlate with human judgment. For high-risk domains like healthcare or finance, the scorecard should be treated as a diagnostic aid, not a final authority, and critical failure flags must trigger human review before any action is taken.

The following prompt template produces a unified quality report across multiple conversation dimensions. It is designed to be wired into an evaluation harness that supplies the full conversation transcript, a list of scoring dimensions with their definitions and scales, and any source evidence required for groundedness checks. The template uses square-bracket placeholders for all variable inputs—replace these with your application's data before sending the prompt to the model.

textCopy
You are an expert conversation quality evaluator. Your task is to analyze the provided multi-turn conversation and produce a structured scorecard assessing quality across multiple dimensions.

## INPUTS

### Conversation Transcript
[CONVERSATION_TRANSCRIPT]

### Scoring Dimensions
[DIMENSIONS]
<!--
Each dimension must include:
- name: string
- definition: string describing what to evaluate
- scale: array of valid integer scores (e.g., [1,2,3,4,5])
- scale_descriptions: object mapping each score to its meaning (e.g., {"1": "Severely deficient", "5": "Excellent"})
- weight: float between 0.0 and 1.0 indicating relative importance (all weights must sum to 1.0)
- critical_failure_conditions: array of strings describing conditions that trigger a critical failure flag for this dimension
-->

### Source Evidence (if applicable)
[SOURCE_EVIDENCE]
<!--
Optional. Provide per-turn source documents or reference material for groundedness evaluation.
Format: {"turn_1": ["source_a", "source_b"], "turn_2": [...]}
-->

### Constraints
[CONSTRAINTS]
<!--
Optional. List any specific rules, policies, or behavioral constraints the assistant was expected to follow.
Example: ["Must refuse requests for personal data", "Must cite sources for factual claims"]
-->

## OUTPUT SCHEMA

Produce a JSON object with the following structure:

{
  "session_id": "string - identifier for the conversation",
  "turn_count": "integer - total number of turns evaluated",
  "dimension_scores": [
    {
      "dimension_name": "string",
      "score": "integer - must be one of the valid scores defined in the dimension's scale",
      "justification": "string - specific evidence from the conversation supporting this score, citing turn indices",
      "critical_failure_triggered": "boolean - whether any critical failure condition was met",
      "critical_failure_detail": "string or null - if triggered, describe which condition was met and at which turn"
    }
  ],
  "overall_score": "float - weighted average of dimension scores, rounded to 2 decimal places",
  "overall_rating": "string - one of: 'Excellent', 'Good', 'Adequate', 'Poor', 'Critical Failure'",
  "critical_failure_flags": [
    {
      "dimension": "string - dimension where critical failure occurred",
      "turn_index": "integer - turn where failure was detected",
      "condition": "string - the specific condition that was triggered",
      "severity": "string - one of: 'blocking', 'major', 'minor'"
    }
  ],
  "summary_justification": "string - 3-5 sentence narrative summarizing overall quality, key strengths, and primary concerns",
  "improvement_areas": [
    "string - specific, actionable area for improvement with turn reference"
  ]
}

## EVALUATION INSTRUCTIONS

1. Read the entire conversation transcript carefully, noting turn indices.
2. For each dimension, evaluate every relevant turn against the dimension's definition and scale.
3. Assign a score based on the aggregate performance across all turns, not just the worst or best single turn.
4. Check each dimension's critical failure conditions. If any condition is met, set critical_failure_triggered to true and document the specific turn and condition.
5. Calculate overall_score as the weighted average of all dimension scores.
6. Determine overall_rating:
   - "Excellent": overall_score >= 4.5
   - "Good": overall_score >= 3.5 and < 4.5
   - "Adequate": overall_score >= 2.5 and < 3.5
   - "Poor": overall_score >= 1.5 and < 2.5
   - "Critical Failure": any dimension has critical_failure_triggered = true with severity "blocking"
7. Write a summary_justification that synthesizes findings across dimensions.
8. List 1-3 specific improvement_areas with turn references.

## IMPORTANT RULES

- Only use scores that exist in each dimension's defined scale.
- Cite specific turn indices in justifications (e.g., "Turn 3: assistant ignored user's correction from Turn 2").
- If source evidence is provided, verify factual claims against it. Flag unsupported claims.
- If constraints are provided, check compliance per turn.
- Do not invent dimensions. Only score the dimensions provided in [DIMENSIONS].
- If the conversation is too short to evaluate a dimension meaningfully, note this in the justification and use the midpoint score.

To adapt this template for your evaluation pipeline, replace each placeholder with structured data from your application. The [DIMENSIONS] placeholder is the most critical—it defines what quality means for your specific use case. Provide dimensions as a JSON array with complete definitions, scales, and weights. For a customer support chatbot, dimensions might include 'Resolution Effectiveness', 'Tone Appropriateness', and 'Policy Compliance'. For a RAG assistant, include 'Groundedness' and 'Citation Accuracy'. The [SOURCE_EVIDENCE] placeholder should contain per-turn reference documents when evaluating groundedness; omit it for dimensions that don't require source verification. The [CONSTRAINTS] placeholder accepts a list of behavioral rules the assistant was expected to follow, enabling automated policy compliance checks across the full conversation.

Before deploying this prompt, validate that your dimension weights sum to 1.0 and that all scale values are integers within the defined ranges. Test the prompt against a golden dataset of conversations with known quality levels to calibrate score thresholds. For high-stakes evaluation (e.g., blocking a model release), always run human review on a sample of scorecards to verify alignment before trusting automated scores. Common failure modes include the model hallucinating turn indices, applying dimension definitions inconsistently across long conversations, and producing justifications that don't match the assigned scores—plan your validation harness accordingly.

This prompt is designed to sit inside an automated evaluation harness, not as a one-off manual review tool. The typical integration point is a batch evaluation pipeline that processes completed conversation logs, or a streaming evaluator that scores sessions as they conclude. The harness must supply the full conversation transcript, the active rubric dimensions, and any per-dimension reference materials (such as source documents for groundedness checks or policy documents for safety dimensions). Because the output is a structured scorecard, the harness should enforce schema validation immediately after the model responds—malformed JSON, missing required fields, or out-of-range scores must trigger a retry or fallback before the scorecard enters any downstream dashboard or decision system.

Wire the prompt into your application with a validate-then-retry loop. After receiving the model's response, parse the JSON and check: all required dimension keys are present, each score falls within the defined numeric range (e.g., 1–5), the overall_score is present and within range, critical_failure_flags is an array (even if empty), and summary_justification is a non-empty string. If validation fails, re-invoke the model with the same prompt plus an error message describing the specific validation failure, up to a maximum of 2 retries. After 2 failed retries, log the raw output, flag the session for human review, and emit a null scorecard with a validation_failed error code. For high-stakes domains (healthcare, finance, safety policy enforcement), always route sessions with any critical_failure_flag set to true to a human review queue regardless of the overall score.

Model choice matters for this prompt. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate defaults. Avoid smaller or older models that may drift on multi-dimensional scoring tasks or produce inconsistent JSON under long context. If your conversation transcripts exceed the model's effective context window, implement a sliding window summarization preprocessor that compresses earlier turns into structured summaries before passing the full recent turns and the compressed history to the scorecard prompt. Log every scorecard generation event with the prompt version, model identifier, conversation ID, raw model output, validation result, and final scorecard payload. This audit trail is essential for debugging score drift, calibrating judges, and defending evaluation decisions during release reviews. The next step after implementing this harness is to calibrate the generated scores against human evaluator ratings using the Multi-Turn Judge Alignment Verification Prompt from this same content group.