Multi-Turn User Intent Tracking Accuracy Prompt

This prompt is designed for conversational AI teams who need to verify that their assistant correctly tracks user intent as it evolves across multiple conversation turns. Single-turn intent classification misses critical failures: an assistant that correctly identifies 'book a flight' on turn one but treats 'actually, I want a train instead' on turn three as a new unrelated query has failed at intent tracking. The core job-to-be-done is automated, repeatable evaluation of intent tracking accuracy before shipping a new model, prompt, or conversation handler. The ideal user is an ML engineer, evaluation lead, or product engineer who has conversation transcripts with ground-truth intent labels and needs per-turn accuracy scores, drift flags, and actionable misclassification diagnoses without manually reviewing hundreds of sessions.

To use this prompt effectively, you must provide three inputs: a full conversation transcript with turn-by-turn speaker labels, the assistant's inferred intent at each turn, and the ground-truth intent label sequence. The prompt acts as an LLM judge that compares inferred intents against ground truth, producing a structured evaluation payload that includes per-turn binary accuracy, intent drift detection when the assistant fails to follow a user's mid-conversation correction, and a diagnostic explanation for each misclassified turn. This is not a single-turn classification benchmark replacement—it specifically tests whether the assistant's intent model updates correctly when users change their minds, add constraints, or pivot goals mid-session.

Do not use this prompt when you lack ground-truth intent labels, when evaluating single-turn intent classifiers in isolation, or when the conversation contains ambiguous intent shifts that even human annotators would disagree on. The judge prompt assumes that ground-truth labels are authoritative; if your labeling process has low inter-annotator agreement, the evaluation scores will reflect label noise rather than model quality. For conversations where intent is genuinely ambiguous, pair this with a human calibration step or use the Multi-Turn Judge Alignment Verification Prompt to measure judge-human agreement before relying on automated scores. If you need to evaluate broader conversation quality beyond intent tracking—such as tone, safety, or groundedness—combine this prompt with the Multi-Turn Conversation Scorecard Generation Prompt for a unified quality report.

This prompt template is designed to be pasted directly into your evaluation harness. It instructs the model to act as an evaluator, comparing the assistant's inferred intent at each turn against a set of ground-truth intent labels. The primary job is to detect intent loss, misinterpretation, and silent intent shifts that degrade conversation quality. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to integrate with your existing logging and evaluation infrastructure. Before using it, ensure you have a conversation log and a corresponding list of ground-truth intents for each user turn.

textCopy
You are an expert evaluator of conversational AI systems. Your task is to assess how accurately an assistant tracks the user's evolving intent across a multi-turn conversation.

You will be provided with:
- A full conversation log between a user and an assistant.
- A list of ground-truth user intent labels for each user turn.

For each user turn in the conversation, you must:
1. Infer the user's primary intent based on the conversation history up to that point.
2. Compare your inferred intent with the provided ground-truth intent for that turn.
3. Assign a binary score: `true` if the inferred intent matches the ground-truth intent, `false` otherwise.
4. Provide a brief, evidence-based justification for your score, citing the specific user messages that support your inference.

After evaluating all turns, produce a final summary including:
- An overall intent tracking accuracy score (percentage of turns with a `true` match).
- A list of turns where intent was lost or misinterpreted, with the ground-truth intent, the incorrectly inferred intent, and the justification.

---
CONVERSATION LOG:
[CONVERSATION_LOG]

GROUND-TRUTH INTENTS (one per user turn, in order):
[GROUND_TRUTH_INTENTS]

OUTPUT_SCHEMA:
{
  "turn_evaluations": [
    {
      "turn_index": <integer>,
      "user_message": "<exact user text>",
      "ground_truth_intent": "<string>",
      "inferred_intent": "<string>",
      "intent_match": <boolean>,
      "justification": "<string>"
    }
  ],
  "overall_accuracy_score": <float between 0.0 and 1.0>,
  "misinterpreted_turns": [
    {
      "turn_index": <integer>,
      "ground_truth_intent": "<string>",
      "inferred_intent": "<string>",
      "justification": "<string>"
    }
  ]
}

CONSTRAINTS:
- Infer intent based on the entire conversation history, not just the current user message.
- If the user's intent is ambiguous, state that in your justification and compare against the most likely interpretation.
- Do not invent intents that are not present in the conversation.
- The `overall_accuracy_score` must be calculated as the number of `true` matches divided by the total number of turns.

To adapt this template, replace the [CONVERSATION_LOG] and [GROUND_TRUTH_INTENTS] placeholders with your data. The conversation log should be formatted clearly, with each turn prefixed by 'User:' or 'Assistant:'. The ground-truth intents must be a list of strings, one for each user turn, in the exact order they appear. For high-stakes evaluations, such as those used in a CI/CD release gate, always pair this prompt with a post-processing validation step. This validator should check that the output is valid JSON, that the number of evaluated turns matches the input, and that the calculated accuracy score is mathematically correct. If the output fails validation, implement a retry with a more explicit instruction or escalate for human review to diagnose systematic evaluation errors.

This prompt is designed to operate as a batch evaluation step within a larger conversation quality pipeline, not as a real-time chat module. The harness should feed the prompt a complete conversation transcript, a list of ground-truth intent labels indexed by turn, and the assistant's inferred intents for the same turns. The model's job is to compare the inferred intent against the ground truth at each turn and produce a structured accuracy report. Because intent tracking is a high-signal metric for conversational AI teams, the harness must enforce strict output validation before accepting scores into your evaluation database.

Wire the prompt into an evaluation runner that iterates over a dataset of scored conversations. For each conversation, assemble the [CONVERSATION_TRANSCRIPT], [GROUND_TRUTH_INTENTS] (a JSON array of {turn_index, intent_label, description} objects), and [INFERRED_INTENTS] (the same structure, produced by your assistant or an upstream intent classifier). Pass these into the prompt template along with your [INTENT_TAXONOMY] and [MISINTERPRETATION_THRESHOLD]. Configure the model call with temperature=0 and a low top_p value to maximize deterministic scoring. Use a model with strong instruction-following and JSON output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent. Request structured output matching the expected JSON schema: a per_turn_results array with turn_index, ground_truth_intent, inferred_intent, match (boolean), confidence (0-1), and misinterpretation_flag (boolean), plus a session_level_summary containing overall_accuracy, intent_loss_turns, and misinterpretation_turns.

After receiving the model response, validate the output before recording scores. Check that per_turn_results contains an entry for every turn in the ground-truth list, that match values are consistent with the intent comparison, and that overall_accuracy matches the computed ratio of matches to total turns. If validation fails, retry once with the same prompt plus the validation error message appended as a [CORRECTION_FEEDBACK] block. If the second attempt also fails validation, log the conversation as an eval failure and exclude it from aggregate metrics rather than accepting a potentially incorrect score. For high-stakes evaluations—such as pre-release quality gates or model selection decisions—route a sample of scored conversations to human reviewers for calibration. Store the raw prompt, model response, validation result, and any human override in your evaluation database for auditability and judge-alignment analysis over time.

Common failure modes in the harness include: the model swapping intent labels between turns when the conversation has rapid topic shifts, the model marking a match when the inferred intent is semantically close but not identical to the ground truth, and JSON parse failures when the output contains trailing text or markdown fences. Mitigate these by including explicit examples of near-miss intents in the [EXAMPLES] section of the prompt, by using strict JSON mode or structured output APIs where available, and by implementing a JSON repair step in the harness that strips markdown fences and trailing commas before parsing. Do not use this prompt for real-time intent correction during live conversations—it is an offline evaluation tool. For production intent tracking, implement a separate classifier with its own latency and reliability characteristics.