Reference Resolution Evaluation Rubric Prompt

This prompt is for AI engineering teams who have built or are iterating on a production reference resolver and need a structured, repeatable way to evaluate its quality. The primary job-to-be-done is generating a detailed evaluation rubric that an LLM judge can use to score the resolver's output on precision, recall, and the appropriateness of its clarification requests. The ideal user is an MLOps engineer, a backend developer working on a conversational AI pipeline, or a technical lead responsible for the quality of a multi-turn chat or copilot system. You should have a clear definition of your resolver's input (the ambiguous utterance and dialogue history) and its expected output (a resolved utterance, a clarification question, or an entity mapping) before using this prompt.

Use this prompt when you need to move beyond ad-hoc spot-checking and want a calibrated, criteria-based evaluation that can run automatically in a CI/CD pipeline or as part of a nightly regression suite. It is particularly valuable when you are A/B testing two different resolver prompts or architectures and need a judge that can perform pairwise comparisons with clear reasoning. The rubric it produces is designed to be granular, covering specific failure modes like incorrect gender agreement, entity type mismatches, and unnecessary clarifications. This allows you to decompose a vague 'the resolver is bad' signal into actionable, fixable sub-problems.

Do not use this prompt if you are still in the earliest prototyping phase and have no stable input/output contract for your resolver. The rubric's value comes from its specificity to a defined task; applying it to a moving target will produce noisy, unhelpful evaluations. This prompt is also not a substitute for human evaluation in high-stakes domains like healthcare or legal tech, where a model-graded rubric should be one signal among many. In those cases, use the rubric to triage and surface potential failures for mandatory human review, not as the final quality gate. If your resolver is a simple string match or a deterministic rule, this LLM-based evaluation is likely overkill; invest instead in unit tests for your resolution logic.

This prompt template instructs a language model to act as an evaluator for your production reference resolver. It is designed to be used in a CI/CD pipeline or an offline evaluation harness to score individual predictions against a ground-truth or expected output. The core job is to produce a structured, granular scorecard that goes beyond a simple pass/fail, breaking down performance into precision, recall, and clarification appropriateness. Use this template as the foundation for your automated eval judge, adapting the placeholders to fit your specific dialogue domain and output schema.

markdownCopy
You are an expert evaluator for a dialogue system's reference resolution module. Your task is to compare the [RESOLVED_UTTERANCE] against the [GOLD_REFERENCE] within the context of the [DIALOGUE_HISTORY]. You must produce a structured evaluation rubric in JSON format.

**Evaluation Criteria:**
1. **Precision:** Does the resolved utterance correctly identify the specific entity or entities the user was referring to, without including extraneous or incorrect entities? Score from 0.0 to 1.0.
2. **Recall:** Did the resolution capture all intended referents from the user's ambiguous phrase? Score from 0.0 to 1.0.
3. **Clarification Appropriateness:** If the input was ambiguous and the system chose to ask a clarifying question instead of resolving, was this the correct decision? If the system resolved, should it have asked for clarification instead? Score as 'correct_decision', 'missed_clarification', or 'unnecessary_clarification'.

**Input Data:**
- **Dialogue History:**
  [DIALOGUE_HISTORY]
- **User's Ambiguous Utterance:**
  [AMBIGUOUS_UTTERANCE]
- **System's Resolved Utterance (to evaluate):**
  [RESOLVED_UTTERANCE]
- **Gold-Standard Resolved Utterance:**
  [GOLD_REFERENCE]

**Output Schema:**
You must respond with a single JSON object conforming to this schema:
{
  "precision_score": <float 0.0-1.0>,
  "recall_score": <float 0.0-1.0>,
  "clarification_decision": "<correct_decision|missed_clarification|unnecessary_clarification>",
  "rationale": "<A concise explanation for each score and the clarification decision.>"
}

**Pairwise Comparison (Optional):**
If a [RESOLVED_UTTERANCE_B] is provided, add a `comparison` field to the output JSON:
{
  ...
  "comparison": {
    "winner": "<A, B, or tie>",
    "reason": "<Brief justification for the pairwise comparison result.>"
  }
}

**Constraints:**
- Base your evaluation strictly on the provided dialogue history and gold reference.
- If the gold reference is a clarification question, the `clarification_decision` for a resolved utterance must be 'missed_clarification'.
- Do not evaluate the fluency or grammar of the resolved utterance, only the correctness of the entity reference.

To adapt this template, start by replacing the placeholder descriptions with your actual data structures. The [DIALOGUE_HISTORY] should be a serialized representation of prior turns, which could be a simple \n-separated string or a JSON array of {role, content} objects. The [GOLD_REFERENCE] is your labeled expected output. For A/B testing, provide a second system's output in [RESOLVED_UTTERANCE_B] and the pairwise comparison logic will be activated. The output schema is a contract; ensure your parsing code expects these exact field names and enum values. For high-stakes domains, always log the rationale field for human auditability, and consider routing cases with a precision_score below 0.8 for manual review.

This prompt is designed to be used as an LLM judge inside an evaluation harness, not as a one-off manual review tool. The primary integration point is a CI/CD pipeline for prompt engineering, where every change to a reference resolution prompt triggers a regression run. The harness sends the rubric prompt to a strong evaluator model (GPT-4o, Claude 3.5 Sonnet, or equivalent) alongside the candidate resolver's output and the ground-truth dialogue context. The evaluator returns a structured scorecard that your test framework can parse and assert against.

To wire this in, construct a test case object with four fields: dialogue_history (the multi-turn conversation up to the ambiguous utterance), current_utterance (the turn containing the reference to resolve), candidate_resolution (the output from the resolver prompt you're testing), and ground_truth_referent (the correct entity ID or span). Feed these into the rubric prompt's [DIALOGUE_CONTEXT], [AMBIGUOUS_UTTERANCE], [RESOLVER_OUTPUT], and [GROUND_TRUTH] placeholders respectively. Validate the judge's output against a strict JSON schema before accepting scores—if the judge returns malformed JSON, retry once with a repair prompt, then fail the test case as unevaluable. Log every judge response with a trace ID that links back to the resolver prompt version, the test case ID, and the model snapshot used for judging.

For A/B comparison runs, extend the harness to call the rubric prompt twice: once for resolver variant A and once for variant B. After collecting both scorecards, use the pairwise comparison instructions embedded in the rubric to produce a preference judgment. Store these pairwise results in a structured results database (fields: test_case_id, variant_a_version, variant_b_version, winner, confidence, key_differentiator). Set a release gate: if variant B loses on precision or clarification appropriateness for any test case in your golden set, block the promotion. For high-stakes domains like healthcare or legal references, add a human review sampling step where 5% of judge evaluations are spot-checked by a domain expert, and track judge-human agreement rates over time to detect judge drift.

Model choice matters. The evaluator model should be at least as capable as the resolver model, and ideally from a different model family to reduce correlated errors. If you're testing a GPT-4o resolver, judge with Claude 3.5 Sonnet. If both resolver and judge are from the same provider, add a second judge from a different provider for arbitration on borderline cases (scores within 0.2 of the pass/fail threshold). Avoid running the judge on the same model as the resolver—this masks systematic failures that both models share. Set a maximum evaluation latency budget (2 seconds per test case is a reasonable target) and batch evaluations to control cost. Cache judge outputs keyed by (resolver_prompt_hash, test_case_hash) to avoid re-evaluating unchanged combinations.