Rubric Clarity and Judge Divergence Diagnostic Prompt Template

This prompt is for evaluation infrastructure teams who observe high inter-rater disagreement and need to determine whether the scoring divergence originates from ambiguous rubric language rather than judge bias or model inconsistency. The core job-to-be-done is isolating the rubric as a root cause: you have a rubric, a set of scored items with multiple judge ratings, and you need to identify which specific rubric sections or phrases correlate with disagreement. Without this diagnostic, teams waste effort recalibrating judges or adjusting scoring thresholds when the real fix is rewriting unclear criteria. The ideal user is a rubric designer, evaluation lead, or ML engineer who already has disagreement data and is ready to act on rubric improvements, not just measure agreement metrics.

You should use this prompt when you have at least 20-30 scored items with ratings from two or more judges, and you've already computed basic agreement metrics that show problematic divergence. The prompt requires structured input: the full rubric text with labeled sections, a matrix of judge scores per item, and optionally judge rationales if available. It produces a diagnostic report with per-section clarity scores, divergence hotspots mapped to specific rubric phrases, and concrete rewording suggestions ranked by expected impact. The output is designed to be actionable—you should be able to take the top rewording suggestion and test whether it reduces disagreement in a follow-up scoring round.

Do not use this prompt when you have fewer than 10 scored items, when judges haven't been calibrated at all, or when the disagreement is clearly caused by model output quality variance rather than rubric interpretation. This prompt assumes the judges are reasonably competent and the primary variable is rubric clarity. If your judges are producing wildly different scores because they're different models with different capabilities, start with the Cross-Model Judge Agreement Comparison prompt instead. If you suspect individual judges are outliers, use the Outlier Judge Detection prompt first. This diagnostic is specifically for the case where agreement metrics are borderline—not catastrophically broken—and you need to decide whether to fix the rubric or the judges.

The prompt is designed for single-round diagnosis. After receiving the output, you should manually review the top divergence hotspots and rewording suggestions before applying them. Do not automatically rewrite your rubric based on the model's suggestions without human review—the model can identify ambiguous language but may propose rewordings that change the scoring intent. Run a small calibration round with the revised rubric and the same items to verify that disagreement decreases before rolling out the changes to your full evaluation pipeline. If disagreement persists after two rubric revision cycles, the root cause likely lies elsewhere and you should escalate to the Disagreement Root Cause Analysis prompt.

This prompt template is designed to be the core diagnostic tool for investigating why LLM judges diverge on specific evaluation items. It forces a structured analysis that links ambiguous rubric language directly to observed scoring disagreements, producing actionable rewording suggestions rather than vague commentary. The template expects you to provide the exact rubric text, a sample of items where judges disagreed, the scores they assigned, and any rationales they produced. It then isolates the specific rubric sections causing confusion and proposes concrete language fixes.

textCopy
You are an evaluation rubric diagnostician. Your task is to analyze a scoring rubric and a set of disagreement cases to identify which rubric criteria are causing judge divergence.

## INPUTS

### RUBRIC
[RUBRIC_TEXT]

### DISAGREEMENT DATA
[DISAGREEMENT_ITEMS]
<!-- Format: For each item, include the item ID, the output being evaluated, the scores from each judge, and any judge rationales provided. -->

### CONFIGURATION
- Minimum divergence frequency to flag a rubric section: [MIN_FREQUENCY]
- Maximum number of rewording suggestions per section: [MAX_SUGGESTIONS]

## TASK

1. **Parse the Rubric**: Identify every distinct scoring criterion, dimension, or section in the rubric. Assign each a unique ID.

2. **Map Disagreements to Criteria**: For each disagreement case, determine which rubric criteria the judges are interpreting differently. Cite specific phrases from the rubric and specific excerpts from judge rationales as evidence.

3. **Score Clarity**: For each rubric criterion, assign a clarity score from 1 (highly ambiguous) to 5 (perfectly clear) based on the frequency and severity of linked disagreements.

4. **Identify Divergence Hotspots**: List the top [TOP_N] rubric sections most responsible for judge divergence, ranked by impact.

5. **Propose Rewordings**: For each hotspot, provide [MAX_SUGGESTIONS] alternative phrasings that would reduce ambiguity. Each suggestion must:
   - Preserve the original evaluation intent.
   - Replace vague terms (e.g., "good," "sufficient," "appropriate") with observable criteria.
   - Add examples or counterexamples where helpful.

## OUTPUT FORMAT

Return a JSON object with this exact schema:
{
  "rubric_sections": [
    {
      "section_id": "string",
      "original_text": "string",
      "clarity_score": number,
      "linked_disagreement_count": number,
      "ambiguity_phrases": ["string"],
      "rewordings": [
        {
          "suggested_text": "string",
          "rationale": "string"
        }
      ]
    }
  ],
  "divergence_hotspots": [
    {
      "section_id": "string",
      "impact_rank": number,
      "disagreement_examples": ["string"]
    }
  ],
  "global_recommendations": ["string"]
}

## CONSTRAINTS
- Do not invent disagreement cases. Only analyze the provided data.
- If a rubric section has zero linked disagreements, set clarity_score to 5 and provide no rewordings.
- If judge rationales are missing or insufficient, note this in global_recommendations and do not fabricate evidence.
- Preserve all original scoring scale values and labels in any rewording suggestions.

To adapt this template, start by populating [RUBRIC_TEXT] with the full, unedited text of the evaluation rubric you are debugging. The [DISAGREEMENT_ITEMS] placeholder requires structured data: for each item where judges disagreed, provide the item ID, the raw output being evaluated, the score from Judge A, the score from Judge B, and any free-text rationales the judges generated. If your evaluation pipeline does not capture judge rationales, you must first implement that logging; without rationales, the diagnostic power of this prompt drops significantly, and the model will be forced to speculate about the cause of divergence. Set [MIN_FREQUENCY] to a value that filters out noise—start with 2 or 3 if you have a large disagreement sample, or 1 if you are debugging a small set of critical failures. Set [MAX_SUGGESTIONS] to 3 for focused output. The [TOP_N] parameter in step 4 should be set to the number of hotspots you want to investigate first, typically 3 to 5.

Before relying on the output, validate that the model correctly parsed your rubric into its constituent sections. A common failure mode is the model merging distinct criteria or splitting a single criterion into artificial sub-parts. Compare the section_id list against your rubric's actual structure. If the model's rewording suggestions change the scoring scale or introduce new evaluation dimensions, reject them and re-run with tighter constraints. For high-stakes evaluation systems where rubric changes affect product decisions, always route the rewordings through a human rubric designer for approval before deploying them to production judges. The JSON output schema is designed to be machine-readable so you can programmatically diff the original and suggested rubric text, but the final decision to accept a rewording should remain a human judgment call.

This prompt is not a one-off debugging tool; it is a diagnostic step inside a larger judge reliability pipeline. The implementation harness should treat the prompt as a function that accepts a rubric, a set of scored items with judge rationales, and a divergence threshold. The output is a structured diagnostic report that can be logged, versioned, and fed into downstream repair workflows. Because the prompt analyzes judge behavior, the harness must ensure that the input data is complete and representative before invocation—running the diagnostic on a single divergent item will produce anecdotal results, not actionable patterns.

Wire the prompt into a scheduled or trigger-based evaluation job. When inter-rater agreement drops below a configured threshold (e.g., Cohen's Kappa < 0.6), the harness should automatically collect the last N scored items, group them by rubric section, and invoke this prompt with the full rubric text, the score matrix, and the divergence threshold. Validate the output against a strict JSON schema that requires rubric_sections (array of objects with section_id, clarity_score, divergence_hotspot, and rewording_suggestions) and diagnostic_summary (object with primary_cause, confidence, and recommended_actions). If the output fails schema validation, retry once with a more explicit [OUTPUT_SCHEMA] constraint injected into the prompt. Log every invocation—input parameters, raw output, validation result, and retry count—so that the diagnostic trail is auditable. For high-stakes evaluation pipelines (e.g., safety scoring, regulatory compliance checks), route recommended_actions that suggest rubric changes to a human review queue before any automated rubric update is applied.

Avoid running this diagnostic on every evaluation batch. The prompt is designed for investigation, not continuous monitoring. Overuse will generate noise and erode trust in the diagnostic signal. Instead, gate invocation on a statistical process control rule: trigger only when agreement metrics breach a control limit for two consecutive evaluation windows. Pair this prompt with the Inter-Rater Agreement Score Calculation Prompt Template to produce the input score matrix and the Disagreement Root Cause Analysis Prompt Template to drill into specific items flagged as divergence hotspots. The output of this diagnostic should feed into a rubric versioning system—each clarity report becomes evidence for a rubric change proposal, linking specific phrases to measured judge disagreement.