Inter-Rater Reliability Calculation Prompt for LLM Judges

This prompt is for evaluation platform teams who need to measure how consistently their fleet of LLM judges scores the same outputs. The job-to-be-done is calculating agreement coefficients across multiple judges, identifying which evaluation dimensions produce the most disagreement, and surfacing outlier judges whose scoring patterns diverge from the group. The ideal user is an MLOps engineer, evaluation infrastructure lead, or AI quality engineer responsible for maintaining a trustworthy automated evaluation pipeline. Required context includes a dataset of scored items where each item has been evaluated by at least two LLM judges on the same rubric dimensions, along with the raw scores and any judge-provided justifications.

Do not use this prompt when you only have a single judge, when you need real-time scoring latency guarantees, or when the evaluation dimensions are purely objective and deterministic (e.g., exact string match, regex validation). This prompt is designed for subjective quality dimensions where inter-rater agreement is a meaningful signal. It is also not a replacement for human calibration studies—use the LLM Judge Calibration Against Human Ratings Prompt Template for that workflow. The prompt assumes your judges are already producing structured scores; it does not design the rubric or fix broken judge instructions. For misalignment diagnosis, pair this with the Score Discrepancy Root Cause Analysis Prompt.

Before running this prompt, ensure your input data includes judge identifiers, item identifiers, per-dimension scores, and optional justifications. The prompt produces agreement coefficients (such as Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha depending on your judge count), a disagreement breakdown by score range and dimension, and pairwise judge similarity metrics. Acceptable agreement thresholds should be configured per dimension based on your downstream decision impact—a pass/fail gate for production releases demands higher agreement than an exploratory quality dashboard. Escalation criteria for low-agreement dimensions should route to human review or trigger the Judge Instruction Rewriting Prompt After Misalignment Detection. Always log the raw agreement outputs and threshold decisions for audit trails; this workflow is high-risk when automated scores gate production deployments.

This template is designed to be dropped into an evaluation pipeline that already has multiple LLM judge outputs for the same set of items. It expects structured input containing per-judge scores, item metadata, and the evaluation rubric that was used. The prompt instructs the model to act as a measurement analyst, not as a judge itself—its job is to compute agreement statistics, identify disagreement patterns, and flag dimensions or judge pairs that fall below acceptable reliability thresholds. Use this when you need a programmatic reliability report before trusting aggregated judge scores in production.

codeCopy
You are a measurement analyst calculating inter-rater reliability for a fleet of LLM judges. Your task is to produce a structured reliability report from raw judge scores. Do not re-score any items. Do not express opinions about the rubric quality. Only compute and report agreement metrics.

## INPUT DATA

### Evaluation Rubric
[RUBRIC_DEFINITION]

### Items and Judge Scores
[ITEMS_WITH_SCORES]

Each item includes:
- item_id: unique identifier
- scores: object mapping judge_id to score (numeric or categorical per rubric)
- [OPTIONAL_ITEM_METADATA]

### Configuration
- Agreement coefficient: [COEFFICIENT_TYPE] (e.g., "kappa", "weighted_kappa", "icc", "krippendorff_alpha", "percent_agreement")
- Acceptable threshold: [AGREEMENT_THRESHOLD]
- Disagreement severity levels: [SEVERITY_LEVELS]
- Minimum items per judge: [MIN_ITEMS]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "report_metadata": {
    "coefficient_type": "string",
    "acceptable_threshold": number,
    "total_items": number,
    "total_judges": number,
    "score_type": "numeric | categorical | ordinal",
    "computation_timestamp": "ISO 8601"
  },
  "overall_agreement": {
    "coefficient_value": number,
    "confidence_interval_95": [number, number],
    "interpretation": "poor | fair | moderate | substantial | almost_perfect",
    "meets_threshold": boolean
  },
  "per_dimension_agreement": [
    {
      "dimension": "string",
      "coefficient_value": number,
      "meets_threshold": boolean,
      "item_count": number
    }
  ],
  "pairwise_judge_similarity": [
    {
      "judge_a": "string",
      "judge_b": "string",
      "coefficient_value": number,
      "shared_items": number,
      "interpretation": "string"
    }
  ],
  "disagreement_breakdown": [
    {
      "score_range": "string",
      "disagreement_rate": number,
      "item_count": number,
      "example_item_ids": ["string"]
    }
  ],
  "low_agreement_dimensions": [
    {
      "dimension": "string",
      "coefficient_value": number,
      "gap_to_threshold": number,
      "contributing_judge_pairs": ["string"]
    }
  ],
  "outlier_judges": [
    {
      "judge_id": "string",
      "mean_pairwise_agreement": number,
      "deviation_from_fleet_mean": number,
      "severity_tendency": "harsh | lenient | inconsistent",
      "affected_dimensions": ["string"]
    }
  ],
  "escalation_required": boolean,
  "escalation_items": [
    {
      "trigger": "string",
      "severity": "warning | critical",
      "detail": "string",
      "recommended_action": "string"
    }
  ]
}

## COMPUTATION RULES

1. Compute the specified agreement coefficient across all judges for all items.
2. For per-dimension agreement, group scores by dimension and compute separately. If a judge did not score a dimension, exclude that judge from that dimension's calculation.
3. For pairwise similarity, compute the coefficient for every judge pair that shares at least [MIN_ITEMS] items.
4. Disagreement breakdown: group items by their score range (e.g., "1-2", "3-4", "5") and compute the proportion of items where judges disagreed by more than 1 point (or 1 category).
5. Flag any dimension where the coefficient falls below [AGREEMENT_THRESHOLD] as low_agreement.
6. Identify outlier judges as those whose mean pairwise agreement with all other judges falls more than 2 standard deviations below the fleet mean.
7. Set escalation_required to true if any dimension fails the threshold OR any judge is flagged as an outlier.
8. For each escalation trigger, provide a severity level and a concrete recommended action (e.g., "review judge instructions for dimension X", "recalibrate judge Y against human ratings", "investigate items in score range Z for rubric ambiguity").

## CONSTRAINTS

- Do not fabricate item_ids or scores. Use only the data provided.
- If fewer than 3 judges or fewer than [MIN_ITEMS] items are provided, set overall_agreement.coefficient_value to null and include an escalation item explaining insufficient data.
- If the score type is categorical and the coefficient requires numeric input, use one-hot encoding and note this in report_metadata.
- Round all coefficient values to 4 decimal places.
- If a judge has missing scores for some items, handle pairwise deletion (exclude only the missing pair, not the entire judge).

## EXAMPLES

[FEW_SHOT_EXAMPLES]

Adapt this template by replacing each square-bracket placeholder with values from your evaluation pipeline. [RUBRIC_DEFINITION] should contain the exact scoring dimensions, scale anchors, and criteria that judges used—this context helps the model interpret score ranges correctly. [ITEMS_WITH_SCORES] is typically a JSON array generated by your judge orchestration layer; ensure each item has a stable item_id and that judge identifiers are consistent across items. [COEFFICIENT_TYPE] must match your score type: use weighted kappa or ICC for ordinal/numeric scales, Fleiss' kappa for categorical, Krippendorff's alpha for incomplete designs. Set [AGREEMENT_THRESHOLD] based on your use case risk tolerance—0.61 for moderate agreement is a common starting point, but raise it to 0.81 for high-stakes evaluation dimensions. [FEW_SHOT_EXAMPLES] should include 1-2 worked examples showing correct coefficient computation and escalation logic for your specific rubric structure.

Before wiring this into production, validate the output against a known statistical library (e.g., irr in R, statsmodels in Python, or pingouin) on a small test batch. The LLM is computing approximations—not running a statistical package—so discrepancies above 0.05 on any coefficient warrant human review or switching to a code-based computation step. For high-stakes evaluation pipelines where judge reliability directly gates model releases, use this prompt as a diagnostic layer that flags problems, then confirm findings with deterministic statistical code before taking action.

This prompt is designed as a batch processing step within an LLM evaluation platform, not a one-off chat interaction. It expects a structured payload of judge scores and produces a structured reliability report. The implementation harness must validate both the input payload before calling the model and the output JSON after receiving it. Because the prompt performs statistical calculations (agreement coefficients, pairwise similarity), the harness should treat any malformed output or missing required field as a hard failure that triggers a retry or escalation—never a silent fallback to a default value.

Wire the prompt into a pipeline stage that runs after all judges have scored a batch of items. The input assembler must collect per-item scores from each judge, validate that all judges scored all items (no missing data), and format the [JUDGE_SCORES] array. Before calling the model, run a structural pre-check: confirm the number of judges is at least 2, the number of items is at least 5 for meaningful agreement statistics, and all scores fall within the declared [SCORE_RANGE]. If any pre-check fails, abort and return a clear error to the pipeline orchestrator rather than sending bad data to the model. After receiving the model response, validate the output JSON against the expected schema: required fields include agreement_coefficient, pairwise_similarity_matrix, disagreement_breakdown, and low_agreement_dimensions. If validation fails, retry once with the same input and a stronger format constraint appended to the prompt. If the retry also fails, log the raw output and escalate to a human reviewer with the failed validation details.

For model choice, prefer a model with strong JSON-following behavior and sufficient context window to hold the full score matrix. If the score matrix is large (many judges × many items), consider splitting by evaluation dimension and running the prompt once per dimension, then aggregating results in application code. Log every invocation: input hash, model version, output schema validation status, agreement coefficient, and any retry events. This log becomes audit evidence for judge fleet health over time. Set an alert threshold on the agreement coefficient—if it drops below your organization's minimum (commonly 0.7 for Cohen's kappa or 0.8 for percent agreement, depending on domain risk), trigger a review of judge instructions and calibration data. Do not use this prompt's output to automatically override individual judge scores; it is a monitoring and diagnosis tool, not a correction mechanism.