Inter-Rater Agreement Score Calculation Prompt Template

This prompt is designed for evaluation infrastructure teams who need to move from 'the judges seem to agree' to a statistical understanding of inter-rater reliability. The core job-to-be-done is computing agreement metrics—Cohen's Kappa for two judges, Fleiss' Kappa for three or more, and raw percent agreement—from a complete score matrix. You should use this prompt when you have already collected scores from multiple LLM judges rating the same set of items and need to decide whether your automated evaluation pipeline is consistent enough to trust without human oversight on every sample. The ideal user is an AI engineer or evaluation lead who has a structured dataset of judge outputs and needs a rigorous, repeatable statistical check before shipping a judge-based evaluation system.

The prompt requires a complete score matrix as input, meaning every judge has rated every item on the same categorical or ordinal scale. It assumes scores are already collected, formatted, and ready for computation. The output includes the primary agreement statistics plus divergence diagnostics that flag which specific items or judge pairs are driving disagreement. This is a batch computation workflow, not a real-time streaming check. Do not use this prompt when judges use incompatible rating scales (e.g., one judge uses 1–5 Likert and another uses binary pass/fail), when your score matrix is sparse with missing ratings, or when you need continuous live monitoring rather than a point-in-time reliability assessment. For sparse matrices, you will need imputation or pairwise deletion logic that this prompt does not handle.

Before running this prompt, verify that your score matrix is complete and that all judges used the same rating scale. If you have more than two judges, Fleiss' Kappa will be computed; for exactly two, Cohen's Kappa applies. The divergence diagnostics are your most actionable output—they tell you not just that agreement is low, but which judge pairs or specific items are the source of the problem. After receiving results, if agreement is below your threshold (typically Kappa < 0.6 warrants investigation), proceed to the sibling playbooks for Outlier Judge Detection or Disagreement Root Cause Analysis rather than trusting the evaluation pipeline. If agreement is acceptable, document the Kappa score and sample size as part of your judge fleet qualification evidence before promoting the pipeline to production.

This prompt template is designed to be copied directly into an LLM interface and adapted for your specific inter-rater agreement calculation task. It accepts a matrix of scores from multiple judges across multiple items and produces a comprehensive agreement report. The template uses square-bracket placeholders for all variable inputs, making it easy to swap in your own data, constraints, and output format requirements without modifying the core instruction structure.

codeCopy
You are a statistical computation engine specializing in inter-rater agreement analysis. Your task is to compute agreement metrics from a provided score matrix and produce a structured diagnostic report.

## INPUT DATA

### Score Matrix
[JUDGE_SCORE_MATRIX]

Format: Provide scores as a JSON array of arrays, where each inner array represents one item and contains the scores from all judges for that item. Example: [[3, 4, 3], [2, 2, 1], [5, 5, 4]] represents 3 items rated by 3 judges.

### Scale Type
[SCALE_TYPE]

Specify one of: "nominal", "ordinal", "interval", or "ratio".

### Judge Identifiers (Optional)
[JUDGE_IDS]

Provide as a JSON array of judge labels. If omitted, judges will be referenced by index position.

### Item Identifiers (Optional)
[ITEM_IDS]

Provide as a JSON array of item labels. If omitted, items will be referenced by index position.

## COMPUTATION REQUIREMENTS

1. Compute the following agreement metrics:
   - Percent agreement (pairwise and overall)
   - Cohen's Kappa for each judge pair (if exactly 2 judges)
   - Fleiss' Kappa (if 3 or more judges)
   - Krippendorff's Alpha as a supplementary metric

2. For each metric, include:
   - The computed value rounded to 4 decimal places
   - A 95% confidence interval where calculable
   - An interpretation label: "Poor" (<0.20), "Fair" (0.21-0.40), "Moderate" (0.41-0.60), "Good" (0.61-0.80), "Excellent" (0.81-1.00)

3. Produce a divergence diagnostic section that:
   - Identifies the judge pair with the lowest agreement
   - Identifies items with the highest score variance across judges
   - Flags any judge whose mean score deviates more than [DEVIATION_THRESHOLD] standard deviations from the group mean
   - Lists the top [TOP_N_DIVERGENT_ITEMS] items where judges disagree most

4. Run assumption checks and report warnings for:
   - Sparse rating patterns (judges using only a subset of the scale)
   - Low variance items (all judges giving identical or near-identical scores)
   - Small sample size (fewer than [MIN_ITEMS_THRESHOLD] items)
   - Unbalanced judge counts per item

## CONSTRAINTS
[CONSTRAINTS]

Specify any additional constraints, such as: "Exclude items with missing scores from all calculations", "Treat score value [X] as missing data", or "Only compute pairwise metrics for judge pairs with at least [N] co-rated items".

## OUTPUT FORMAT

Return a JSON object with this exact structure:
{
  "agreement_metrics": {
    "percent_agreement": {
      "overall": number,
      "pairwise": { "judge_i_judge_j": number },
      "confidence_interval_95": [lower, upper]
    },
    "cohens_kappa": {
      "pairwise": { "judge_i_judge_j": { "value": number, "ci_95": [lower, upper], "interpretation": string } },
      "note": "Computed only when exactly 2 judges present"
    },
    "fleiss_kappa": {
      "value": number,
      "ci_95": [lower, upper],
      "interpretation": string,
      "note": "Computed when 3+ judges present"
    },
    "krippendorff_alpha": {
      "value": number,
      "ci_95": [lower, upper],
      "interpretation": string
    }
  },
  "divergence_diagnostics": {
    "lowest_agreement_pair": { "judges": [string, string], "metric": string, "value": number },
    "highest_variance_items": [{ "item_id": string, "variance": number, "scores": [number] }],
    "outlier_judges": [{ "judge_id": string, "mean_score": number, "group_mean": number, "deviation_sds": number }],
    "top_divergent_items": [{ "item_id": string, "score_range": number, "scores": [number] }]
  },
  "assumption_warnings": [
    { "type": string, "severity": "warning" | "error", "detail": string, "recommendation": string }
  ],
  "summary": "One-paragraph plain-English interpretation of the overall agreement picture, highlighting the most important finding and any critical warnings."
}

## EXAMPLES (for reference)
[EXAMPLES]

Provide 1-2 worked examples with small score matrices and expected outputs to calibrate the computation. If omitted, the model will compute without examples.

## RISK LEVEL
[RISK_LEVEL]

Specify "low", "medium", or "high". For high-risk contexts (e.g., clinical assessment, legal evaluation, safety-critical scoring), the output must include an explicit statement that results require human review before operational use.

To adapt this template, replace each square-bracket placeholder with your actual data. The [JUDGE_SCORE_MATRIX] is the only strictly required input; all others have sensible defaults or can be omitted. For high-stakes evaluation pipelines, set [RISK_LEVEL] to "high" and ensure the [CONSTRAINTS] field specifies how to handle missing or edge-case data. The [DEVIATION_THRESHOLD] and [TOP_N_DIVERGENT_ITEMS] parameters let you tune the sensitivity of the divergence diagnostics to match your operational tolerance for judge disagreement. After running the prompt, validate the output JSON against the schema before ingesting it into your monitoring dashboard or alerting system.

This prompt is designed to be a stateless function within a larger evaluation pipeline. It expects a pre-assembled score matrix, not raw conversation logs. The calling application should first collect scores from all judges for all items, validate that the matrix is complete enough for the requested metrics, and then inject the matrix into the [SCORE_MATRIX] placeholder. Do not use this prompt to collect scores; it only computes agreement statistics from already-collected data.

Wrap the LLM call in a harness that performs pre-flight validation before sending the prompt. Check that the [SCORE_MATRIX] is a valid JSON array of arrays, that all rows have the same number of columns, and that the number of raters meets the minimum requirement for the requested statistic (e.g., Fleiss' Kappa requires at least 3 raters). If the matrix is sparse, pre-process it to either drop incomplete rows or impute values based on a configured strategy before passing it to the prompt. The harness should also enforce a maximum matrix size to avoid token limit errors—if the input exceeds 50 items or 10 raters, split the computation into batches and aggregate results in application code.

After receiving the LLM output, validate the JSON structure against the expected [OUTPUT_SCHEMA]. Confirm that all requested metrics (Cohen's Kappa, Fleiss' Kappa, percent agreement) are present and are numeric values within valid ranges (e.g., Kappa values between -1 and 1). If validation fails, retry once with an explicit error message injected into the [PREVIOUS_ERROR] field of the prompt. If the retry also fails, log the raw output and matrix for manual inspection and fall back to computing agreement scores using a deterministic statistical library like scikit-learn's cohen_kappa_score or statsmodels' fleiss_kappa.

Log every computation with the prompt version, model identifier, input matrix hash, output metrics, and validation status. For high-stakes evaluation pipelines where judge reliability directly impacts model release decisions, route all agreement scores below a configurable threshold (e.g., Kappa < 0.6) to a human review queue before the scores are used in downstream reporting. Never treat an LLM-computed agreement score as authoritative without verifying it against a deterministic implementation, especially when the score will be cited in model cards or compliance documentation.