Score Normalization Across Judges Prompt Template

This prompt is for evaluation infrastructure teams who run multiple LLM judges and discover that raw scores are not directly comparable. One judge might use the full 1–5 scale while another clusters around 3–4, making it impossible to aggregate scores or set uniform thresholds. The job-to-be-done is harmonizing score distributions across judges so that a '4' from Judge A means the same thing as a '4' from Judge B. You need this when you are building automated evaluation pipelines, computing inter-rater reliability metrics, or preparing judge scores for downstream consumption by dashboards, CI/CD gates, or model selection workflows.

Use this prompt when you already have a raw score matrix—multiple judges scoring the same items—and you need to apply a transformation that preserves relative ordering while removing scale bias. The prompt expects you to provide the raw scores, scale metadata (min/max per judge, whether the scale is ordinal or interval), and your preferred normalization method (z-score, min-max, or rank-based). It produces normalized scores plus a transformation rationale you can audit. Do not use this prompt when judges are scoring entirely different items, when you lack scale metadata, or when you need to detect why judges disagree—that belongs to the Disagreement Root Cause Analysis or Judge Bias Detection playbooks.

This prompt is a statistical harmonization tool, not a judge quality evaluator. It assumes your judges are competent but differently calibrated. If your judges are producing random or adversarial scores, normalization will mask the problem rather than fix it. Always run inter-rater agreement checks before normalization. The output includes before/after distribution summaries so you can verify the transformation didn't distort relative rankings. For high-stakes evaluation pipelines where normalized scores feed automated decisions, require human review of the transformation rationale and spot-check at least 5% of normalized scores against the original judgments.

This template is the core instruction set you will send to an LLM to perform score normalization across a fleet of judges. It accepts a raw score matrix, metadata about each judge's scale usage, and a target normalization method. The prompt is designed to produce not just normalized scores but also the transformation parameters and a before/after distribution summary, making the normalization process auditable and reproducible. Use this template as the starting point for building a normalization microservice or a step in your evaluation pipeline.

codeCopy
You are a statistical normalization engine. Your task is to harmonize scores from multiple judges who used different scale ranges, distributions, or severity patterns.

## INPUT DATA

### Raw Score Matrix
[JUDGE_SCORE_MATRIX]

### Judge Scale Metadata
[JUDGE_METADATA]

### Target Normalization Method
[NORMALIZATION_METHOD]

### Constraints
[CONSTRAINTS]

## OUTPUT SCHEMA

Return a single JSON object with the following structure:
{
  "normalization_method": "string (the method applied)",
  "transformation_parameters": {
    // Method-specific parameters, e.g., mean, std_dev for z-score; min, max for min-max
  },
  "normalized_scores": {
    "judge_id": [array of normalized scores in the same order as input]
  },
  "distribution_comparison": {
    "before": {
      "judge_id": {
        "mean": number,
        "std_dev": number,
        "min": number,
        "max": number,
        "q1": number,
        "median": number,
        "q3": number
      }
    },
    "after": {
      "judge_id": {
        "mean": number,
        "std_dev": number,
        "min": number,
        "max": number,
        "q1": number,
        "median": number,
        "q3": number
      }
    }
  },
  "transformation_rationale": "string explaining why this method was chosen and any caveats",
  "warnings": ["array of strings flagging potential issues like small sample size or extreme outliers"]
}

## NORMALIZATION METHODS

- **z-score**: Transform each judge's scores to have mean 0 and standard deviation 1 using their own distribution. Use when judges have different means and variances but you assume the underlying quality distribution is similar.
- **min-max**: Scale each judge's scores to [0, 1] using their observed min and max. Use when judges use different scale ranges but you want to preserve relative ordering within each judge.
- **rank-based**: Replace scores with percentile ranks within each judge's distribution. Use when you want to eliminate scale effects entirely and only preserve ordering.

## RULES

1. Normalize each judge independently using only their own score distribution.
2. If a judge has fewer than [MIN_SAMPLE_SIZE] scores, include a warning and use the method cautiously.
3. If the chosen method is inappropriate for the data (e.g., min-max with a single unique value), flag it in warnings and fall back to rank-based normalization.
4. Preserve the original item ordering so normalized scores can be aligned with the input matrix.
5. Round normalized scores to [DECIMAL_PLACES] decimal places.
6. If [CONSTRAINTS] specifies a target range, apply an additional linear transformation after the primary method.

## EXAMPLE

Input:
- Judge A scores: [8, 9, 7, 8, 9] (scale 1-10)
- Judge B scores: [3, 4, 2, 5, 4] (scale 1-5)
- Method: z-score

Output:
{
  "normalization_method": "z-score",
  "transformation_parameters": {
    "judge_A": {"mean": 8.2, "std_dev": 0.84},
    "judge_B": {"mean": 3.6, "std_dev": 1.14}
  },
  "normalized_scores": {
    "judge_A": [-0.24, 0.95, -1.43, -0.24, 0.95],
    "judge_B": [-0.53, 0.35, -1.40, 1.23, 0.35]
  },
  "distribution_comparison": { ... },
  "transformation_rationale": "Z-score normalization was applied to center both judges at mean 0 with unit variance, removing scale and location differences while preserving relative distances.",
  "warnings": []
}

Now process the input data above and return only the JSON object.

To adapt this template for your pipeline, replace the square-bracket placeholders with live data. [JUDGE_SCORE_MATRIX] should be a JSON object mapping judge IDs to arrays of numeric scores. [JUDGE_METADATA] should include each judge's declared scale range and any known calibration offsets. [NORMALIZATION_METHOD] accepts one of the three supported methods, but you can extend the method list if your use case requires robust scaling or quantile normalization. [CONSTRAINTS] allows you to specify a target output range (e.g., "scale all scores to 0-100") or exclude specific judges from normalization. The [MIN_SAMPLE_SIZE] and [DECIMAL_PLACES] tokens let you tune the prompt for statistical rigor versus readability. Always validate the output JSON against the schema before passing normalized scores to downstream evaluation metrics or dashboards.

The Score Normalization Across Judges prompt is designed to sit between raw judge score collection and downstream decision-making. It accepts a matrix of raw scores from multiple judges, scale metadata, and a normalization method, then returns transformed scores with before/after distribution summaries and transformation rationale. This harness is critical when judges use different scale ranges, exhibit leniency or severity skew, or apply inconsistent score spacing—problems that make raw score aggregation unreliable.

Wire the prompt into your evaluation pipeline as a post-collection transformation step. After all judges have scored a batch of items, construct the input payload: a raw_score_matrix where rows are items and columns are judges, a scale_metadata object specifying each judge's min/max/type (e.g., Likert 1-5, 0-100 slider, binary pass/fail), and a normalization_method parameter (z-score, min-max, or rank-based). Validate the input before calling the model: check that the matrix has no missing cells beyond a configurable threshold (sparse matrices with >20% missing values should trigger imputation or exclusion), that scale types are compatible with the chosen method (rank-based normalization handles mixed scale types; z-score assumes interval scales), and that at least three judges are present for meaningful distribution comparison. Log the raw matrix, metadata, and method choice as an immutable record before transformation.

After receiving the model's output, run structural validation: confirm the normalized_scores matrix has the same dimensions as the input, verify that transformation_rationale is present and non-empty, and check that distribution_comparison contains before/after summary statistics (mean, std, min, max, quartiles) for each judge. If the output fails structural checks, retry once with the same prompt and a [CONSTRAINTS] addition emphasizing dimension preservation. For high-stakes evaluation pipelines—such as model quality gates, safety assessments, or regulatory evidence—route outputs with anomalous distribution shifts (e.g., a judge's normalized scores showing >2 standard deviation change in rank order) to a human reviewer before accepting the transformation. Store the normalized scores alongside the raw scores, preserving full lineage for auditability.

Model choice matters: use a model with strong numerical reasoning and instruction-following (GPT-4, Claude 3.5 Sonnet, or equivalent). Avoid small or quantized models that may miscalculate z-scores or mishandle edge cases like zero-variance judge columns. Set temperature to 0 for deterministic transformations. If your pipeline processes hundreds of items, batch the score matrix into chunks of 50-100 items to stay within context windows while maintaining per-batch distribution fidelity. Do not use this prompt for real-time scoring adjustments without caching—the transformation should be computed once per evaluation batch, not per user request. The normalized scores become the input for downstream agreement metrics (Cohen's Kappa, Fleiss' Kappa) and consensus derivation, so treat this step as a data integrity boundary: if normalization fails, halt the pipeline rather than propagating distorted scores.