Score Normalization Prompt Across Multiple LLM Judges

This prompt is designed for evaluation infrastructure teams who operate multiple LLM judges against the same set of outputs and need to combine their scores into a single, normalized distribution. The core job-to-be-done is post-hoc score aggregation when your judges operate on heterogeneous numeric scales (e.g., one judge uses 1–5, another uses 1–10) or exhibit different severity tendencies (one judge is consistently harsher than another). Use this prompt when you already have a raw score matrix—rows are evaluated items, columns are judges—and per-judge metadata describing their scale, observed mean, and variance. The ideal user is an MLOps engineer or evaluation lead who needs to produce apples-to-apples comparisons before feeding scores into downstream ranking, thresholding, or reporting systems.

The prompt assumes you have completed your evaluation runs and collected raw scores. It requires three inputs: a [RAW_SCORE_MATRIX] mapping items to judge scores, a [JUDGE_METADATA] block specifying each judge's scale range and calibration statistics, and an [OUTPUT_SCHEMA] defining the desired normalized range (typically 0–1 or 1–5). The prompt produces normalized scores with per-judge mapping functions, a rank-order preservation check, and an agreement matrix showing pairwise judge alignment after normalization. Do not use this prompt when all judges share an identical scale and calibration, or when you need real-time scoring rather than batch normalization. It is also inappropriate when you lack per-judge statistics—the normalization math requires observed distributions, not just scale declarations.

Before invoking this prompt, verify that your raw scores are complete and that each judge has scored every item. Missing scores will produce misleading normalization artifacts. If your judge fleet includes models with known failure modes on certain input types, consider running the Judge Bias Detection Prompt first to flag systematic scoring patterns that normalization alone cannot fix. After normalization, always validate that rank ordering is preserved within acceptable tolerance—a normalized score that inverts the original ranking indicates a broken mapping function and requires human review of the judge metadata or raw scores.

This prompt template accepts raw scores from multiple LLM judges—each potentially using different scales, severity tendencies, or internal calibration—and produces a normalized score distribution with per-judge mapping functions. The output enables apples-to-apples comparison across judges and detects when normalization would distort the underlying rank ordering. Replace every square-bracket placeholder with your actual data before sending the prompt to the model. The template is designed to work as a standalone evaluation step in a pipeline: feed it judge outputs, receive normalized scores and validation checks.

textCopy
You are a score normalization engine for an LLM evaluation pipeline. Your job is to take raw scores from multiple judges—each potentially using different numeric scales, scoring philosophies, or severity profiles—and produce a normalized score distribution that enables fair comparison across judges.

## INPUT
[JUDGE_SCORES]

## CONSTRAINTS
- Preserve rank ordering within each judge's scores. If Judge A ranked item X above item Y in raw scores, the normalized scores must maintain that ordering.
- Map all scores to the target scale: [TARGET_SCALE_MIN] to [TARGET_SCALE_MAX].
- If a judge's raw scores are on a categorical or ordinal scale, first convert them to numeric values using this mapping: [CATEGORICAL_MAPPING].
- Do not assume any judge is the "reference" or "ground truth" judge unless explicitly specified.
- If fewer than [MIN_SAMPLES_PER_JUDGE] scores are available for a judge, flag that judge as having insufficient data and exclude them from normalization.

## OUTPUT_SCHEMA
Return a JSON object with this structure:
{
  "normalized_scores": [
    {
      "item_id": "string",
      "judge_id": "string",
      "raw_score": number,
      "normalized_score": number,
      "normalization_method": "string"
    }
  ],
  "per_judge_mapping": [
    {
      "judge_id": "string",
      "raw_scale_min": number,
      "raw_scale_max": number,
      "mapping_function": "string",
      "mapping_parameters": {},
      "samples_used": number
    }
  ],
  "rank_preservation_checks": [
    {
      "judge_id": "string",
      "rank_violations": number,
      "violation_details": ["string"],
      "rank_correlation": number
    }
  ],
  "warnings": ["string"],
  "excluded_judges": ["string"]
}

## NORMALIZATION METHODS
Use the most appropriate method per judge based on score distribution characteristics:
- "min_max": Linear scaling to target range. Use when scores are roughly uniform.
- "z_score": Standardization to target mean and standard deviation. Use when scores are approximately normal.
- "quantile": Rank-based mapping. Use when score distributions are skewed or have outliers.
- "categorical_map": Direct mapping from categorical labels. Use only when raw scores are categorical.

## VALIDATION RULES
- After normalization, compute Spearman rank correlation between raw and normalized scores per judge. Flag any judge with correlation below [RANK_CORRELATION_THRESHOLD].
- If any judge has rank violations, include the specific item pairs where ordering flipped in violation_details.
- Add a warning if the normalized score range for any judge is compressed to less than [MIN_EFFECTIVE_RANGE] percent of the target scale, indicating potential information loss.
- Add a warning if any judge's mapping function is extrapolating beyond their observed raw score range.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

Return only the JSON object. Do not include explanations outside the JSON.

Adaptation guidance: Replace [JUDGE_SCORES] with a JSON array of objects containing item_id, judge_id, and raw_score fields. Set [TARGET_SCALE_MIN] and [TARGET_SCALE_MAX] to your desired normalized range (commonly 1–5 or 0–100). If any judges use categorical labels like "Poor/Fair/Good/Excellent", provide the numeric mapping in [CATEGORICAL_MAPPING] as a JSON object. Set [MIN_SAMPLES_PER_JUDGE] to at least 10 for reliable normalization; lower values produce unstable mappings. Set [RANK_CORRELATION_THRESHOLD] to 0.99 for strict rank preservation or 0.95 if minor reordering is acceptable. Set [MIN_EFFECTIVE_RANGE] to 50 (percent) to catch judges whose normalized scores collapse into a narrow band. Include [FEW_SHOT_EXAMPLES] with 2–3 worked examples showing input scores and expected normalized output for your specific score ranges and judge count. Run this prompt after every evaluation batch, not once per judge, because normalization parameters should update as score distributions shift. Validate the output JSON against the schema before ingesting normalized scores into downstream dashboards or decision systems.

Score normalization across multiple LLM judges is not a one-off prompt execution. It belongs inside an evaluation harness that treats the normalization step as a pipeline component with defined inputs, validation gates, retry logic, and audit logging. The harness receives raw scores from N judges across M evaluation items, invokes the normalization prompt, validates the output schema, and writes the normalized scores and per-judge mapping functions to a downstream evaluation store. Without this harness, normalization results are unreproducible and untrustworthy for production score aggregation.

The harness should enforce several concrete checks before accepting the prompt output. First, validate that the output contains a normalized_scores array with the same dimensions as the input matrix (N judges × M items). Second, verify that each per-judge mapping function preserves rank ordering—if judge A scored item X higher than item Y in the raw scores, the normalized scores must maintain that ordering. Third, check that normalized scores fall within the target range (e.g., 0.0–1.0 or 1–5) and that no judge's mapping introduces score collapse (all items receiving the same normalized value). Fourth, compute the inter-judge agreement on normalized scores and flag cases where normalization increased disagreement rather than reduced it. If any validation fails, retry with the error details injected into the prompt's [CONSTRAINTS] or [ERROR_CONTEXT] placeholder, up to a configured maximum of 3 attempts before escalating for human review.

Logging and traceability are critical because normalization decisions affect every downstream score comparison. The harness should log the raw input scores, the full prompt sent to the model, the model response, validation results, retry count, and the final normalized output. Store these artifacts keyed by evaluation_batch_id and normalization_run_id so that score disputes can be traced back to the exact normalization step. For model choice, prefer a model with strong numeric reasoning and instruction-following (GPT-4o or Claude 3.5 Sonnet are reasonable defaults). Set temperature to 0 or near-zero to maximize reproducibility. If the normalization prompt uses few-shot examples, pin them to a versioned example store rather than embedding them inline, so calibration examples can be updated without changing the prompt template. Avoid running normalization on batches larger than 50 items per judge without splitting—large matrices increase the risk of the model dropping rows or producing truncated JSON. For high-stakes evaluation pipelines where normalized scores gate model releases or product decisions, add a human review step that spot-checks a stratified sample of normalized scores against the raw inputs before the normalized data enters the reporting layer.