Calibration Data Quality Audit Prompt for Judge Training Sets

Use this prompt when you are preparing to calibrate an LLM judge against human ratings and need to verify that your calibration dataset is trustworthy. The primary job is to produce a structured quality report that surfaces label noise, ambiguous examples, underrepresented score ranges, and potential human rater biases before those flaws get baked into your judge's scoring behavior. The ideal user is an evaluation engineer, MLOps lead, or AI quality architect who owns the judge alignment pipeline and needs evidence that the training set won't produce a miscalibrated or biased judge. You should have a dataset of scored examples with human-assigned labels, access to the original source material for each example, and a clear understanding of the scoring rubric that human raters were supposed to follow.

This prompt is not a replacement for statistical analysis of your dataset. It is a qualitative audit that complements distribution checks, agreement coefficients, and coverage metrics you should already be running. Do not use this prompt when you need a single numeric calibration score, when you are actively calibrating a judge rather than auditing the dataset, or when your dataset is too large to fit in a single context window without sampling. For datasets exceeding roughly 50-100 examples, sample strategically across score ranges, difficulty tiers, and rater cohorts rather than truncating. The prompt works best when you provide the full rubric, representative examples from every score bucket, and metadata about which human rater produced each label.

Before running this prompt, confirm that you have: (1) the scoring rubric with clear dimension definitions and scale anchors, (2) a stratified sample of scored examples with human labels and rater identifiers, (3) the original input context that human raters saw, and (4) any known edge cases or disagreement zones you want the audit to probe. The output is a structured report, not a pass/fail gate. Expect findings that require human judgment to resolve—such as rubric clarifications, rater retraining, or example removal. Wire the output into a review step where an evaluation lead can accept, reject, or modify each remediation suggestion before the dataset proceeds to judge calibration.

This prompt template performs a structured quality audit on a calibration dataset used to align LLM judges against human ratings. It expects a set of examples with human labels, judge scores, and optional metadata, then produces a diagnostic report covering label noise, ambiguous examples, underrepresented score ranges, and potential rater biases. Use this before running judge alignment workflows to catch data problems that would otherwise produce miscalibrated judges.

textCopy
You are a calibration data quality auditor. Your task is to analyze a judge training dataset and produce a structured quality report.

## INPUT DATA
[INPUT_DATA]

## AUDIT DIMENSIONS
Analyze the dataset across these dimensions:

1. **Label Noise Detection**: Identify examples where human labels appear inconsistent, contradictory, or likely erroneous given the input context. Flag examples where multiple human raters disagree beyond an acceptable threshold.

2. **Ambiguity Assessment**: Find examples where the input context is too vague, underspecified, or open to multiple reasonable interpretations. These examples cannot produce reliable ground truth.

3. **Score Distribution Analysis**: Check whether all score ranges in [SCORE_RANGE] are adequately represented. Flag underrepresented ranges and extreme class imbalance that would bias judge training.

4. **Rater Bias Patterns**: Detect systematic tendencies in human labels, such as leniency bias, central tendency bias, halo effects, or demographic skew. Note any rater whose labels consistently deviate from peers.

5. **Coverage Gaps**: Identify missing combinations of difficulty, topic, input length, or other relevant dimensions specified in [COVERAGE_DIMENSIONS].

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "dataset_summary": {
    "total_examples": <int>,
    "score_distribution": {<score>: <count>},
    "num_human_raters": <int>,
    "inter_rater_agreement": <float 0-1>
  },
  "label_noise": [
    {
      "example_id": <string>,
      "severity": "high" | "medium" | "low",
      "issue": <description>,
      "evidence": <specific data from example>,
      "recommendation": "remove" | "re-label" | "review"
    }
  ],
  "ambiguous_examples": [
    {
      "example_id": <string>,
      "ambiguity_type": <string>,
      "description": <why it's ambiguous>,
      "recommendation": "clarify" | "remove" | "split"
    }
  ],
  "distribution_issues": [
    {
      "score_range": <string>,
      "current_count": <int>,
      "recommended_minimum": <int>,
      "severity": "critical" | "warning"
    }
  ],
  "rater_biases": [
    {
      "rater_id": <string>,
      "bias_type": <string>,
      "evidence": <statistical pattern>,
      "affected_examples": <int>,
      "recommendation": <string>
    }
  ],
  "coverage_gaps": [
    {
      "dimension": <string>,
      "missing_values": [<string>],
      "impact": <description>
    }
  ],
  "overall_quality": {
    "grade": "ready" | "needs_work" | "unusable",
    "critical_issues": <int>,
    "summary": <one-paragraph assessment>
  },
  "remediation_plan": [
    {
      "priority": <int 1-highest>,
      "action": <string>,
      "affected_examples": [<string>],
      "expected_improvement": <description>
    }
  ]
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## INSTRUCTIONS
- Flag examples conservatively. Only mark as noisy or ambiguous when evidence is clear.
- For distribution issues, recommend minimum counts based on the score granularity and expected model variance.
- When detecting rater bias, require statistical evidence, not single-instance patterns.
- If the dataset is too small for reliable statistical analysis, note this limitation explicitly.
- Prioritize remediation actions by impact: fix label noise before addressing coverage gaps.

Adaptation notes: Replace [INPUT_DATA] with your calibration examples in a structured format—typically JSON with fields for example_id, input_context, human_label, judge_score, and rater_id. Set [SCORE_RANGE] to your scoring scale, such as 1-5 or 0-100. Use [COVERAGE_DIMENSIONS] to specify what axes matter for your use case, such as ["difficulty", "topic", "input_length"]. The [CONSTRAINTS] placeholder lets you add domain-specific rules, such as minimum inter-rater agreement thresholds or required sample sizes per score bucket. [EXAMPLES] should contain one or two annotated examples showing correct audit output for your domain. If your dataset exceeds context window limits, split it into batches by score range or topic and run the audit per batch, then aggregate results in a separate pass.

The Calibration Data Quality Audit Prompt is designed to run as a batch evaluation step before judge alignment or training workflows begin. It should be wired into your pipeline as a pre-flight check that gates downstream calibration work. The prompt expects a structured calibration dataset as input and produces a quality report with remediation suggestions. Because this audit directly impacts the trustworthiness of your LLM judges, the implementation must include strict validation of both inputs and outputs, and the results should be reviewed by a human before any dataset corrections are applied automatically.

To implement this in production, wrap the prompt in a function that accepts a JSON array of calibration examples, each containing [INPUT], [REFERENCE_OUTPUT], [HUMAN_SCORE], and optional [RUBRIC_DIMENSION] fields. Before calling the model, validate that the input array is non-empty, that score distributions are not catastrophically imbalanced, and that required fields are present. After receiving the model response, parse the output against a strict schema that includes quality_flags, label_noise_instances, underrepresented_ranges, bias_concerns, and remediation_suggestions. If the model returns malformed JSON, implement a retry loop with exponential backoff (max 3 attempts) using a repair prompt that includes the original output and the expected schema. Log every audit run with a unique audit_id, the input dataset hash, the model version used, and the full output for traceability. For high-stakes evaluation pipelines, route audit results to a human review queue before any automated dataset modifications are applied—this is not a fire-and-forget step.

Model choice matters here. Use a model with strong reasoning capabilities and a large context window, as the prompt must process many examples simultaneously to detect distribution-level patterns. GPT-4o or Claude 3.5 Sonnet are appropriate defaults. Avoid using the same model family for auditing that you use for your production judges—this reduces the risk of shared blind spots. If your calibration dataset exceeds the model's context window, implement a chunked audit strategy: split the dataset into overlapping strata, run the prompt on each chunk, and then run a synthesis pass that merges findings and detects cross-chunk patterns. Store all intermediate results. For evaluation teams running continuous calibration monitoring, schedule this audit on a cadence (weekly or per-dataset-update) and compare audit reports over time to detect degradation in dataset quality before it corrupts judge alignment.