Calibration Feedback Loop Prompt for Continuous Alignment

This prompt is for evaluation platform teams running LLM judges in production who need a systematic way to close the loop between detected misalignments and judge improvement. Use it when you have a batch of recent evaluation records where the LLM judge's scores diverged from human ratings or reference standards, and you need to extract actionable feedback rather than just logging the discrepancy. The prompt produces three outputs: a structured extraction of what the judge got wrong and why, a set of prioritized instruction updates targeting the root causes, and a retraining recommendation that balances fixing recent errors against preserving general calibration.

This is not a prompt for initial judge calibration or one-off score comparison. It assumes you already have a calibration baseline, a rubric, and a stream of scored examples with ground truth or human labels. The prompt belongs in a continuous alignment pipeline where judge performance is monitored, drift is detected, and feedback is fed back into instruction refinement or fine-tuning data selection. You should have at least 20-50 recent misaligned examples before running this prompt; smaller batches produce noisy root cause attributions and brittle instruction updates that overfit to individual cases rather than patterns.

Do not use this prompt when you are building a judge from scratch, comparing two judges for the first time, or debugging a single anomalous score. Those workflows need calibration set construction, pairwise comparison, or root cause analysis prompts instead. This prompt assumes the misalignment pattern is already confirmed and the team is ready to act on it. Before running it, verify that your ground truth labels are reliable—feeding this prompt noisy human ratings will produce instruction updates that chase label error rather than judge error. If you cannot confirm label quality, run a calibration data quality audit first.

This prompt is the engine of a continuous alignment loop. You feed it a batch of examples where your LLM judge's scores diverged from human ratings, along with the current judge instructions and evaluation rubric. The model returns a structured JSON payload containing extracted misalignment patterns, concrete instruction rewrites ranked by impact, and a data-driven recommendation on whether to update instructions, recalibrate thresholds, or expand the calibration set. The prompt is designed for a single LLM call with no tool use, making it suitable for scheduled batch jobs, CI hooks, or evaluation pipeline automation.

textCopy
You are a calibration analyst for an LLM-based evaluation system. Your job is to diagnose why an automated judge's scores diverged from human ratings and produce actionable corrections.

## INPUTS

### Current Judge Instructions
[JUDGE_INSTRUCTIONS]

### Evaluation Rubric
[RUBRIC]

### Misalignment Examples
A batch of examples where the LLM judge score differs from the human reference score by at least [MIN_SCORE_GAP] points. Each example includes the item being evaluated, the judge's score and justification, and the human reference score and justification.

[MISALIGNMENT_EXAMPLES]

### Calibration Window Context
Number of examples in this batch: [BATCH_SIZE]
Overall judge-human agreement rate in recent window: [AGREEMENT_RATE]
Previous calibration actions taken (if any): [PREVIOUS_ACTIONS]

## OUTPUT SCHEMA

Return valid JSON matching this structure:

{
  "misalignment_patterns": [
    {
      "pattern_name": "string (short label)",
      "description": "string (what the judge consistently does wrong)",
      "affected_dimensions": ["string (rubric dimension names)"],
      "example_count": "integer (how many misalignments exhibit this pattern)",
      "severity": "high | medium | low",
      "direction": "over_score | under_score | inconsistent"
    }
  ],
  "instruction_updates": [
    {
      "priority": "integer (1 = highest)",
      "target_pattern": "string (must match a pattern_name above)",
      "current_wording": "string (excerpt from judge instructions causing the issue)",
      "proposed_wording": "string (replacement text)",
      "rationale": "string (why this change addresses the pattern)",
      "regression_risk": "high | medium | low",
      "affected_test_cases": ["string (IDs or descriptions of cases to re-test)"]
    }
  ],
  "threshold_adjustments": [
    {
      "dimension": "string (rubric dimension name)",
      "current_boundary": "string (current pass/fail or score boundary)",
      "proposed_boundary": "string (adjusted boundary)",
      "rationale": "string"
    }
  ],
  "calibration_set_recommendation": {
    "action": "add_examples | remove_examples | re_label | no_change",
    "rationale": "string",
    "target_score_ranges": ["string (underrepresented score buckets)"],
    "suggested_example_count": "integer"
  },
  "overall_recommendation": {
    "primary_action": "update_instructions | recalibrate_thresholds | expand_calibration_set | investigate_human_labels | no_action",
    "confidence": "high | medium | low",
    "summary": "string (one-paragraph explanation for the evaluation team)",
    "risk_of_overfitting": "high | medium | low",
    "overfitting_rationale": "string (why this batch might not generalize)"
  }
}

## CONSTRAINTS

1. Only propose instruction updates that are directly supported by patterns observed in the misalignment examples. Do not invent patterns.
2. If fewer than [MIN_EXAMPLES_FOR_ACTION] examples exhibit a pattern, set severity to "low" and deprioritize corresponding updates.
3. Flag any proposed instruction change that could cause regression on previously well-calibrated dimensions. Set regression_risk accordingly and list specific test cases to re-evaluate.
4. If the misalignment batch is small (fewer than [SMALL_BATCH_THRESHOLD] examples) or covers a narrow range of inputs, set risk_of_overfitting to "high" and recommend caution.
5. When human labels themselves show inconsistency or possible error, recommend investigate_human_labels before changing judge instructions.
6. Do not propose threshold changes unless the pattern shows systematic bias in one direction across multiple examples.
7. Preserve the judge's ability to handle edge cases. Do not optimize only for the examples in this batch.

## OUTPUT

Return only the JSON object. No preamble, no commentary.

To adapt this prompt for your pipeline, replace the square-bracket placeholders with real data before each call. [JUDGE_INSTRUCTIONS] should contain the full system prompt or scoring instructions your LLM judge currently uses. [RUBRIC] is the evaluation rubric with dimension definitions and scoring scale. [MISALIGNMENT_EXAMPLES] is the critical input: format each example consistently with the item text, the judge's score and justification, and the human reference score and justification. Set [MIN_SCORE_GAP] to filter out trivial disagreements that don't warrant instruction changes—typically 1.5–2 points on a 5-point scale. Set [MIN_EXAMPLES_FOR_ACTION] to at least 3 to avoid reacting to noise, and [SMALL_BATCH_THRESHOLD] to 10–15 examples below which overfitting risk is elevated. The [AGREEMENT_RATE] and [PREVIOUS_ACTIONS] fields provide context that helps the model avoid repeating failed interventions.

Before deploying this prompt into an automated calibration loop, validate the output JSON against the schema and run a human review step on the proposed instruction updates. A judge that silently rewrites its own instructions based on a narrow batch can drift catastrophically. Log every calibration run's inputs, outputs, and the human accept/reject decision for each proposed change. This audit trail is essential when stakeholders ask why scores shifted. For high-stakes evaluation pipelines, gate instruction updates behind a manual approval step and re-run the calibration set after any change to confirm alignment improved without regressions.

The calibration feedback loop prompt is not a one-shot generation task—it is a diagnostic step inside a continuous alignment pipeline. Wire it to run on a schedule (daily or per-evaluation-batch) whenever new human-annotated samples or judge-scored outputs accumulate. The prompt expects a structured [MISALIGNMENT_LOG] containing recent discrepancies between human ratings and LLM judge scores, plus the current [JUDGE_INSTRUCTIONS] that produced those scores. The output is a structured [FEEDBACK_PACKET] with prioritized instruction updates, retraining recommendations, and a severity assessment. Because this prompt directly proposes changes to your evaluation infrastructure, treat its output as a recommendation, not an automated commit.

Implement a validation layer that parses the [FEEDBACK_PACKET] before any action is taken. At minimum, confirm that each proposed instruction change includes a rationale tied to a specific misalignment pattern from the input log. Reject changes that lack evidence grounding or that propose sweeping rewrites without traceable justification. Add a retry wrapper: if the model returns malformed JSON, missing required fields, or changes that contradict the input evidence, retry once with the validation errors appended as [CORRECTION_HINTS]. If the second attempt also fails, escalate to a human reviewer queue with the raw output and validation log. Log every run—input hash, output hash, validation status, retry count, and reviewer decision—so you can audit the feedback loop's own behavior over time.

Include a human review gate before any instruction update reaches production. Even when validation passes, route the proposed changes to a review interface that shows the current instructions, the proposed diff, the misalignment evidence that triggered the change, and a preview of how the updated judge would score a holdout set of recent examples. The reviewer should confirm that the change addresses real drift rather than overfitting to a noisy batch. Set a severity threshold: if the prompt's output indicates high-severity drift or recommends retraining, require two reviewers. Never auto-apply instruction changes from this prompt without human sign-off—the calibration feedback loop is a diagnostic tool, not an auto-healing mechanism. After approval, version the judge instructions, record the approval metadata, and run a regression test against your golden evaluation set before promoting the change.