Judge Alignment Report Generation Prompt Template

This prompt is for evaluation leads and MLOps engineers who need to communicate LLM judge calibration results to stakeholders. It takes raw alignment metrics, drift data, and judge health scores as input and produces a structured report with an executive summary, per-dimension breakdowns, and a technical appendix. Use this when you have completed a calibration run and need to produce a consistent, auditable artifact for product managers, engineering directors, or compliance reviewers. The prompt does not perform the calibration itself—it assumes calibration data already exists and focuses entirely on report generation.

Before using this prompt, you must have completed a calibration run and have access to the following inputs: per-judge alignment scores against human ratings, drift detection results from a monitoring window, judge health metrics (latency, error rates, agreement coefficients), and any threshold configurations used during evaluation. The prompt expects structured data, not raw log files. If your calibration data is incomplete, unvalidated, or still in progress, do not use this prompt—it will produce a report that looks authoritative but rests on shaky foundations. Instead, complete the calibration pipeline first, validate the outputs, and then feed the verified results into this report generation step.

This prompt is not a replacement for a calibration dashboard, a real-time monitoring system, or a human review of judge behavior. It produces a point-in-time narrative artifact suitable for weekly stakeholder reviews, compliance audits, or release gate documentation. If you need a live health dashboard, use the Evaluation Judge Pool Health Dashboard Data Prompt instead and pipe its structured output into your observability stack. If you need to debug why a specific judge is misaligned, use the Score Discrepancy Root Cause Analysis Prompt before generating this summary report. The report generation prompt is the last mile—it turns verified calibration evidence into a decision-ready document, but it cannot fix bad data or replace the engineering work that produces trustworthy calibration results.

This prompt template is designed to be the core instruction you send to a model for producing a structured Judge Alignment Report. It accepts calibration metrics, drift status, and judge health scores as inputs and produces an executive summary with a technical appendix. The template uses square-bracket placeholders for all variable inputs, constraints, and output schema requirements. Before using this in production, replace each placeholder with the concrete data, rules, and format specifications from your evaluation pipeline. The prompt is self-contained and can be tested independently before being wired into an application harness.

textCopy
You are an expert evaluation analyst producing a Judge Alignment Report for stakeholders who need to trust automated evaluation scores.

Your task is to generate a structured report from the provided calibration data. The report must communicate alignment quality, drift status, and recommended actions clearly, without overstating confidence.

## INPUT DATA
[JUDGE_CALIBRATION_DATA]

## REPORT REQUIREMENTS
Generate a report with exactly two sections:

### 1. Executive Summary
- A one-paragraph summary of overall alignment health.
- A bulleted list of key metrics: overall agreement rate, number of aligned dimensions, number of drifted dimensions, and a judge health status indicator (Healthy / Warning / Critical).
- A "Recommended Actions" subsection with 1-3 prioritized, actionable items. If alignment is healthy, recommend continued monitoring cadence.

### 2. Technical Appendix
- A table with one row per evaluation dimension containing: Dimension Name, Agreement Score (0-1), Drift Status (Stable / Minor Drift / Significant Drift), Human-Judge Correlation, and Sample Count.
- A "Raw Agreement Data" section showing the confusion matrix or contingency table for the overall judge-human alignment.
- A "Drift Analysis" subsection. If any dimension shows drift, include the detected drift magnitude, the time window where drift was detected, and a hypothesized root cause from the following list: [Concept Drift, Prompt Sensitivity, Data Distribution Shift, Human Rater Shift, Unknown].

## CONSTRAINTS
[REPORT_CONSTRAINTS]

## OUTPUT FORMAT
[OUTPUT_SCHEMA]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, start by replacing [JUDGE_CALIBRATION_DATA] with the structured output from your calibration pipeline—this should include per-dimension agreement scores, correlation coefficients, sample counts, and any drift detection results. The [REPORT_CONSTRAINTS] placeholder should be replaced with specific rules, such as 'Do not recommend retraining without at least two consecutive weeks of Significant Drift' or 'Round all percentages to one decimal place.' For [OUTPUT_SCHEMA], provide the exact JSON schema or Markdown structure you expect, ensuring downstream parsers can consume the report without manual reformatting. The [RISK_LEVEL] placeholder should be set to 'Low', 'Medium', 'High', or 'Critical' based on the decision impact of the scores being reported on. If the risk level is High or Critical, the prompt should be followed by a human review step before the report is distributed. Always validate the generated report against your schema and spot-check agreement metrics against the raw input data to catch hallucinated statistics.

This prompt is designed to be called programmatically after a calibration run or drift detection job completes. The typical harness receives structured alignment metrics, per-judge statistics, and raw agreement data as inputs, then invokes the LLM to produce a stakeholder-ready report. Because the output is consumed by evaluation leads and engineering managers, the harness must enforce output schema compliance, validate metric accuracy, and gate the report behind a human review step before distribution.

Wire the prompt into a reporting service that aggregates data from your evaluation store or calibration database. The service should assemble the [ALIGNMENT_METRICS], [JUDGE_HEALTH_SCORES], [DRIFT_STATUS], [PER_DIMENSION_BREAKDOWN], and [RAW_AGREEMENT_DATA] placeholders from structured sources—never from unstructured logs. Before calling the model, validate that all required fields are present and that numeric values fall within expected ranges (e.g., agreement coefficients between 0 and 1, drift p-values between 0 and 1). After receiving the model response, run a schema validator against the expected output structure: the executive summary must contain a verdict field, the technical appendix must include per-dimension arrays with dimension_name, agreement_score, and drift_flag fields, and recommended actions must map to specific dimensions or judges. If validation fails, retry once with the validation errors appended to the [CONSTRAINTS] placeholder. If the second attempt also fails, log the failure and escalate to a human report writer rather than silently shipping a malformed report.

For high-stakes calibration reports—such as those used to approve judge fleet changes or justify evaluation budget—add a human approval step before the report reaches stakeholders. Route the generated report to a review queue where an evaluation lead can verify that the alignment metrics match the source data, that the drift interpretation is reasonable, and that recommended actions are actionable. Log every report generation event with the input metrics, model version, output report, validation results, and reviewer decision. This audit trail is essential when stakeholders question why a judge was recalibrated or when drift went undetected. Choose a model with strong instruction-following and structured output capabilities for this task; avoid models known to hallucinate numeric values or reorder structured data. If your evaluation pipeline runs on a schedule, cache the prompt prefix containing the report structure and formatting instructions to reduce latency and cost on repeated runs.