Rubric with Justification Requirement Prompt Template

This prompt is for evaluation engineers, compliance leads, and AI audit teams who need automated model-graded scores that are explainable, traceable, and defensible. The core job-to-be-done is converting a subjective quality assessment into a structured, evidence-backed judgment that a human reviewer or auditor can quickly verify. Unlike a standard rubric that outputs only a numeric score, this prompt forces the LLM judge to produce a justification, cite specific evidence from the evaluated output, and flag its own uncertainty. This is essential when scores feed into regulated workflows, customer-facing quality guarantees, or internal governance reports where a naked number is insufficient.

Use this prompt when the cost of an unexplained score is high—for example, when grading AI outputs for compliance with legal or safety policies, when scoring candidate responses in a high-stakes evaluation pipeline, or when building a dataset of graded examples that will be audited later. The prompt is designed to be wired into an automated evaluation harness where each justification is logged alongside the score, enabling downstream spot-checks, inter-rater reliability analysis, and audit trail construction. It is not a lightweight replacement for a simple pass/fail check; it adds tokens, latency, and complexity that are only justified when explainability is a hard requirement.

Do not use this prompt for high-throughput, low-risk scoring where speed and cost are the primary constraints. If you are grading thousands of short-form outputs where a numeric score is sufficient and human review is unlikely, a standard rubric without justification will be faster and cheaper. Similarly, avoid this prompt when the evaluation criteria are purely mechanical and can be checked with deterministic code—a regex or schema validator is more reliable than an LLM judge for format compliance. Finally, if the justification itself could leak sensitive information from the evaluated output, ensure your logging and storage practices redact PII before persisting the judge's response.

Before implementing, you must define the evaluation dimensions, the scoring scale, and the expected structure of the justification. The prompt template expects you to provide these as inputs. You should also prepare a small set of 5-10 human-annotated examples that demonstrate the desired relationship between scores, justifications, and evidence. These examples will be used to calibrate the judge and to build a regression test suite that gates any future changes to the rubric. The next section provides the copy-ready template you will adapt with your specific criteria.

This prompt template is the core contract for an explainable, audit-grade LLM judge. It forces the model to produce not just a score, but a structured justification, a direct citation to the source material, and an explicit uncertainty flag. This design is critical for compliance, audit, and debugging workflows where a numeric score alone is a black box. The template uses square-bracket placeholders for all dynamic inputs, making it safe to copy directly into your application code or a prompt management system. Adapt the [RUBRIC_DIMENSIONS], [OUTPUT_SCHEMA], and [CONSTRAINTS] to match your specific evaluation task.

textCopy
You are an expert evaluator. Your task is to score a piece of AI-generated content against a provided rubric. You must produce a structured output that includes your score, a detailed justification, a direct citation from the source material, and an uncertainty flag.

## EVALUATION INPUT
[INPUT_TEXT]

## EVALUATION CONTEXT
[SOURCE_MATERIAL]

## RUBRIC DIMENSIONS
[RUBRIC_DIMENSIONS]

## SCORING SCALE
[SCORING_SCALE]

## OUTPUT SCHEMA
You must respond with a single JSON object that strictly follows this schema:
{
  "score": <number>,
  "justification": "<string explaining the score, referencing specific rubric criteria and evidence>",
  "evidence_citation": "<a direct, verbatim quote from the SOURCE_MATERIAL that supports the justification>",
  "uncertainty_flag": "<one of: 'none', 'low', 'high'>",
  "uncertainty_reason": "<explain why the flag was set, e.g., ambiguous source, conflicting evidence, edge case>"
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## INSTRUCTIONS
1. Read the INPUT_TEXT and SOURCE_MATERIAL carefully.
2. Apply each RUBRIC_DIMENSION to the INPUT_TEXT.
3. Determine a score based on the SCORING_SCALE.
4. Write a justification that explicitly connects the score to the rubric criteria and the evidence.
5. Extract a direct, verbatim quote from the SOURCE_MATERIAL as your evidence_citation. If no single quote is sufficient, use the most relevant one and note the limitation in the uncertainty_reason.
6. Set the uncertainty_flag to 'high' if the evidence is contradictory, missing, or highly ambiguous. Set it to 'low' if there is minor ambiguity. Set it to 'none' only if the evidence is clear and directly supports the score.
7. Output only the JSON object. Do not include any other text.

To adapt this template, start by defining your [RUBRIC_DIMENSIONS] and [SCORING_SCALE] with precise, non-overlapping language. The [CONSTRAINTS] field is where you should inject rules like 'If the source material is empty, set the uncertainty_flag to high and score as 0.' The [EXAMPLES] block is not optional for high-stakes use; provide at least one few-shot example for each uncertainty level to calibrate the judge's behavior. When wiring this into an application, validate the output JSON against the schema before accepting the score. A common failure mode is the model providing a paraphrased justification in the evidence_citation field instead of a verbatim quote; a post-processing check for exact substring matches in the source material can catch this and trigger a retry.

This prompt is designed to be called by an LLM judge service that evaluates another model's output. The judge receives the original input, the candidate output, and the rubric with justification requirements. The prompt template should be treated as a function: evaluate(input, output, rubric) -> {score, justification, evidence, uncertainty}. The calling application must supply all square-bracket placeholders before sending the request. Do not use this prompt for real-time user-facing scoring where latency is critical—justification generation adds tokens and time. Reserve it for offline evaluation, regression testing, audit sampling, and compliance review workflows where explainability is required.

Wiring the prompt into an application requires a structured output contract. Configure your model API call with JSON mode or a strict function-calling schema that enforces the output shape: score (numeric or categorical), justification (string), evidence (array of strings with source citations), and uncertainty (object with level and reason fields). On the application side, validate the response before accepting it: check that score falls within the defined scale, that justification is non-empty and references specific rubric dimensions, that evidence items point to actual source spans, and that uncertainty.level matches a predefined enum (e.g., low, medium, high). If validation fails, retry once with the validation errors injected into the prompt as additional context. After two failures, log the raw response and escalate for human review rather than silently accepting a malformed score.

Model choice matters. Use a model with strong instruction-following and reasoning capabilities for the judge role—frontier models like GPT-4, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Smaller or faster models often produce justifications that sound plausible but fail evidence-matching checks. Temperature should be set low (0.0–0.2) to reduce variance in scoring and justification quality. If your evaluation pipeline processes high volumes, consider batching evaluations and using a separate async worker queue. Store every evaluation result—including the prompt version, model version, timestamp, raw response, and validation outcome—in an audit log for downstream review. This is especially important for compliance use cases where score justifications may be challenged or reviewed by auditors.

Retry and failure handling must distinguish between format failures and content failures. A format failure (missing JSON field, wrong type) can be retried with stricter schema instructions. A content failure (justification contradicts the score, evidence is fabricated) should not be retried—it indicates the judge model is not reliable for that evaluation task and may require rubric simplification, model upgrade, or human fallback. Implement a justification quality check as a secondary validation step: use a lightweight classifier or regex-based check to verify that the justification text references the specific rubric dimensions and evidence items it claims to use. If the justification is generic or circular (e.g., 'scored 4 because it deserves a 4'), flag the evaluation for human review.

Integration with existing eval frameworks is straightforward. If you use Braintrust, LangSmith, or Arize, wrap this prompt as a custom scorer that returns the structured JSON. The score field maps to your metric, while justification and evidence fields become metadata for trace inspection. For CI/CD pipelines, gate deployment on a minimum percentage of evaluations passing both score thresholds AND justification quality checks. A passing score with a hallucinated justification is a silent failure that undermines the audit trail. Finally, periodically sample and manually review a subset of justifications against the original outputs to detect judge drift—LLM judges can develop systematic biases over time, and justification requirements help surface those biases faster than numeric scores alone.