Qualitative Rubric Gate Prompt for LLM Outputs

This prompt is for QA leads and platform engineers who need a release gate that aligns with human judgment, not just aggregate statistics. Use it when a numeric score—like an average helpfulness rating or a simple pass/fail percentage—fails to capture the nuance of a failing output. For example, a prompt version might have a 95% pass rate on a golden dataset but still produce subtly unsafe or confusing responses in 2% of cases that a numeric gate would miss. This rubric gate forces a structured, dimension-level evaluation (clarity, safety, task completion) with calibrated severity ratings (e.g., 'Minor,' 'Major,' 'Critical'), producing a machine-readable artifact that can block a release or flag it for human review.

The ideal user is someone responsible for the final quality sign-off before a prompt moves from staging to production. They need more than a binary pass/fail signal; they need an auditable justification for why a version was promoted or rejected. The required context includes the raw LLM output to be evaluated, the original input and any retrieved context, the system prompt or behavioral contract, and a predefined rubric defining the dimensions and severity levels. This prompt is not a replacement for quantitative regression tests or golden dataset evaluations. Instead, it acts as a complementary, human-aligned layer that catches failures those methods miss, such as outputs that are factually correct but violate a tone policy or fail to complete a critical sub-task.

Do not use this prompt as a real-time guardrail in a high-throughput production path. It is a gate artifact, designed for asynchronous, pre-release evaluation where latency is less critical than depth. It is also unsuitable as the sole gate for prompts where factual accuracy is the only dimension that matters; in those cases, a dedicated hallucination rate gate with source grounding is more appropriate. Before implementing, ensure you have a mechanism to route 'Critical' findings to a human reviewer and a process for calibrating the LLM judge against your human QA team's decisions to prevent drift in the rubric's interpretation over time.

This prompt template turns a qualitative rubric into an executable evaluation. It instructs an LLM judge to assess a target output across multiple dimensions—such as clarity, safety, task completion, and tone—and return a structured verdict with per-dimension scores, severity ratings, and specific evidence from the output. The template is designed to be copied directly into your evaluation harness, with square-bracket placeholders you replace before each run. Do not ship this prompt without first calibrating it against human judgments on a sample of at least 20–30 outputs to measure inter-rater reliability.

textCopy
You are an expert output quality evaluator. Your job is to assess a target LLM output against a qualitative rubric and return a structured evaluation. You must be precise, evidence-based, and consistent.

## RUBRIC DIMENSIONS
[RUBRIC_DIMENSIONS]
<!-- Format each dimension as:
- Dimension Name: [description of what good looks like]
- Severity Levels: Critical | Major | Minor | None
- Scoring Guide: 1 (unacceptable) to 5 (exemplary), with anchors for each level
-->

## TARGET OUTPUT TO EVALUATE
[TARGET_OUTPUT]

## CONTEXT (OPTIONAL)
[CONTEXT]
<!-- Include user prompt, system instructions, source documents, or expected behavior -->

## EVALUATION INSTRUCTIONS
1. For each rubric dimension, assign a score from 1 to 5 based on the scoring guide.
2. For each dimension, assign a severity rating: Critical, Major, Minor, or None.
3. For any score below 4, cite specific excerpts from the target output that justify the score.
4. If the target output is empty, malformed, or a refusal, score all dimensions as 1 with severity Critical and explain why.
5. Do not evaluate dimensions not listed in the rubric.
6. Do not speculate about intent. Evaluate only what is present in the target output.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "overall_pass": boolean,
  "overall_score": number,
  "dimensions": [
    {
      "name": "string",
      "score": number,
      "severity": "Critical" | "Major" | "Minor" | "None",
      "evidence": ["string"],
      "explanation": "string"
    }
  ],
  "critical_failures": ["string"],
  "evaluation_notes": "string"
}

## PASS/FAIL RULES
- overall_pass is false if any dimension has severity Critical.
- overall_pass is false if any dimension has score below [MINIMUM_DIMENSION_SCORE].
- overall_pass is false if overall_score is below [MINIMUM_OVERALL_SCORE].
- Otherwise overall_pass is true.

## CONSTRAINTS
[CONSTRAINTS]
<!-- Example: "Do not penalize brevity if the output is complete." or "Treat missing citations as Critical severity." -->

After copying the template, replace each placeholder before running the evaluation. [RUBRIC_DIMENSIONS] is the most important: define 3–7 dimensions with clear scoring anchors. For a support-ticket quality gate, dimensions might include Resolution Completeness, Tone Appropriateness, Policy Compliance, and Actionability. [MINIMUM_DIMENSION_SCORE] and [MINIMUM_OVERALL_SCORE] set your pass/fail thresholds—start with 3 and 3.5 respectively and tighten based on calibration data. [CONSTRAINTS] lets you inject domain-specific rules, such as "Outputs that suggest off-label medical use are Critical severity." Always run this judge against a golden set of human-scored examples before trusting it in a release gate. If the judge's scores diverge from human raters by more than one point on any dimension, revise the rubric anchors or add few-shot examples to the [RUBRIC_DIMENSIONS] section. For high-stakes domains like healthcare or finance, route any output flagged with Critical severity to human review regardless of the automated pass/fail result.

The qualitative rubric gate prompt is designed to operate as a post-generation evaluation step within a CI/CD or release pipeline, not as a one-off manual review tool. After a candidate prompt produces outputs on a golden dataset or canary traffic, each output is sent to this rubric prompt alongside the original input, the evaluation dimensions, and the severity calibration. The harness is responsible for collecting these rubric scores, aggregating them across the evaluation set, and comparing the results against predefined promotion thresholds. This means the prompt itself must be treated as a deterministic evaluation function: same input, same rubric, same model, same temperature (typically 0), and same output schema every time.

Integration pattern: The harness should call the rubric prompt for each [INPUT, OUTPUT] pair in the evaluation batch. The prompt expects [RUBRIC_DIMENSIONS] (e.g., clarity, safety, task_completion), [SEVERITY_LEVELS] (e.g., critical, major, minor, none), and [CALIBRATION_EXAMPLES] to anchor the judge. The output must conform to a strict JSON schema with per-dimension ratings and a structured justification. Implement a schema validator that rejects any response missing required fields, containing unknown severity values, or producing malformed JSON. On validation failure, retry once with the same input and an explicit error message appended to the prompt context. If the second attempt also fails, log the failure, mark that evaluation item as EVAL_ERROR, and alert the release manager—do not silently skip or impute scores.

Human review integration: For high-risk domains or when rubric scores fall into a borderline band (e.g., any critical finding or aggregate scores within 5% of the pass/fail threshold), the harness should route the input, output, and rubric result to a manual review queue. Store the full evaluation trace—including the prompt version, model version, timestamp, raw response, and validator results—in an immutable audit log. This trace becomes evidence for release readiness decisions and inter-rater reliability monitoring. Avoid using this rubric prompt as the sole gate for safety-critical outputs; pair it with quantitative metrics (e.g., hallucination rate, format compliance) and human spot-checking for a defense-in-depth approach.

Model selection and cost control: Use a model with strong instruction-following and low output variability for the judge role—frontier models at temperature 0 are the default choice. If cost or latency is a concern, consider routing low-risk evaluation dimensions to a smaller, fine-tuned judge model while reserving the full rubric prompt for high-severity dimensions or borderline cases. Cache rubric results by [INPUT_HASH, OUTPUT_HASH, RUBRIC_VERSION] to avoid re-evaluating identical pairs across repeated test runs. Finally, version-lock the rubric prompt itself: a change to the rubric dimensions or severity definitions is a breaking change that requires re-baselining all historical scores and re-validating inter-rater agreement before the new rubric can gate production releases.