Rubric with Example Outputs and Scores Prompt Template

This prompt is for evaluation engineers and AI platform teams who need to reduce judge variance by anchoring a scoring rubric with concrete, few-shot examples. The core job is to produce a rubric that doesn't just describe what a score means in the abstract, but shows it—with scored exemplars at each level, counter-examples near decision boundaries, and annotation rationales that explain why each example received its score. Use this when you're building a model-graded evaluation pipeline and human spot checks reveal that different judges (or the same judge across runs) interpret your scoring criteria inconsistently. The prompt is designed for teams that already have a scoring scale and dimension definitions, but need to harden them into a repeatable, automatable contract.

Do not use this prompt when you are still defining your evaluation dimensions or scoring scale from scratch—start with the LLM Judge Rubric Design Prompt Template or Scoring Scale Definition Prompt Template instead. This prompt assumes you have a working rubric and need to anchor it with examples. It is also not a substitute for human calibration; the exemplars it produces should be reviewed by domain experts before they become the ground truth for automated scoring. Avoid using this prompt for high-stakes, one-shot decisions where a single score triggers an irreversible action. In those cases, the rubric output should feed into a human review queue, not directly into an automated gate.

After generating the rubric, validate that the exemplars do not leak evaluation criteria into the output—a common failure mode where the example text accidentally contains the scoring rules, teaching the judge to pattern-match rather than evaluate. Run the rubric through a small set of known cases with both human and model judges, then compare score distributions. If inter-rater reliability doesn't improve after adding exemplars, the examples may be ambiguous or the underlying dimension definitions need refinement. The next step is to wire this rubric into your evaluation harness with the Rubric Calibration Against Human Ratings Prompt to measure alignment and detect drift over time.

This template generates a complete evaluation rubric with example outputs and scores anchored at each performance level. Unlike a bare scoring scale, this prompt instructs the model to produce exemplars that demonstrate what each score level looks like in practice, plus counter-examples near decision boundaries to reduce judge variance. The output includes annotation rationales so evaluators understand why an exemplar belongs at a particular level. Use this when you need to calibrate multiple LLM judges or human raters against a shared, example-anchored standard.

codeCopy
You are an evaluation rubric designer. Your task is to produce a scored rubric with example outputs at each performance level.

## Evaluation Context
- Task being evaluated: [TASK_DESCRIPTION]
- Quality dimensions to score: [DIMENSIONS]
- Scoring scale: [SCALE_TYPE] with [NUM_LEVELS] levels
- Domain constraints: [DOMAIN_CONSTRAINTS]
- Risk level: [RISK_LEVEL]

## Rubric Requirements
For each score level, produce:
1. A clear level definition with distinguishing characteristics
2. At least [MIN_EXEMPLARS] scored exemplars that demonstrate the level
3. At least [MIN_COUNTER_EXAMPLES] counter-examples near the boundary with the adjacent level, with explanation of why they don't cross the boundary
4. Annotation rationales for each exemplar and counter-example explaining the scoring decision

## Output Schema
Return a JSON object with this structure:
{
  "rubric_name": "string",
  "task_description": "string",
  "dimensions": [
    {
      "name": "string",
      "definition": "string",
      "weight": number
    }
  ],
  "scale": {
    "type": "[SCALE_TYPE]",
    "levels": [
      {
        "score": "[LEVEL_VALUE]",
        "label": "string",
        "definition": "string",
        "distinguishing_characteristics": ["string"],
        "exemplars": [
          {
            "input": "string",
            "output": "string",
            "rationale": "string"
          }
        ],
        "boundary_counter_examples": [
          {
            "input": "string",
            "output": "string",
            "adjacent_level": "[LEVEL_VALUE]",
            "why_not_adjacent_level": "string"
          }
        ]
      }
    ]
  },
  "scoring_rules": {
    "aggregation_method": "string",
    "tiebreaking_rules": "string",
    "edge_case_handling": "string"
  },
  "calibration_notes": "string"
}

## Constraints
- Exemplars must be realistic and representative of production inputs
- Counter-examples must sit near genuine decision boundaries, not obvious failures
- Rationales must reference specific rubric criteria, not vague impressions
- Do not include evaluation criteria inside exemplar outputs that would leak scoring rules to the evaluated system
- If [RISK_LEVEL] is high, flag exemplars that require human review
- All exemplars must be self-contained and evaluable without external context unless [DOMAIN_CONSTRAINTS] explicitly requires it

Adaptation guidance: Replace [TASK_DESCRIPTION] with the specific task being evaluated (e.g., "customer support email responses"). Define [DIMENSIONS] as the quality axes that matter (accuracy, tone, completeness). Set [SCALE_TYPE] to numeric, categorical, or binary depending on your evaluation pipeline. [MIN_EXEMPLARS] and [MIN_COUNTER_EXAMPLES] control rubric density—start with 2-3 each and increase if judges show high variance. The [RISK_LEVEL] field triggers human-review flags in the output; set to "high" for regulated domains. The constraint against leaking evaluation criteria into exemplar outputs is critical: if your exemplars contain scoring instructions, downstream models may game the evaluation rather than demonstrate genuine quality.

Next steps: After generating the rubric, validate that exemplars at adjacent levels are genuinely distinguishable by running them through a separate judge prompt. If boundary counter-examples produce inconsistent scores across judges, add more counter-examples or refine the level definitions. Store the rubric as a versioned artifact in your evaluation pipeline and re-validate after any task definition or domain constraint changes.

This prompt is not a one-off query; it is a rubric factory that should be integrated into your evaluation infrastructure. The primary integration point is a rubric generation service that accepts a task description, a set of example outputs, and optional constraints, then returns a validated rubric object. The service should call the LLM with this prompt template, parse the structured output, and run a series of automated checks before the rubric is stored or used in any downstream scoring pipeline.

Validation and error handling are critical. After the model returns the rubric, you must validate that: (1) the output is valid JSON matching your expected schema; (2) every score level has at least one exemplar; (3) no exemplar text appears verbatim in the evaluation criteria (to prevent leakage); (4) the scoring scale is consistent and monotonic; and (5) counter-examples are provided near decision boundaries. If validation fails, implement a retry loop that feeds the specific validation errors back into the prompt as additional [CONSTRAINTS] and requests a corrected rubric. After three failed retries, escalate to a human reviewer via a review queue rather than silently accepting a broken rubric.

Model choice matters for this workflow. Use a model with strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid smaller or older models that may struggle with the nested JSON structure and the precision required for non-leaky exemplars. For production deployments, log every generated rubric along with its validation results, the model version, and the prompt template version. This audit trail is essential for debugging judge drift later. When the rubric will be used in regulated or high-stakes domains, always require a human domain expert to review and sign off on the rubric before it goes live, even if all automated validations pass.