Scoring Scale Definition Prompt Template

This prompt is a design-time tool for evaluation engineers and AI platform teams who need to create a formal contract for what each score level means. Use it when you are standardizing evaluation metrics across multiple judges, models, or rubric versions and require a numeric or categorical scale with clear level definitions, anchor examples, and differentiation rules. The ideal user is someone who already understands the quality dimensions they care about—such as factual accuracy, instruction adherence, or groundedness—but needs to translate those dimensions into a repeatable scoring protocol that resists interpretation drift over time.

The prompt produces a complete scoring scale definition that includes level descriptions, boundary examples, and explicit rules for distinguishing adjacent scores. It also generates consistency validation instructions that you can embed into runtime scoring prompts to detect when a judge's application of the scale begins to drift. This is critical for production systems where scores feed into automated decisions, dashboards, or model improvement pipelines. Do not use this prompt if you already have a calibrated rubric and only need to apply it—use a rubric application prompt instead. Similarly, if you need a full multi-dimensional rubric with weighted dimensions and aggregation logic, start with the Multi-Dimensional Rubric Design Prompt Template. This prompt focuses narrowly on defining the scale itself, not on scoring specific outputs.

Before running this prompt, gather examples of outputs that represent different quality levels, including borderline cases that are hard to classify. The prompt works best when you provide concrete anchor examples for at least the top, bottom, and middle of your intended scale. After generating the scale definition, validate it by having multiple human raters apply it to a held-out set of examples and measuring inter-rater agreement. If agreement is below your threshold, iterate on the level definitions and differentiation rules before deploying the scale to automated judges. Avoid using this prompt for binary pass/fail decisions—use the Pass/Fail Binary Decision Rubric Prompt Template for go/no-go gates where a numeric scale adds unnecessary complexity.

This template produces a complete, structured scoring scale definition that an LLM judge can use to assign consistent numeric or categorical scores. It forces explicit level definitions, anchor examples, and differentiation rules so that multiple judges or repeated runs produce aligned results. Use this when you need a scoring contract that can be versioned, tested, and audited. Do not use this template for binary pass/fail decisions—those require a separate threshold rubric.

Below is the copy-ready prompt template. Replace every square-bracket placeholder before sending it to the model. The placeholders control the scale's granularity, the quality dimensions being scored, and the output format required by your evaluation harness.

textCopy
You are designing a scoring scale for evaluating AI-generated outputs. Your task is to produce a complete, unambiguous scale definition that multiple judges can apply consistently.

## Scale Requirements
- Scale Name: [SCALE_NAME]
- Scale Type: [NUMERIC_SCALE | CATEGORICAL_SCALE]
- Number of Levels: [NUMBER_OF_LEVELS]
- Quality Dimension Being Scored: [DIMENSION_NAME]
- Dimension Definition: [DIMENSION_DEFINITION]

## Level Definition Rules
For each level on the scale, provide:
1. A clear, non-overlapping label
2. A description of what outputs at this level look like
3. At least one concrete anchor example of an output that belongs at this level
4. At least one boundary example of an output that nearly falls into the adjacent level but stays in this one
5. Explicit differentiation rules that separate this level from the levels immediately above and below

## Additional Constraints
- Levels must be mutually exclusive: no output should satisfy the criteria for two levels simultaneously.
- Levels must be collectively exhaustive: every possible output must fit into exactly one level.
- Anchor examples must be realistic and varied.
- Differentiation rules must reference observable output characteristics, not vague quality adjectives.
- If the scale is numeric, define the numeric range and whether fractional scores are permitted.
- If the scale is categorical, define the ordered categories and their mapping to any downstream numeric scores.

## Output Format
Return a JSON object with this exact schema:
{
  "scale_name": "string",
  "scale_type": "numeric | categorical",
  "dimension": "string",
  "levels": [
    {
      "label": "string",
      "score": number | string,
      "description": "string",
      "anchor_example": "string",
      "boundary_example": "string",
      "differentiation_from_lower": "string | null",
      "differentiation_from_higher": "string | null"
    }
  ],
  "differentiation_rules_summary": "string"
}

## Context
[DIMENSION_CONTEXT]

## Examples of Good and Bad Outputs for Calibration
[ANNOTATED_EXAMPLES]

## Risk Level
[RISK_LEVEL]

After copying the template, adapt it by filling the placeholders with your specific evaluation context. For high-risk domains, add a human review step before the scale is used in production: have a domain expert verify that level descriptions match real quality distinctions and that anchor examples don't introduce unintended bias. Wire the completed scale into your evaluation harness by parsing the JSON output and using the levels array as the scoring reference for your LLM judge. Validate that the returned JSON matches the schema before storing it as a versioned artifact. If the model produces overlapping levels or missing differentiation rules, reject the output and retry with a stronger constraint on mutual exclusivity.

The Scoring Scale Definition Prompt is not a one-off chat interaction—it is a configuration generator for your evaluation infrastructure. The prompt produces a structured scoring scale definition (typically JSON) that downstream LLM judges will reference when assigning scores. Treat the output as a critical configuration artifact: it must be validated, versioned, and tested before any judge consumes it. The implementation harness should wrap this prompt in a function that accepts the required inputs—evaluation context, number of levels, differentiation rules, and anchor examples—and returns a validated scale object ready for integration into your judge prompt templates.

Build the harness with these concrete components. Input validation: before calling the model, verify that [NUM_LEVELS] is an integer between 2 and 10, that [EVALUATION_DIMENSION] is a non-empty string, and that [ANCHOR_EXAMPLES] contains at least one example per requested level. Model selection: use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller models for scale definition—they tend to produce overlapping level descriptions that cause judge confusion downstream. Output validation: parse the response as JSON and check that the number of levels matches [NUM_LEVELS], that each level has a non-empty name, description, and differentiation_rules field, and that level definitions are mutually exclusive (no two levels should describe the same quality threshold). Retry logic: if validation fails, retry up to 2 times with the validation errors injected into the [CONSTRAINTS] field as explicit correction instructions. After 3 total attempts, escalate to a human reviewer rather than silently accepting a broken scale.

Integrate the validated scale into your judge prompt by injecting it as a structured [SCORING_SCALE] variable. Store the scale definition in a version-controlled registry alongside its generation metadata: timestamp, model version, input parameters, and the human reviewer who approved it. Before deploying a new scale, run a consistency check: have two independent LLM judge instances score the same set of 20-30 outputs using the scale, then measure inter-rater agreement (Cohen's kappa or quadratic weighted kappa depending on ordinal vs. interval assumptions). If agreement falls below 0.7, flag the scale for human recalibration. What to avoid: do not generate scales dynamically per evaluation run—this introduces uncontrolled variance that makes score trends uninterpretable. Do not skip the anchor example validation step; scales without concrete anchors at each level produce high judge variance. Do not use the scale in production without a human reviewing the level definitions for domain appropriateness and edge-case coverage.