Rubric for Instruction Following Assessment Prompt Template

This prompt is for prompt engineers and evaluation leads who need to verify that an AI-generated output follows a set of explicit instructions, format requirements, exclusion rules, and behavioral constraints. The primary job-to-be-done is producing a structured, machine-readable rubric that an LLM judge can use to grade instruction adherence, including partial-compliance scoring rules for outputs that follow some constraints but miss others. Use this when you are shipping structured outputs to an API, a downstream parser, or a user-facing product where partial compliance is still a failure. The rubric it generates is the contract that turns subjective quality into repeatable, automatable model-graded scores.

The ideal user has a concrete set of constraints they need enforced—such as a required JSON schema, a list of forbidden topics, word limits, or mandatory sections—and needs a scoring artifact they can pass directly to an evaluation harness. Required context includes the original system prompt or user instructions, the expected output schema, and a clear definition of what constitutes a violation. The prompt works best when constraints are explicit and verifiable, not when quality is purely stylistic or subjective. For example, 'the output must be valid JSON with fields A, B, and C' is a strong fit; 'the output should sound professional' is too vague without further operationalization.

Do not use this prompt for open-ended creative quality assessment or for evaluating factual accuracy against external evidence. Those require separate rubrics for groundedness or summarization quality. Also avoid this prompt when the instruction set is ambiguous or when compliance cannot be determined from the output alone—for instance, if verifying adherence requires knowledge of private user context not present in the evaluation window. In high-risk domains such as healthcare, legal, or finance, the generated rubric should be treated as a screening tool, not a final arbiter; always pair it with human review and evidence grounding. If your evaluation needs include multi-turn conversation tracking, tool-use correctness, or safety policy adherence, use the dedicated rubric prompts for those surfaces instead.

This template is the core of the playbook. It instructs an LLM judge to produce a detailed, multi-level scoring rubric specifically for evaluating how well another AI's output adheres to a set of explicit instructions. The prompt is designed to be copy-pasted and adapted by replacing the square-bracket placeholders with your specific task instructions, format requirements, and behavioral constraints. The resulting rubric should be a standalone artifact that can be used by another LLM judge or a human evaluator to produce consistent, repeatable scores.

textCopy
You are an expert evaluation rubric designer. Your task is to create a detailed scoring rubric for assessing an AI's adherence to a specific set of instructions. The rubric must be clear, objective, and designed for use by another LLM judge.

### Instructions to be Evaluated
[INSTRUCTIONS]

### Rubric Design Requirements
- **Scoring Scale:** Define a [SCALE_RANGE, e.g., 1-5] scoring scale.
- **Score Levels:** For each score on the scale, provide a clear, behavioral description (an anchor) of what constitutes that level of performance.
- **Dimensions:** Break down the evaluation into distinct, non-overlapping dimensions. For example: `format_adherence`, `content_constraint_compliance`, `exclusion_rule_following`, `behavioral_tone`.
- **Partial Compliance:** Explicitly define how to score partial compliance. For example, if a response follows 3 out of 4 format rules, how is that scored differently from following 0 rules?
- **Edge Cases:** Include guidance for scoring edge cases, such as when the model correctly refuses a task based on a constraint, or when two constraints conflict.
- **Justification:** The rubric must require the judge to output a brief justification for each dimension's score, citing the specific part of the instruction that was followed or violated.

### Output Format
Output the rubric as a structured JSON object with the following schema:
{
  "rubric_name": "string",
  "scoring_scale": {
    "min": integer,
    "max": integer,
    "description": "string"
  },
  "dimensions": [
    {
      "name": "string",
      "description": "string",
      "scoring_guide": {
        "[SCORE]": "string describing the anchor for this score",
        ...
      }
    }
  ],
  "partial_compliance_rules": "string",
  "edge_case_guidance": "string"
}

### Constraints
- Do not include any text outside the JSON object.
- Ensure the scoring anchors are mutually exclusive and collectively exhaustive.
- The rubric must be directly usable by another LLM to grade a new response against the provided [INSTRUCTIONS].

To adapt this template, start by replacing the [INSTRUCTIONS] placeholder with the exact system prompt, user query, and any formatting rules you gave to the AI being evaluated. Next, define your [SCALE_RANGE], such as 1-5 or 0-3. The most critical adaptation is in the dimensions array: you must decompose your instructions into specific, testable components. For example, if your instructions require a JSON output with three specific fields and a professional tone, you should create separate dimensions for schema_compliance and tone_adherence. After generating the rubric, always validate it by manually scoring a few known outputs to ensure the anchors are clear and the partial compliance rules produce the expected results. This calibration step is essential before deploying the rubric in an automated evaluation pipeline.

This prompt is designed to run as a scoring step inside an automated evaluation harness, not as a one-off chat interaction. The typical integration pattern places it after a candidate output has been generated by the system under test. The harness feeds the original instructions, the candidate output, and the rubric prompt template into an LLM judge, then parses the structured score and justification for downstream decision-making. The primary integration points are: input assembly, model invocation, output validation, score extraction, and pipeline gating.

Input assembly requires merging three data sources before calling the judge: the original instruction set that was given to the system under test, the candidate output produced by that system, and the rubric prompt template itself. Store these as separate variables in your pipeline configuration. The instruction set should be the exact text the system received, not a summary. The candidate output should be captured raw, before any post-processing or repair steps. Model invocation should use a low-temperature setting (0.0–0.2) to maximize scoring consistency. Choose a model with strong instruction-following and structured output capabilities. If your pipeline runs hundreds of evaluations, consider batching multiple scoring requests or using a model with lower per-token cost for this judge step, since the judge model does not need to be the same model that produced the candidate output.

Output validation is critical because a malformed score breaks the pipeline. Parse the judge's response and verify that: the score field is present and within the defined numeric range, the justification field is non-empty, and any required sub-fields like partial_compliance_flags or violated_constraints are present when the score is below maximum. If validation fails, implement a retry loop with a maximum of 2 additional attempts, feeding the validation error message back to the judge as additional context. If all retries fail, log the raw response, flag the evaluation as SCORING_ERROR, and route it for human review rather than silently assigning a default score. Score extraction should write the validated score, justification, and metadata (judge model, timestamp, retry count) to your evaluation database or logging system for traceability.

For CI/CD integration, wrap this prompt in a test harness that runs against a golden dataset of instruction-output pairs with known expected scores. Set a minimum score threshold as a pipeline gate: if the system under test falls below the threshold on critical instruction-following dimensions, block the release. Store historical scores per prompt version to detect drift. Common failure modes in production include: the judge hallucinating constraints that weren't in the original instructions, the judge being overly lenient on partial compliance, and score inflation when the candidate output is verbose but non-compliant. Mitigate these by periodically calibrating the judge against human ratings using a held-out calibration set, and by logging judge justifications for spot-check review. Do not treat the LLM judge score as ground truth—it is a signal that reduces, but does not eliminate, the need for human evaluation on high-risk outputs.