Rubric for Output Format Compliance Prompt Template

This prompt is for API-facing product teams and evaluation engineers who need to automate the grading of structured output compliance. Use it when you have a target schema—JSON, XML, or a typed object—and you need an LLM judge to score whether generated outputs contain the right fields, correct types, valid enum values, and proper structure. The core job-to-be-done is not generating structured outputs, but building a consistent, automatable rubric that evaluates them. You wire this rubric into your evaluation harness so every model response gets a repeatable format compliance score, enabling regression testing and pre-release gates.

This prompt is not a good fit for evaluating the semantic quality, factual accuracy, or helpfulness of an output. If you need to grade whether a summary is faithful to its source text, use a groundedness rubric. If you need to check if a model followed a complex behavioral policy, use an instruction adherence rubric. This prompt is specifically for structural compliance: field presence, type correctness, enum adherence, and schema shape. It is also not a replacement for a deterministic JSON Schema validator. Use this prompt when you need partial-compliance scoring for near-miss formats—for example, when a model returns the right fields but with a string instead of an integer, or when an optional field is missing but the rest of the structure is valid. A deterministic validator would give a binary pass/fail, but this rubric captures degrees of compliance.

The ideal user is an evaluation engineer or product developer who already has a target schema and a batch of model outputs to score. You should have a clear understanding of which fields are required, which types are expected, and which enum values are valid. Before using this prompt, define your schema as a concrete artifact—a JSON Schema document, a typed object definition, or a set of field specifications. The prompt will ask you to provide this schema as [TARGET_SCHEMA]. If you cannot articulate the expected structure precisely, this prompt will produce a vague rubric that fails to catch real format errors. Once you have the rubric, integrate it into your evaluation pipeline alongside other quality checks, and run it against a golden dataset of known-correct and known-incorrect outputs to calibrate the scoring thresholds before relying on it in production.

This prompt template instructs an LLM to act as an evaluation rubric designer. Its job is to generate a detailed, structured scoring guide for assessing whether an AI-generated output conforms to a specific data format. You provide the target schema, and the judge produces a rubric that grades field presence, type correctness, enum adherence, and structural constraints. The resulting rubric is a contract you can use with another LLM judge to automate format compliance checks in your CI/CD pipeline or API gateway.

codeCopy
You are an expert evaluation rubric designer specializing in structured output validation. Your task is to create a detailed, multi-level scoring rubric for assessing format compliance of an AI-generated output against a provided schema.

**Target Schema:**
[OUTPUT_SCHEMA]

**Scoring Scale:**
Use a 0-4 scale where:
- 4 (Perfect): Output fully conforms to the schema with no errors.
- 3 (Minor Issue): Output is valid and usable but has minor stylistic deviations (e.g., extra whitespace, key ordering).
- 2 (Partial Compliance): Output has the correct structure but contains type errors, missing optional fields, or invalid enum values that a parser could reject.
- 1 (Major Non-Compliance): Output is recognizable as an attempt but is missing required fields, has a fundamentally wrong structure, or is wrapped in an unexpected object.
- 0 (Not Compliant): Output is not valid [FORMAT_TYPE], is completely unparseable, or is natural language text instead of structured data.

**Rubric Requirements:**
For each score level, provide:
1. A clear, one-sentence summary of the level.
2. A checklist of specific, testable criteria based on the schema (e.g., "`id` field is present and is a string", "`status` value is one of ['active', 'inactive']").
3. A concrete example of an output that would receive this score.

**Additional Constraints:**
[CONSTRAINTS]

**Output Format:**
Return the rubric as a JSON object with the following structure:
{
  "rubric_name": "Format Compliance Rubric",
  "scoring_scale": "0-4",
  "levels": [
    {
      "score": 4,
      "label": "Perfect",
      "summary": "...",
      "criteria": ["..."],
      "example_output": "..."
    }
    // ... repeat for scores 3, 2, 1, 0
  ]
}

To adapt this template, replace the [OUTPUT_SCHEMA] placeholder with your exact target schema, written as clearly as possible. Use JSON Schema, TypeScript interfaces, or a detailed plain-text description. The [CONSTRAINTS] placeholder is for any special rules, such as "the id field must be a UUIDv4" or "the entire output must be wrapped in a top-level data key." The [FORMAT_TYPE] placeholder in the score 0 description should be replaced with the expected format, like "JSON" or "XML", to make the failure condition explicit. The generated rubric is a static asset; you should version it alongside your prompt and review it for accuracy before using it in automated evaluation.

The format compliance rubric prompt is designed to be called programmatically as a judge model within an evaluation harness, not as a one-off chat interaction. The typical integration pattern is a post-generation validation step: your application generates a structured output, parses it, and if parsing fails or the output is suspect, the rubric prompt is invoked to produce a granular compliance score and diagnostic report. This score then drives automated decisions—accept the output, trigger a repair loop, or escalate for human review.

To wire this into a pipeline, wrap the prompt in a function that accepts the generated output and the expected schema definition as inputs. The schema should be passed as a JSON Schema object or a TypeScript interface definition in the [EXPECTED_SCHEMA] placeholder. After calling the judge model, parse its JSON response and extract the overall_compliance_score and critical_failures array. Implement a decision threshold: scores above 0.95 can auto-accept, scores between 0.8 and 0.95 should trigger a lightweight repair prompt, and scores below 0.8 or any presence of critical failures should halt the pipeline and route to a human review queue. Log every judge invocation with the input, output, score, and model version for auditability.

For production resilience, add a retry wrapper around the judge call itself—judge models can also produce malformed JSON. If the judge's output fails to parse, retry once with a stricter instruction appended: 'You must respond with valid JSON only. No markdown fences, no commentary.' If the retry also fails, log the raw response and escalate. When using this rubric in CI/CD pipelines for prompt regression testing, store the schema and a set of known-bad and known-good outputs as a golden test suite. Run the judge against these on every prompt change and fail the build if the compliance score on known-good outputs drops or if known-bad outputs start passing. This turns the rubric from a loose guideline into a hard release gate.