Example Consistency Check Prompt Template

This prompt is designed for prompt QA engineers and AI developers who maintain few-shot example libraries. Its job is to audit a set of input-output examples against a defined contract and flag any that deviate in format, contradict each other's labels, or violate the expected output schema. Use it before promoting an example set to production, after adding new examples, or when debugging unexpected model behavior that might originate from inconsistent demonstrations. The prompt assumes you already have a collection of examples and a target contract. It does not generate new examples or evaluate model outputs directly.

The ideal user is someone responsible for the quality of a prompt's behavioral source of truth. This is not a prompt for end-users or for generating creative content. You should use it when you have a structured set of examples (JSON, CSV, or a clearly delimited text format) and a defined output contract, such as a JSON schema, a set of allowed labels, or a strict formatting rule. The prompt is most valuable when the example set is large enough that manual review is unreliable—typically more than 20 examples—or when the cost of a single bad example in production is high, such as in regulated classification, safety-critical routing, or customer-facing structured outputs.

Do not use this prompt to evaluate the model's runtime outputs, to generate new training data, or to compare different prompt templates. It is strictly an example-set auditing tool. It also does not replace a full regression test suite; it checks internal consistency, not whether the examples produce the correct model behavior. If you need to measure how well the examples teach the model, use the 'Few-Shot Example Effectiveness Scoring Prompt' instead. After running this consistency check, you should feed any flagged examples into a repair workflow or remove them before the example set is used in a production prompt.

The following template is designed to be dropped directly into your prompt evaluation harness. It accepts a set of examples and a contract definition, then produces a structured consistency report. Every placeholder is enclosed in square brackets. Replace them with your specific data before running the check. The prompt assumes you have already extracted your example set into a machine-readable format; if your examples are still embedded in a larger system prompt, extract them first.

codeCopy
You are an example consistency auditor. Your task is to review a set of input-output examples and flag any that violate the provided contract.

[EXAMPLES]

[CONTRACT]

For each example, check:
1. Does the output conform to the required format and schema?
2. Is the label or classification consistent with the input and with other examples that have similar inputs?
3. Are there any contradictions where two examples with nearly identical inputs produce different outputs?
4. Are there any missing required fields, extra fields, or type mismatches?

Produce a JSON report with the following structure:
{
  "summary": {
    "total_examples": <int>,
    "consistent_examples": <int>,
    "flagged_examples": <int>,
    "contradiction_pairs": <int>
  },
  "flagged_examples": [
    {
      "example_id": "<string>",
      "input_preview": "<first 80 chars of input>",
      "issues": [
        {
          "type": "format_violation | label_contradiction | schema_violation | field_mismatch",
          "description": "<specific description of the issue>",
          "conflicting_example_id": "<string or null>"
        }
      ]
    }
  ],
  "contradiction_pairs": [
    {
      "example_a_id": "<string>",
      "example_b_id": "<string>",
      "similarity_reason": "<why these examples are considered similar>",
      "output_difference": "<description of the conflicting outputs>"
    }
  ]
}

Only flag issues you are confident about. If an example is ambiguous but not clearly wrong, do not flag it. Include the exact example_id as provided in the input.

To adapt this template, replace [EXAMPLES] with your serialized example set. Each example must include a unique example_id field so the report can reference it precisely. Replace [CONTRACT] with a clear specification of the expected input-output relationship, including the output schema, classification taxonomy, and any formatting rules. If your contract is long, consider splitting it into a separate [OUTPUT_SCHEMA] and [CONSTRAINTS] section for clarity. For high-stakes domains where a false negative in the consistency report could cause production incidents, always route flagged examples to a human reviewer before accepting automated removal or correction.

The Example Consistency Check prompt is designed to operate as a gate in a prompt QA pipeline, not as a one-off manual review tool. When wired into an application, it should run automatically whenever an example set is updated, a new prompt version is proposed, or a scheduled audit is triggered. The harness must treat the prompt's output as structured evidence—a consistency report—that downstream systems can parse, log, and act on without manual interpretation. This means the prompt must be called with a strict output schema, and the response must be validated before any automated decision is made.

To implement this, wrap the prompt call in a function that accepts the example set and a configuration object specifying the expected input-output contract, allowed format variants, and severity thresholds. The function should: (1) assemble the prompt with the example set injected into the [EXAMPLES] placeholder and the contract definition injected into [CONTRACT]; (2) call the model with response_format set to a JSON schema that matches the expected consistency report structure—fields like violations, contradictory_pairs, schema_errors, and overall_score; (3) validate the response against that schema and retry once on parse failure with a repair prompt; (4) log the full prompt, response, and validation result to an audit store; and (5) route the report to a human review queue if the overall_score falls below a configurable threshold or if any violation.severity is critical. For high-stakes domains like healthcare or legal example sets, always require human sign-off before the example set is promoted to production, regardless of the automated score.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities, such as gpt-4o or claude-3-5-sonnet, because the task requires precise comparison across many examples and reliable JSON generation. Avoid smaller or older models that may hallucinate comparisons or drop fields. If the example set is large, chunk it into batches of 20–30 examples and run the check per batch, then merge the reports with a deduplication step for contradictory pairs that span batches. Do not use this prompt as a real-time guard on every inference call—it is a pre-deployment QA tool, not a runtime validator. The next step after receiving a clean consistency report is to run the Golden Example Set Regression Test to confirm that the consistent examples actually produce the intended model behavior.