Example-to-Instruction Fidelity Audit Prompt

This prompt is for prompt designers and QA engineers who need to verify that their few-shot examples actually teach the model what the system instructions claim. The core job-to-be-done is detecting misalignment between stated rules and demonstrated behaviors before a prompt ships to production. You should use this prompt when you have a system prompt with explicit behavioral rules and a set of input-output examples that are supposed to illustrate those rules. The ideal user is someone who owns prompt quality and needs evidence—not intuition—that examples reinforce rather than undermine the instruction set.

The prompt works by comparing each example's input-output pair against every stated instruction rule, producing a fidelity report with per-example traceability scoring. It requires three concrete inputs: a complete system instruction block, a set of few-shot examples with clear input-output boundaries, and a defined output schema for the audit report. The prompt is most effective when instructions are explicit and testable—vague instructions like 'be helpful' will produce noisy fidelity scores. You should also provide a risk tolerance threshold so the prompt can flag examples that silently violate safety, refusal, or format constraints rather than just style preferences.

Do not use this prompt for evaluating instruction quality in isolation or for grading examples without a reference instruction set. It is not a general example quality scorer—it specifically measures instruction-to-example alignment. Avoid using it on multi-turn conversation examples without first flattening them into single-turn input-output pairs with clear state markers. If your examples intentionally override or extend instructions for specific edge cases, you must supply an explicit exceptions list or the prompt will flag those as fidelity violations. For high-risk domains where instruction violations could cause safety incidents, always route the fidelity report through human review before accepting automated pass/fail decisions.

The following prompt template is designed to be copied directly into your prompt development environment. It accepts a set of system instructions and a corresponding set of few-shot examples, then produces a structured fidelity report. The report identifies misaligned demonstrations, scores per-example traceability to stated rules, and flags contradictions. All placeholders are enclosed in square brackets and must be replaced with your actual content before use.

textCopy
You are an example-to-instruction fidelity auditor. Your job is to compare a set of few-shot examples against the system instructions they are meant to demonstrate. You will produce a structured fidelity report that identifies misalignments, scores traceability, and flags contradictions.

## INPUTS

### System Instructions
[SYSTEM_INSTRUCTIONS]

### Few-Shot Examples
[FEW_SHOT_EXAMPLES]

### Output Schema
[OUTPUT_SCHEMA]

### Constraints
[CONSTRAINTS]

## TASK

For each example in the provided set, perform the following analysis:

1. **Instruction Mapping**: Identify which specific instruction(s) from the System Instructions this example is intended to demonstrate. If no clear mapping exists, note this as a gap.

2. **Behavioral Alignment Check**: Compare the example's input-output behavior against the mapped instruction(s). Determine whether the example:
   - **FULLY ALIGNS**: The example perfectly demonstrates the instruction.
   - **PARTIALLY ALIGNS**: The example demonstrates some but not all aspects of the instruction.
   - **CONTRADICTS**: The example violates or teaches behavior opposite to the instruction.
   - **IS IRRELEVANT**: The example does not relate to any stated instruction.

3. **Traceability Score**: Assign a score from 0 to 100 indicating how clearly this example can be traced back to a specific instruction. A score of 100 means the example is an unambiguous, direct demonstration of a single instruction. A score of 0 means the example has no discernible connection to any instruction.

4. **Contradiction Detection**: If the example contradicts any instruction, describe the contradiction explicitly. If the example contradicts another example in the set, flag both examples and describe the conflict.

5. **Ambiguity Assessment**: Note whether the example could reasonably be interpreted as demonstrating multiple, conflicting instructions. Flag any ambiguity that could confuse the model during training or inference.

## OUTPUT FORMAT

Produce a JSON object conforming to the Output Schema provided. If no Output Schema is provided, use the following default structure:

{
  "fidelity_report": {
    "summary": {
      "total_examples": <integer>,
      "fully_aligned_count": <integer>,
      "partially_aligned_count": <integer>,
      "contradictory_count": <integer>,
      "irrelevant_count": <integer>,
      "average_traceability_score": <float>
    },
    "per_example_results": [
      {
        "example_id": "<string>",
        "mapped_instructions": ["<string>"],
        "alignment": "FULLY_ALIGNED | PARTIALLY_ALIGNED | CONTRADICTS | IRRELEVANT",
        "traceability_score": <integer 0-100>,
        "contradictions": [
          {
            "contradicts": "<instruction_id or example_id>",
            "description": "<string>"
          }
        ],
        "ambiguity_notes": "<string or null>"
      }
    ],
    "global_contradictions": [
      {
        "example_a": "<example_id>",
        "example_b": "<example_id>",
        "description": "<string>"
      }
    ],
    "uncovered_instructions": ["<instruction_id>"],
    "recommendations": ["<string>"]
  }
}

## CONSTRAINTS

- Do not modify or reinterpret the System Instructions. Treat them as the ground truth.
- Do not invent instructions that are not present in the System Instructions.
- If an example demonstrates behavior not covered by any instruction, flag it as IRRELEVANT rather than forcing a mapping.
- When scoring traceability, penalize examples that require inference or interpretation to connect to an instruction.
- Flag all contradictions, even if they appear minor.
- If the provided Constraints include additional rules, follow them strictly.

To adapt this template for your workflow, replace the placeholders with your actual content. The [SYSTEM_INSTRUCTIONS] placeholder should contain the complete system prompt or instruction set under audit. The [FEW_SHOT_EXAMPLES] placeholder should contain your example set, formatted consistently as input-output pairs. If your application requires a specific JSON schema for downstream processing, provide it in [OUTPUT_SCHEMA]; otherwise, the default schema will be used. Use [CONSTRAINTS] to add domain-specific rules, such as requiring human review for contradictions in regulated contexts or setting minimum traceability score thresholds for production readiness.

Before deploying this prompt into a production pipeline, wire the output into a validation step that checks the JSON structure against your expected schema. For high-risk domains such as healthcare or legal workflows, route any report containing contradictions or traceability scores below a defined threshold to a human reviewer. Log every audit run with the prompt version, example set version, and instruction set version to maintain an audit trail for governance reviews.

The Example-to-Instruction Fidelity Audit Prompt is designed to operate as a batch evaluation step within a prompt QA pipeline, not as a real-time user-facing prompt. You feed it a system instruction and a set of examples, and it returns a structured fidelity report. The primary integration point is a CI/CD pipeline for prompt releases, a scheduled QA job, or a manual review tool used by prompt engineers before deployment. The prompt expects two inputs: [SYSTEM_INSTRUCTION] (the rules the model should follow) and [EXAMPLES] (a list of input-output pairs to audit). The output is a JSON report with per-example fidelity scores, misalignment flags, and traceability mappings. Because this is an offline audit workflow, latency is not critical, but accuracy and reproducibility are. Run this prompt with a low-temperature setting (0.0–0.2) to maximize deterministic scoring, and always use the same model version you intend to deploy the audited prompt against—cross-model audits produce misleading fidelity scores.

To wire this into an application, wrap the prompt in a validation and retry harness. After receiving the JSON report, validate it against a strict schema: each example must have a fidelity_score (0–1 float), an instruction_alignment array mapping each instruction clause to a compliance flag, and a misalignment_severity enum (none, minor, major, critical). If the output fails schema validation, retry once with an explicit repair instruction appended to the prompt: The previous output failed JSON schema validation. Ensure all required fields are present and correctly typed. If the retry also fails, log the raw output and escalate for human review. For high-stakes prompt releases (safety-critical systems, regulated domains, customer-facing agents), add a human approval gate after the audit report is generated. The gate should require sign-off on any example flagged with misalignment_severity: critical or any instruction clause with less than 80% example compliance across the set. Store the audit report alongside the prompt version in your prompt registry for audit trail purposes.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid smaller or older models that may conflate example quality with instruction fidelity or produce inconsistent JSON. If you're auditing a prompt intended for a smaller model, still use a capable auditor model; the goal is to detect misalignment in the examples, not to simulate the target model's behavior. For large example sets (>50 examples), split the audit into batches of 20–30 examples to avoid context-window saturation and attention degradation. Run each batch independently, then merge the reports. Add a batch-level consistency check: if the same instruction clause shows wildly different compliance rates across batches, flag the audit itself for review—this often indicates ambiguous instructions or examples that are sensitive to surrounding context. Log every audit run with the prompt version, model version, timestamp, and report hash for traceability. Do not auto-apply fixes based on the audit report; the report identifies problems, but remediation (rewriting examples, clarifying instructions, or removing contradictory demonstrations) requires human judgment.