Golden Example Set Validation Prompt

This prompt is designed for prompt engineering teams that need to certify a set of few-shot examples before it ships to production. The core job-to-be-done is producing a structured validation report that answers a critical release question: are these examples complete, consistent, diverse, and behaviorally sound? The ideal user is a prompt engineer or QA lead holding a candidate example set and needing a go/no-go release decision backed by evidence, not intuition. You should use this when you have a finalized or near-finalized set of examples and need a rigorous, automated pre-release check before those examples are embedded into a system prompt, a RAG pipeline, or an agent's tool-use demonstrations.

To use this effectively, you must provide the full candidate example set, the intended model behavior specification, and any known constraints or risk areas. The prompt will then systematically audit the examples for coverage gaps, contradictory pairs, diversity metrics across demographic and linguistic dimensions, and behavioral consistency scores. It will produce a report with specific, actionable findings—not just a pass/fail flag. For example, it might identify that your examples over-represent formal language registers while missing casual or confrontational user tones, or that two examples teach conflicting refusal boundaries for the same class of request. This level of detail allows you to remediate issues before they become production incidents.

Do not use this prompt for evaluating a single example in isolation, for runtime example selection, or for generating new examples. It is not a few-shot retriever, a dynamic example selector, or a data augmentation tool. It is a pre-release certification gate, not a runtime tool. If you need to generate edge-case examples, use the Edge-Case Sampling Prompt. If you need to detect drift in already-deployed examples, use the Example Drift Monitoring Trigger Prompt. After running this validation, you should either approve the example set for release, iterate on the flagged issues, or escalate findings that require human judgment—such as stereotype risks or safety boundary conflicts—to your responsible AI or domain expert reviewers.

The following prompt template is designed to be pasted directly into your model interface or API call. It expects a structured set of examples, a reference schema or behavioral contract, and optional diversity dimensions to audit. Replace every square-bracket placeholder with your actual data before execution. The prompt instructs the model to act as an example set auditor, producing a structured validation report with go/no-go criteria rather than freeform commentary.

textCopy
You are an example set auditor. Your task is to validate a golden example set intended for few-shot prompting in a production AI system. Produce a structured validation report. Do not modify the examples. Do not generate new examples. Only audit.

## INPUTS

### EXAMPLE SET
[EXAMPLES]

### EXPECTED BEHAVIORAL CONTRACT
[BEHAVIORAL_CONTRACT]

### DIVERSITY DIMENSIONS TO AUDIT (optional)
[DIVERSITY_DIMENSIONS]

### TOKEN BUDGET CONSTRAINT (optional)
[TOKEN_BUDGET]

### KNOWN FAILURE MODES (optional)
[KNOWN_FAILURE_MODES]

## OUTPUT SCHEMA
Return a single JSON object with this exact structure:
{
  "validation_report": {
    "coverage_completeness": {
      "score": number (0-100),
      "covered_categories": [string],
      "missing_categories": [string],
      "distribution_skew_flags": [string]
    },
    "contradiction_analysis": {
      "contradiction_pairs": [
        {
          "example_a_id": string,
          "example_b_id": string,
          "conflict_type": string,
          "description": string
        }
      ],
      "consistency_score": number (0-100)
    },
    "diversity_metrics": {
      "demographic_coverage": object,
      "linguistic_register_coverage": object,
      "difficulty_distribution": object,
      "entity_diversity": object
    },
    "behavioral_consistency": {
      "alignment_score": number (0-100),
      "deviation_examples": [string],
      "boundary_violation_flags": [string]
    },
    "release_criteria": {
      "go_no_go": "GO" | "NO_GO" | "CONDITIONAL_GO",
      "blocking_issues": [string],
      "recommended_actions": [string]
    }
  }
}

## CONSTRAINTS
- If any example contradicts another, flag it in contradiction_pairs and reduce consistency_score.
- If coverage_completeness score is below 70, release_criteria must be NO_GO.
- If behavioral_consistency alignment_score is below 80, release_criteria must be NO_GO or CONDITIONAL_GO.
- Do not hallucinate missing categories. Only report gaps you can identify from the provided examples and behavioral contract.
- If DIVERSITY_DIMENSIONS is empty, skip demographic_coverage and linguistic_register_coverage.
- If TOKEN_BUDGET is provided, include a token_efficiency note in recommended_actions if the example set exceeds 80% of budget.

## RISK LEVEL
[HIGH_RISK if examples involve regulated domains, safety-critical decisions, or PII; otherwise MEDIUM_RISK]

If RISK_LEVEL is HIGH_RISK, append a human_review_required flag to the release_criteria object and escalate any contradiction_pairs to blocking_issues.

Adapt this template by replacing [EXAMPLES] with your full few-shot example set in a structured format (JSON array of input-output pairs with unique IDs). Replace [BEHAVIORAL_CONTRACT] with a clear description of what correct behavior looks like, including refusal boundaries, format expectations, and output constraints. The [DIVERSITY_DIMENSIONS] placeholder accepts a list of demographic attributes, language registers, or entity types you want audited. Leave it empty if diversity auditing is out of scope. The [KNOWN_FAILURE_MODES] field lets you prime the auditor with failure patterns you've already observed in production, which improves contradiction detection. Always set [RISK_LEVEL] accurately—this controls whether the report includes a mandatory human review flag. After running the prompt, validate the output JSON against the schema before ingesting it into your release pipeline. If the model returns malformed JSON, use a repair prompt rather than accepting partial results.

Integrating this prompt into an application means treating example set validation as a gated step in your prompt release process. The prompt is designed to be called programmatically with a structured input containing the full example set, the system prompt under test, and the evaluation criteria. The output is a machine-readable validation report that your orchestration layer can parse to make a go/no-go decision. Do not treat this as a one-off manual check; it should run automatically whenever example sets are modified, new examples are added, or the underlying model version changes.

The implementation harness should follow a strict validate → parse → decide → escalate pattern. First, call the model with the prompt template, passing the example set as [EXAMPLES], the system prompt as [SYSTEM_PROMPT], and the evaluation dimensions as [EVALUATION_CRITERIA]. The model returns a JSON report conforming to [OUTPUT_SCHEMA]. Your application must validate this JSON against a strict schema before trusting the results. If validation fails, retry once with a repair prompt that includes the raw output and the schema. If the retry also fails, escalate for human review. On successful parse, check the release_decision field: "go" proceeds to the next pipeline stage; "no_go" blocks the release and logs the specific failure reasons from the failure_reasons array. All validation reports should be versioned and stored alongside the example set for auditability.

For production deployments, implement drift-aware triggering. Do not run this validation only on manual example set changes. Instead, trigger it automatically when production data distributions shift beyond a threshold, when the underlying model is upgraded, or when new failure modes are observed in production traces. Log the coverage_completeness_score, contradiction_count, and diversity_metrics as time-series metrics so you can detect slow degradation of example set quality. If the behavioral_consistency_score drops below your defined threshold, block the release and route the report to the prompt engineering team with the specific contradictory example pairs identified in the output. The human-in-the-loop step should only be required when the automated validation produces a no_go decision or when the JSON output fails schema validation after a retry.