Example Diversity Scoring Prompt Template

This prompt is a pre-deployment audit tool for prompt engineers and AI reliability teams who maintain libraries of few-shot examples. Its job is to answer one question before examples ship in production prompts: does this example set actually cover the input patterns and output types the system will encounter? You provide a complete example set as input, and the prompt produces a structured diversity report with similarity scores, cluster assignments, coverage maps, and actionable gap flags. Run it before major prompt version releases, after adding new examples, or when production eval scores show unexpected variance across input categories.

The ideal user is someone responsible for prompt quality in a shipped product—not someone experimenting in a notebook. You need a complete example set in a structured format (JSON lines, CSV, or labeled text blocks), a defined set of input categories or output types you expect the system to handle, and a way to act on the gaps the report surfaces. The prompt does not select examples at runtime; it audits them offline. Use it when you can afford a batch analysis step before deployment. Do not use this prompt when you need real-time example retrieval, when your example set is too small to cluster meaningfully (fewer than 20–30 examples), or when you lack a clear taxonomy of expected input patterns to measure coverage against.

After running this prompt, you should have a concrete list of coverage gaps to fill, redundant examples to prune, and distribution skews to correct. The next step is to act on the report: add examples for uncovered input patterns, remove near-duplicate examples that waste context budget, and rebalance the example distribution so no single input category dominates. Avoid the trap of running the audit and then shipping the same example set unchanged—the report is only valuable if it changes what goes into production.

The following prompt template is designed to be copied directly into your prompt management system, IDE, or orchestration code. It instructs the model to assume the role of an example set auditor and analyze a provided set of few-shot examples for coverage gaps, redundancy, and cluster distribution. The prompt uses square-bracket placeholders—such as [EXAMPLE_SET], [TASK_DESCRIPTION], and [OUTPUT_SCHEMA]—that you must replace with your actual data before inference. No other part of the prompt should contain unresolved tokens.

textCopy
You are an example set auditor. Your job is to analyze a set of few-shot examples for diversity, coverage, and redundancy.

## Task Description
[TASK_DESCRIPTION]

## Example Set
[EXAMPLE_SET]

## Instructions
1. For each example, assign a cluster label based on input pattern similarity. Use short, descriptive labels.
2. Compute a pairwise similarity score (0.0 to 1.0) between every pair of examples. Use semantic similarity of the input patterns, not surface string overlap.
3. Identify clusters that are overrepresented (more than [MAX_EXAMPLES_PER_CLUSTER] examples) and flag them for potential deduplication.
4. Identify input patterns or output types that are missing from the example set based on the task description. List these as coverage gaps.
5. For each coverage gap, suggest a synthetic example that would fill it.
6. Produce a final diversity score from 0.0 (completely homogeneous) to 1.0 (maximally diverse) based on cluster count, spread, and gap severity.

## Output Schema
Return a JSON object with this exact structure:
{
  "cluster_assignments": [
    {"example_id": "string", "cluster_label": "string"}
  ],
  "similarity_matrix": [
    {"example_id_1": "string", "example_id_2": "string", "score": number}
  ],
  "overrepresented_clusters": [
    {"cluster_label": "string", "example_count": number, "recommendation": "string"}
  ],
  "coverage_gaps": [
    {"gap_description": "string", "suggested_example": {"input": "string", "output": "string"}}
  ],
  "diversity_score": number,
  "summary": "string"
}

## Constraints
- Do not invent examples that were not in the provided set.
- If the example set is empty, return a diversity score of 0.0 and note the empty set in the summary.
- If all examples belong to a single cluster, the diversity score must be 0.0.
- Do not include examples that violate the task description in your coverage gap suggestions.

To adapt this template for your own use, replace [TASK_DESCRIPTION] with a clear, one-paragraph explanation of what the examples are meant to teach the model. Replace [EXAMPLE_SET] with your actual few-shot examples, formatted consistently—typically as a JSON array of objects with id, input, and output fields. Set [MAX_EXAMPLES_PER_CLUSTER] to your desired threshold for flagging overrepresentation; a value of 3 is a reasonable default for most sets under 50 examples. If your example set uses a different schema, adjust the example_id field references in the output schema to match your actual identifiers. Before deploying this prompt in a production pipeline, run it against a known, hand-audited example set and compare the model's diversity score and cluster assignments against your own assessment to calibrate expectations.

The Example Diversity Scoring prompt is not a one-off audit tool—it should be wired into a repeatable pipeline that runs whenever your few-shot example library changes. The prompt consumes a set of examples and produces a structured diversity report with similarity scores, cluster assignments, and coverage maps. To make this production-grade, you need a harness that validates the input set, enforces the output schema, retries on malformed responses, and surfaces actionable gaps rather than raw scores. The goal is to catch example drift, redundancy, and blind spots before they degrade model behavior in production.

Start by defining the input contract. The prompt expects a JSON array of examples, each with at minimum an id, input, and output field. Before calling the model, validate that the input array is non-empty, that no two examples share the same id, and that the total token count of the serialized examples fits within your model's context window minus the prompt template overhead. Use a preflight check: if the example set exceeds 80% of the available context budget, either truncate the set with a warning or split the scoring across multiple calls using stratified sampling. For model choice, prefer a model with strong structured output capabilities and a context window large enough to hold your full example library plus the scoring instructions. If your library contains hundreds of examples, consider batching by output type or input pattern category to keep each scoring run focused and under budget.

The output must conform to a strict schema. Define a JSON schema that includes: a similarity_scores array of pairwise comparisons with example_id_a, example_id_b, and score; a clusters array with cluster_id, member_ids, and cluster_description; and a coverage_map object keyed by input pattern category or output type, each containing covered_example_ids, gap_description, and severity. After the model responds, validate the JSON structure, check that all referenced example IDs exist in the input set, and confirm that similarity scores fall within the expected range. If validation fails, retry once with the validation errors appended to the prompt as a correction hint. If the second attempt also fails, log the failure, flag the run for human review, and do not update the diversity report. For high-stakes prompt libraries where example quality directly affects user-facing behavior, require human approval before removing or replacing any example flagged as redundant or stale.

Integrate this into your CI pipeline or scheduled evaluation job. On each run, compare the new diversity report against the previous baseline. Raise an alert if a previously covered input pattern now shows a gap, if a new cluster of near-duplicate examples appears, or if the overall coverage score drops below a configured threshold. Store each report with a timestamp and the example library version so you can trace coverage changes over time. Avoid the trap of treating diversity scores as the final answer—always have a human review step for gap remediation, because the model can identify missing patterns but cannot reliably generate the right examples to fill those gaps without domain expertise.