Example Diversity Scoring Rubric Prompt

This prompt is designed for prompt QA teams and ML engineers who need a quantitative, repeatable method to audit the diversity of a few-shot example set. The primary job-to-be-done is to move beyond a subjective 'looks good to me' review and produce a scored rubric across demographic, linguistic, difficulty, and domain dimensions. Use this when you are preparing a prompt for production, certifying a golden example set, or investigating biased or brittle model behavior that you suspect originates from narrow examples. The ideal user is someone who owns the quality of the prompt's examples and can act on the gap recommendations, such as adding new examples or rebalancing categories.

The prompt requires a complete example set as input, along with a defined context about the application's target users and use cases. It works best when you can provide clear definitions for the dimensions you care about, such as specific demographic axes, language registers, or difficulty tiers. The output is a structured rubric with numerical scores and actionable gap recommendations, not a simple pass/fail check. You should not use this prompt for a single ad-hoc example or when you lack the authority to modify the example set based on the findings. It is also not a replacement for production monitoring; it is a pre-deployment audit tool.

Before running this prompt, ensure you have a clear policy on what demographic or sensitive attributes are relevant and permissible to analyze in your domain. The rubric will flag gaps, but it cannot make ethical judgments about whether a gap is acceptable. Always pair the quantitative scores with a human review of the recommendations, especially when the prompt suggests generating synthetic examples for under-represented groups, as this can introduce authenticity risks. The next step after receiving the scored rubric is to prioritize the high-severity gaps and feed them into your example augmentation or curation workflow, not to blindly generate examples to hit a numerical target.

This prompt template operationalizes the Example Diversity Scoring Rubric. It is designed for prompt QA teams and responsible AI engineers who need a repeatable, structured method to evaluate whether their few-shot examples adequately cover the expected production input space. The prompt instructs the model to act as an evaluator, applying a multi-dimensional rubric to a provided set of examples and returning a scored report. The output is not a simple pass/fail; it is a detailed breakdown of coverage strengths, critical gaps, and specific recommendations for new examples to add.

textCopy
You are an expert evaluator of AI prompt example sets. Your task is to score a provided set of few-shot examples using a multi-dimensional diversity rubric. Your analysis must be quantitative, actionable, and identify concrete gaps.

## INPUT

[EXAMPLES]

## SCORING RUBRIC

For each dimension, assign a score from 1 (Critically Deficient) to 5 (Excellent Coverage). Provide a one-sentence justification for each score.

1.  **Demographic Coverage (1-5):** Does the example set include a balanced representation of names, locales, and personas without stereotyping? Note over- and under-represented groups.
2.  **Linguistic Register Variation (1-5):** Do the examples span formal, informal, technical, and colloquial language registers as expected in production?
3.  **Difficulty Stratification (1-5):** Are there clear easy, moderate, and hard examples? Is there a ceiling effect where all examples are too simple?
4.  **Domain Terminology Coverage (1-5):** Do the examples incorporate key domain-specific jargon, acronyms, and entities from [DOMAIN_GLOSSARY]?
5.  **Edge-Case Representation (1-5):** Do the examples include boundary conditions like null inputs, maximum lengths, and malformed data?

## OUTPUT FORMAT

Return a valid JSON object with the following structure. Do not include any text outside the JSON.

{
  "overall_diversity_score": number (average of the five dimension scores),
  "dimension_scores": {
    "demographic_coverage": { "score": number, "justification": string },
    "linguistic_register_variation": { "score": number, "justification": string },
    "difficulty_stratification": { "score": number, "justification": string },
    "domain_terminology_coverage": { "score": number, "justification": string },
    "edge_case_representation": { "score": number, "justification": string }
  },
  "critical_gaps": [
    {
      "dimension": string,
      "gap_description": string,
      "severity": "HIGH" | "MEDIUM" | "LOW"
    }
  ],
  "recommended_new_examples": [
    {
      "target_dimension": string,
      "example_specification": string (a clear description of the new example to create),
      "priority": "HIGH" | "MEDIUM" | "LOW"
    }
  ]
}

## CONSTRAINTS

- Base your scores strictly on the provided [EXAMPLES].
- If a [DOMAIN_GLOSSARY] is provided, use it as the reference for terminology coverage. If not provided, note its absence as a limitation.
- The "recommended_new_examples" must be specific and directly address the "critical_gaps" you identified.

To adapt this template, replace the [EXAMPLES] placeholder with your full set of formatted few-shot demonstrations. The [DOMAIN_GLOSSARY] is an optional but powerful variable; providing a list of 20-50 key terms forces a precise terminology audit instead of a vague assessment. For high-stakes responsible AI workflows, the raw JSON output should not be the final step. Pipe this output into a secondary validation check that compares the identified gaps against a known list of common failure modes for your specific application. If the rubric is used to gate a release, have two different models or two different temperature settings run the same prompt and compare the overall_diversity_score and critical_gaps arrays as a basic inter-rater reliability check before a human reviews the final recommendation.

The Example Diversity Scoring Rubric prompt is designed to be called programmatically as part of a pre-release prompt QA gate, not as a one-off manual check. In a typical harness, you feed the prompt a JSON array of examples (the [EXAMPLE_SET]) and a reference taxonomy or coverage dimensions list ([DIMENSIONS]), and expect back a structured JSON report with dimension-level scores, gap flags, and actionable recommendations. Because the output is consumed by downstream automation—such as release blockers or coverage dashboards—the harness must enforce strict schema validation, handle malformed outputs gracefully, and log every scoring run for auditability.

Wire the prompt into your CI/CD or evaluation pipeline using a model that supports structured output (JSON mode or function calling). Before calling the model, validate that the [EXAMPLE_SET] input is well-formed: each example must have a unique ID, the text field must be non-empty, and the total token count should be measured to avoid context-window truncation. After receiving the model response, run a JSON schema validator against the expected output shape—fields like dimension_scores (array of objects with dimension, score, gap_flag, recommendation), overall_diversity_score, and inter_rater_notes. If validation fails, implement a single retry with the validation error message appended to the prompt as additional [CONSTRAINTS] context. If the retry also fails, log the raw output and route the run for human review rather than silently proceeding.

For inter-rater reliability checks, run the same prompt against the same example set at least twice (either with different models or with temperature > 0 on the same model) and compare the overall_diversity_score and per-dimension gap_flag agreement. A simple agreement threshold—such as Cohen's kappa > 0.7 on gap flags—can trigger a pass; lower agreement should escalate for human review. Store every scoring run in a versioned registry that links the example set hash, prompt version, model identifier, and scores. This lets you detect rubric drift over time and compare diversity scores across example set iterations. Avoid using this prompt on example sets that exceed 80% of the model's context window; instead, sample or shard the set and aggregate scores with explicit coverage notes in the final report.