Golden Dataset Coverage Gap Analysis Prompt

Use this prompt when you already have a golden dataset and a stable prompt specification, but you suspect the test suite is incomplete. The job-to-be-done is auditing your evaluation coverage: identifying input categories, edge cases, and failure modes that your current tests don't exercise. The ideal user is an evaluation lead or prompt engineer who owns a regression suite and needs evidence to prioritize test expansion before the next prompt release. Required context includes the prompt template under test, its documented constraints and expected behaviors, and the full golden dataset with metadata tags.

This prompt is not a test case generator. It does not produce new test inputs or expected outputs—it analyzes what's missing from what already exists. Don't use it when you have no golden dataset at all; start with the Golden Dataset Construction Prompt instead. Don't use it when you need to run tests or compare prompt versions; those workflows belong to the Regression Test Suite Runner or Prompt Version Comparison prompts. Don't use it when your prompt specification is still in flux—coverage analysis against a moving target produces noise, not signal. The prompt works best when the spec is versioned and the dataset has consistent metadata like input category, difficulty, and expected behavior tags.

The output is a structured gap report, not a pass/fail judgment. Expect prioritized recommendations with rationale, not just a list of missing cases. The prompt will flag underrepresented categories, boundary conditions that lack test coverage, and failure modes the dataset doesn't probe. It may also identify redundant tests that over-represent certain patterns while leaving others untested. Before running this analysis, ensure your golden dataset metadata is clean—inconsistent tags or missing category labels will degrade the quality of the gap detection. If your evaluation pipeline is high-stakes (safety-critical outputs, regulated domains, or customer-facing automation), treat the gap report as an input to human review, not an automated gate. A human should validate that the recommended expansions make sense before committing engineering time to building new test cases.

After running this prompt, the natural next step is to feed the prioritized gap recommendations into the Golden Dataset Expansion from Production Failures prompt or the Test Case Generation from Prompt Spec prompt, depending on whether you're filling gaps from real incidents or from spec analysis. Avoid the temptation to close every gap at once—prioritize by risk and frequency of the uncovered input category in production traffic.

This section provides the core prompt template for auditing a golden dataset against a prompt specification. The template is designed to be copied directly into your evaluation pipeline, with placeholders that you replace with your actual prompt under test, dataset samples, and domain constraints. The prompt instructs the model to act as an evaluation auditor, systematically comparing what your test suite covers against what your prompt specification demands. It produces a structured gap report with prioritized recommendations, making it suitable for automated coverage checks in CI/CD or manual review before a release.

textCopy
You are an evaluation coverage auditor. Your task is to analyze a golden dataset against a prompt specification and identify gaps in test coverage.

## PROMPT SPECIFICATION
[PROMPT_SPECIFICATION]

## GOLDEN DATASET SAMPLE
[DATASET_SAMPLE]

## ANALYSIS INSTRUCTIONS
1. Parse the prompt specification to extract all explicit and implicit requirements, constraints, edge cases, and behavioral expectations.
2. Review the golden dataset sample to understand what input categories, difficulty levels, and failure modes are currently represented.
3. Identify coverage gaps across these dimensions:
   - **Missing Input Categories**: Input types, domains, or formats that the specification handles but the dataset lacks.
   - **Underrepresented Edge Cases**: Boundary conditions, adversarial inputs, or rare scenarios with insufficient examples.
   - **Untested Failure Modes**: Specified error handling, refusal conditions, or constraint violations without corresponding test cases.
   - **Difficulty Imbalance**: Overrepresentation of easy cases and underrepresentation of complex, multi-step, or ambiguous inputs.
   - **Demographic or Linguistic Gaps**: Missing languages, dialects, or user personas that the specification should serve.
4. For each gap, assign a severity score (Critical, High, Medium, Low) based on:
   - Likelihood of encountering the gap in production.
   - Impact severity if the prompt fails on that input.
   - Whether the gap masks a known failure mode from past incidents.
5. Produce prioritized expansion recommendations with concrete example inputs that would fill each gap.

## OUTPUT FORMAT
Return a JSON object with this exact structure:
{
  "specification_summary": "Brief summary of what the prompt specification requires",
  "dataset_profile": {
    "total_examples_reviewed": [NUMBER],
    "input_categories_covered": ["category1", "category2"],
    "difficulty_distribution": {"easy": [NUMBER], "medium": [NUMBER], "hard": [NUMBER]},
    "represented_failure_modes": ["mode1", "mode2"]
  },
  "coverage_gaps": [
    {
      "gap_id": "GAP-001",
      "category": "Missing Input Category | Underrepresented Edge Case | Untested Failure Mode | Difficulty Imbalance | Demographic or Linguistic Gap",
      "description": "What is missing or underrepresented",
      "severity": "Critical | High | Medium | Low",
      "severity_justification": "Why this severity level applies",
      "recommended_examples": ["example input 1", "example input 2"],
      "estimated_examples_needed": [NUMBER]
    }
  ],
  "prioritized_recommendations": [
    {
      "rank": [NUMBER],
      "gap_id": "GAP-XXX",
      "action": "What to add to the dataset",
      "rationale": "Why this is prioritized at this rank"
    }
  ],
  "overall_coverage_score": [NUMBER 0-100],
  "critical_gaps_count": [NUMBER]
}

## CONSTRAINTS
- Only report gaps you can justify from the specification. Do not invent requirements.
- If the dataset sample is too small to draw reliable conclusions, note this limitation and recommend a full-dataset review.
- For each gap, provide at least one concrete example input that would address it.
- If no critical gaps exist, acknowledge this but still report lower-severity gaps.
- Do not recommend removing existing test cases; focus only on expansion.

To adapt this template, replace [PROMPT_SPECIFICATION] with the full text of the prompt you are evaluating, including its system instructions, output format requirements, and any behavioral constraints. Replace [DATASET_SAMPLE] with a representative sample of your golden dataset entries—include at least 20-30 examples spanning different categories if possible. If your dataset is large, provide a stratified sample rather than random rows. The output JSON schema is designed to be machine-readable for automated coverage dashboards, but the severity justifications and concrete example recommendations also support manual review during release readiness checks. For high-risk domains where coverage gaps could cause safety failures, route Critical-severity findings to a human reviewer before accepting the analysis.

The Golden Dataset Coverage Gap Analysis Prompt is designed to run as a pre-release or periodic audit step in your evaluation pipeline, not as a real-time production call. Wire it into a scheduled job or a manual trigger in your prompt versioning workflow—typically after a golden dataset has been updated or before a major prompt release. The prompt expects a prompt specification (system prompt, constraints, expected behavior) and a golden dataset manifest (input categories, edge cases, difficulty distribution) as inputs. It returns a structured gap report with prioritized expansion recommendations. Because this analysis informs downstream test case generation, the output should be reviewed by an evaluation lead before new test cases are created.

For implementation, wrap the prompt call in a function that validates the output schema before accepting results. The expected output is a JSON object with a coverage_gaps array, each entry containing category, gap_description, severity, recommended_test_count, and rationale. Use a JSON schema validator to confirm field presence, types, and enum values for severity (Critical, High, Medium, Low). If validation fails, retry once with the validation error message appended to the prompt as a correction hint. Log both the raw output and the validation result for auditability. For model choice, a reasoning-capable model (such as Claude 3.5 Sonnet or GPT-4o) is recommended because the analysis requires cross-referencing the prompt specification against dataset composition—a task that benefits from structured reasoning. Set temperature to 0 or near-zero to maximize reproducibility across runs.

The most common failure mode is the model producing vague gap descriptions that lack actionable specificity (e.g., 'missing edge cases' without identifying which edge cases). Mitigate this by including a CONSTRAINTS placeholder in the prompt that requires each gap to reference a specific input category or test scenario from the dataset manifest. A second failure mode is hallucinated gaps—the model inventing missing categories that are actually present in the dataset. To catch this, implement a post-processing check that cross-references each reported gap against the dataset manifest. Flag any gap that references a category not present in the manifest for human review. Store gap analysis results alongside the dataset version and prompt version in your evaluation registry so you can track coverage improvement over time and avoid re-analyzing already-addressed gaps.