Coverage Gap Analysis Prompt for Test Suites

This prompt is for QA leads and ML engineers who need to audit the completeness of a golden test suite before it gates a prompt release. It takes your prompt requirements, user personas, failure mode taxonomy, and existing dataset inventory as input, then produces a structured gap report. The report identifies missing scenario categories, assigns priority scores based on risk and usage frequency, and provides generation templates you can feed directly into a synthetic data pipeline. Use this before a release cycle when you suspect blind spots in your regression coverage, or after a production incident reveals a failure mode your tests never covered.

To use this effectively, you must provide four concrete inputs: a written specification of what the prompt under test is supposed to do, a list of user personas with their typical intents and edge behaviors, a taxonomy of known failure modes (from past incidents, red-team exercises, or heuristic analysis), and an inventory of your existing golden dataset with scenario tags. The prompt works by cross-referencing these inputs to find uncovered intersections—for example, a persona that never appears in a high-risk failure scenario, or a requirement clause with zero test cases exercising it. The output includes a priority score for each gap, calculated from the severity of the uncovered failure mode and the frequency of the affected persona or intent. Each gap also comes with a generation template: a structured spec you can pass to a synthetic data generator to create the missing test case.

Do not use this prompt as a substitute for production monitoring or user feedback loops. It analyzes static coverage, not real-world distribution shifts. If your failure mode taxonomy is incomplete—for instance, you have never catalogued a prompt injection vector—this prompt cannot invent it. Pair this analysis with trace-based regression detection and adversarial fuzzing for a complete QA posture. After running the gap report, feed the highest-priority generation templates into your synthetic data pipeline, then re-run the coverage analysis to confirm the gaps are closed before promoting the prompt.

This template is the core instruction set for a coverage gap analysis. It expects a structured inventory of your existing test cases, a definition of your prompt's requirements, target user personas, and a known failure mode taxonomy. The model's job is to cross-reference these inputs and identify what is missing, assign priorities, and suggest templates for new test cases that would close the gaps. Use this prompt when you suspect your regression suite has blind spots but need a systematic way to prove it.

textCopy
You are a test coverage auditor for AI prompt systems. Your task is to analyze a provided test suite inventory against prompt requirements, user personas, and a failure mode taxonomy. Identify coverage gaps, assign priority scores, and produce generation templates for missing scenarios.

INPUTS:
- Test Suite Inventory: [TEST_SUITE_INVENTORY]
- Prompt Requirements: [PROMPT_REQUIREMENTS]
- User Personas: [USER_PERSONAS]
- Failure Mode Taxonomy: [FAILURE_MODE_TAXONOMY]

OUTPUT_SCHEMA:
{
  "gap_report": [
    {
      "gap_id": "string",
      "category": "missing_requirement | uncovered_persona | untested_failure_mode",
      "description": "string",
      "priority_score": 0.0-1.0,
      "rationale": "string",
      "generation_template": {
        "input_description": "string",
        "expected_behavior": "string",
        "failure_mode_triggered": "string | null",
        "persona_targeted": "string | null",
        "requirement_tested": "string | null"
      }
    }
  ],
  "summary": {
    "total_gaps_found": "integer",
    "critical_gaps": "integer",
    "coverage_percentage_estimate": "float"
  }
}

CONSTRAINTS:
- Only report gaps where no existing test case provides adequate coverage.
- Priority scores must reflect both likelihood of occurrence and severity of impact.
- Generation templates must be specific enough for a QA engineer to implement immediately.
- Do not invent test cases that exist in the inventory. Cite the gap.

To adapt this template, replace the square-bracket placeholders with your actual data. The [TEST_SUITE_INVENTORY] should be a structured list of your existing test cases, each with an ID, input, expected output, and tags. The [PROMPT_REQUIREMENTS] is the functional spec for your prompt. The [USER_PERSONAS] defines who uses the system and how. The [FAILURE_MODE_TAXONOMY] is your catalog of known ways the prompt can break. If you lack a formal taxonomy, start with a simple list of categories like 'hallucination,' 'refusal,' 'format error,' and 'prompt injection' and iterate from there. The output is a JSON object you can feed directly into a test generation pipeline or a manual review queue.

Before running this in production, validate the output against the OUTPUT_SCHEMA contract. A common failure mode is the model hallucinating gaps that are already covered; mitigate this by including unique test case IDs in your inventory and asking the model to cite the specific gap. For high-risk domains, always have a QA lead review the gap report before generating new tests. The priority scores are a starting point for triage, not a final decision. Wire this prompt into a scheduled job that runs whenever your test suite, requirements, or failure taxonomy changes to keep coverage analysis continuous.

The Coverage Gap Analysis prompt is designed to run as a batch or on-demand audit step, not as a real-time user-facing call. Wire it into your test suite maintenance workflow so it fires after golden dataset updates, new prompt requirement documents are published, or a failure mode taxonomy is revised. The prompt expects three structured inputs: a golden dataset manifest, a requirements or persona specification, and a failure mode taxonomy. These should be assembled by your application layer before the model call, not left to the user to paste in manually. Use a model with strong reasoning and long-context handling—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate starting points. Avoid small or fast models here; gap analysis requires cross-referencing multiple documents and generating structured taxonomies, which degrades sharply under context truncation or weak instruction following.

Build a thin harness around the prompt that validates the output schema before accepting results. The expected output is a JSON object containing a gaps array, each with category, description, priority_score, affected_requirements, and a generation_template for filling the gap. Your harness should validate that every gap references at least one requirement ID present in the input specification and that priority scores fall within the defined 1–10 range. If validation fails, retry once with the validation errors appended to the prompt as feedback. If the second attempt also fails, log the failure and route the malformed output to a human review queue rather than silently discarding it. For high-stakes test suites—safety-critical systems, regulated domains, or customer-facing AI—require a human QA lead to approve the gap report before new test cases are generated from the templates. Log every run with the prompt version, input hashes, model used, and validation outcome so you can trace coverage decisions back to specific audits.

Do not treat this prompt as a one-shot fix for test coverage. The gap report is a planning artifact, not an executable test suite. After approval, feed each generation_template into a separate test case generation prompt or manual authoring workflow. Track which gaps are filled and when, and re-run the coverage analysis after major requirement changes or every N sprints. Avoid wiring this directly into CI/CD as a gate; a coverage gap report should inform prioritization, not block a deployment. The most common failure mode is the model hallucinating requirement IDs or inventing gap categories that sound plausible but don't map to your actual documents. Your schema validator catches the ID mismatch, but a human reviewer should also spot-check category relevance before acting on the report.