Golden Dataset Edge Case Coverage Prompt

This prompt is for an ML engineer or QA lead who already has a golden dataset of test cases and expected outputs but needs to verify that the dataset actually covers the failure modes that matter. The job-to-be-done is not creating new test cases from scratch—it is auditing an existing regression suite for blind spots. You use this prompt when you suspect your dataset is too clean, too narrow, or too focused on happy-path inputs, and you need a systematic gap report before shipping a prompt change. The ideal user has access to the dataset schema, a few representative samples, and a clear understanding of the production failure modes they are most worried about (hallucination, format drift, policy violations, tool misuse).

Do not use this prompt when you have no golden dataset at all—start with the sibling playbook on Golden Dataset Construction. Do not use it when you need to generate the actual missing test cases; this prompt produces a gap analysis and prioritized recommendations, not executable test inputs. It is also a poor fit for one-off prompt debugging where you just need a quick fix. This is a structured audit tool, not a repair tool. The output is most valuable when fed into a sprint planning session or a release-readiness review, where the gap report becomes a checklist for dataset improvement before the next evaluation run.

Before running this prompt, ensure you can provide: the dataset's current schema (what fields exist, what output formats are expected), a representative sample of at least 5–10 existing test cases, and a ranked list of your top 3–5 production risks. The prompt works best when you are explicit about your risk priorities—vague risk statements produce vague gap reports. After receiving the output, validate that the identified gaps are actionable by cross-referencing them against recent production incidents or user-reported failures. If the gap report flags categories you have never seen fail, treat those as hypotheses to verify, not certainties. The next step is to take the prioritized recommendations and feed them into a test-case generation workflow, then re-run your evaluation suite to measure coverage improvement.

The prompt below is designed to be dropped into your evaluation pipeline. It takes a description of your system's behavior, its existing golden dataset, and your risk taxonomy, then produces a structured gap report. The report identifies under-covered edge cases, missing failure modes, and distribution gaps, with each finding prioritized by risk. Use this template as the core instruction for an LLM call that runs before each regression testing cycle, ensuring your test coverage evolves alongside your prompt.

textCopy
You are an ML quality engineer auditing the completeness of a golden dataset used for regression testing an AI system. Your task is to analyze the provided dataset and identify gaps in edge-case coverage, missing failure modes, and distribution imbalances that could allow regressions to reach production undetected.

## SYSTEM UNDER TEST
[SYSTEM_DESCRIPTION]

## GOLDEN DATASET
[DATASET]

## RISK TAXONOMY
[RISK_TAXONOMY]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "summary": "A 2-3 sentence assessment of overall dataset coverage quality.",
  "gaps": [
    {
      "gap_id": "string",
      "category": "missing_edge_case | missing_failure_mode | distribution_gap | ambiguous_label | stale_example",
      "description": "What specific scenario or input pattern is absent or under-represented.",
      "risk_level": "critical | high | medium | low",
      "rationale": "Why this gap is dangerous—what regression could slip through.",
      "recommended_test_case": {
        "input": "A concrete example input that would fill this gap.",
        "expected_behavior": "What the system should do with this input.",
        "assertions": ["Specific, testable conditions that must hold."]
      },
      "affected_instructions": ["Which parts of the system prompt or behavior contract this gap stresses."]
    }
  ],
  "distribution_analysis": {
    "current_balance": "Assessment of category, difficulty, or input-type balance.",
    "recommended_additions": ["Specific categories or input types to add more examples for."]
  }
}

## CONSTRAINTS
- Do not fabricate gaps. Only report gaps you can justify from the provided dataset and system description.
- Prioritize gaps that could mask regressions in high-risk behaviors: safety policy violations, hallucination, instruction leakage, tool misuse, and format non-compliance.
- Each recommended test case must be concrete and immediately usable, not abstract advice.
- If the dataset is too small to analyze meaningfully, state that in the summary and recommend a minimum size before gap analysis is reliable.
- Flag any existing test cases that appear stale, contradictory, or misaligned with the system description.

Adapting the template: Replace [SYSTEM_DESCRIPTION] with a concise summary of what your AI system does, its boundaries, and its critical behavioral contracts. [DATASET] should contain your full golden dataset—inputs, expected outputs, and any metadata like category tags or difficulty labels. [RISK_TAXONOMY] is your team's classification of what constitutes a failure: include categories like hallucination, refusal bypass, format drift, tool misuse, and PII leakage. If you lack a formal taxonomy, provide a bulleted list of the top five failure modes you cannot afford to miss. The output schema is strict JSON; validate it in your harness before accepting the gap report. For high-risk domains, route the generated gap report to a human reviewer before adding recommended test cases to your regression suite.

This prompt is designed to be a batch analysis step in a regression testing pipeline, not a real-time user-facing feature. It takes a serialized golden dataset as input and returns a structured gap report. The implementation harness must handle large input payloads, validate the output schema, and route high-risk findings for human review before new test cases are committed to the test suite. Treat the output as a proposal, not an automated change—every recommended test case should be reviewed by a QA engineer or ML lead before being added to the golden dataset.

Wire the prompt into a scheduled job or CI/CD gate that triggers after any significant update to the golden dataset (e.g., new examples added, schema changes, or domain expansion). The harness should: (1) Serialize the golden dataset into the [GOLDEN_DATASET] placeholder, including input-output pairs, metadata tags, and expected behavior annotations. If the dataset exceeds the model's context window, chunk by category or failure mode and run multiple passes. (2) Call the model with response_format set to a strict JSON schema matching the expected output: an array of gap objects with fields for gap_id, category, description, severity, recommended_test_case, and rationale. (3) Validate the response: confirm all required fields are present, severity values are in the allowed enum (critical, high, medium, low), and recommended test cases are well-formed. (4) Log the raw prompt, response, and validation result for auditability. (5) Route findings with severity: critical or severity: high to a review queue (e.g., a Jira ticket or Slack notification) requiring human approval before ingestion. Low-severity gaps can be auto-appended to a staging test suite with a flag for later review.

Retry logic should be conservative. If validation fails (malformed JSON, missing fields, or schema violations), retry once with a repair prompt that includes the original output and the specific validation errors. If the retry also fails, log the failure and alert the QA team—do not silently discard the run. Model choice matters: use a model with strong structured-output reliability and long-context handling (e.g., GPT-4o, Claude 3.5 Sonnet). Avoid models known to drift on complex schemas. Do not use this prompt as a real-time guard in production inference; it is a pre-release QA tool. The output is a diagnostic artifact, not a runtime decision. After human review, approved test cases should be versioned alongside the golden dataset in the same repository, with a commit message linking back to the gap report run ID.