Edge Case Coverage Gate Evaluation Prompt

This prompt is a formal release gate for QA leads and platform engineers who need to verify that a prompt under test holds its behavior on boundary inputs, rare scenarios, and stress cases before it reaches production. The job-to-be-done is producing a structured, machine-readable coverage report that includes per-edge-case results, newly discovered failure modes, and a coverage gap analysis. The ideal user has a defined catalog of known edge cases and needs an automated pass/fail decision to integrate into a CI/CD pipeline or a manual release approval process. The required context is a stable prompt version, a curated edge case catalog with expected behaviors, and a clear definition of what constitutes a passing result.

Use this prompt when you are operating a formal release process where prompt changes must clear a documented quality bar. It is appropriate when you have already completed exploratory testing and have a stable set of regression cases. The prompt is designed to be wired into an evaluation harness that can execute each edge case, compare the output against expected behavior, and aggregate the results into a coverage report. The output should be consumed by automated gating logic or a human release manager who needs a summary of residual risk. Do not use this prompt for ad-hoc testing without a defined edge case catalog, or when the prompt under test is still in early prototyping and its expected behavior is undefined. In those scenarios, the evaluation will produce noisy, unactionable results because the ground truth is not yet established.

Before invoking this gate, ensure your edge case catalog is versioned and stored in a system of record. Each entry must include the input, the expected behavior or output constraints, and a severity classification. The evaluation harness should log per-case results, capture raw model outputs, and record any validator failures. After the gate runs, review the coverage gap analysis to identify categories of inputs that are underrepresented in your catalog. A passing gate result does not guarantee production safety; it only confirms that the prompt meets the coverage threshold you defined. Always pair this gate with other release criteria, such as semantic drift detection and golden dataset pass rates, before promoting a prompt to production.

This prompt template is designed to be copied directly into your evaluation harness. It instructs the model to act as a QA gate, systematically testing a target prompt against a catalog of edge cases. The output is a structured coverage report that flags failures, discovers new failure modes, and quantifies coverage gaps. Before using this template, ensure you have a defined edge case catalog and a clear understanding of the expected behavior for each case.

textCopy
You are a QA engineer performing an edge case coverage evaluation for a target prompt. Your goal is to determine if the target prompt behaves correctly on boundary inputs, rare scenarios, and stress cases.

## TARGET PROMPT TO EVALUATE
[PROMPT_UNDER_TEST]

## EDGE CASE CATALOG
You will be provided with a list of edge cases. For each case, you must execute the target prompt with the specified input and evaluate the output.

[EDGE_CASE_CATALOG]
<!-- Expected format:
- case_id: [ID]
  description: [DESCRIPTION]
  input: [INPUT]
  expected_behavior: [EXPECTED_BEHAVIOR]
  severity: [CRITICAL|HIGH|MEDIUM|LOW]
-->

## EVALUATION CONSTRAINTS
[CONSTRAINTS]
<!-- Example: "The target prompt must never produce PII. It must refuse harmful requests. It must output valid JSON." -->

## OUTPUT SCHEMA
You must output a single JSON object conforming to this schema:
{
  "evaluation_id": "string",
  "target_prompt_version": "string",
  "coverage_summary": {
    "total_cases": "integer",
    "passed": "integer",
    "failed": "integer",
    "blocked_or_skipped": "integer",
    "coverage_percentage": "float"
  },
  "per_case_results": [
    {
      "case_id": "string",
      "status": "PASS|FAIL|BLOCKED",
      "actual_output_summary": "string",
      "failure_mode": "string | null",
      "severity": "CRITICAL|HIGH|MEDIUM|LOW"
    }
  ],
  "new_failure_modes_discovered": [
    {
      "failure_mode": "string",
      "description": "string",
      "trigger_condition": "string",
      "observed_in_case_ids": ["string"]
    }
  ],
  "coverage_gap_analysis": {
    "uncovered_categories": ["string"],
    "recommended_new_cases": [
      {
        "description": "string",
        "input": "string",
        "expected_behavior": "string",
        "severity": "CRITICAL|HIGH|MEDIUM|LOW"
      }
    ]
  },
  "overall_pass": "boolean"
}

## INSTRUCTIONS
1. Iterate through each case in the [EDGE_CASE_CATALOG].
2. For each case, simulate the execution of the [PROMPT_UNDER_TEST] with the provided `input`.
3. Compare the simulated output against the `expected_behavior` and [CONSTRAINTS].
4. If the behavior is correct, mark the case as PASS.
5. If the behavior is incorrect, mark the case as FAIL and classify the `failure_mode`. If the failure mode is not already in your list, add it to `new_failure_modes_discovered`.
6. If you cannot execute the case due to missing information or a dependency, mark it as BLOCKED.
7. After processing all cases, analyze the results for coverage gaps. Identify categories of inputs or behaviors not covered by the existing catalog and suggest new cases.
8. Set `overall_pass` to `true` only if there are zero CRITICAL or HIGH severity failures.
9. Output ONLY the JSON object. Do not include any other text.

To adapt this template, replace the square-bracket placeholders with your specific data. The [PROMPT_UNDER_TEST] is the system or user prompt you are evaluating. The [EDGE_CASE_CATALOG] should be populated with a structured list of your known edge cases. The [CONSTRAINTS] block is critical for defining non-negotiable rules, such as output format requirements or safety policies. After execution, the resulting JSON report should be programmatically validated against the defined schema before being used in a CI/CD gate.

The Edge Case Coverage Gate Evaluation Prompt is designed to run as a blocking gate in a CI/CD pipeline before a prompt version is promoted to staging or production. The harness must supply a structured edge case catalog—each case containing an input, expected behavior constraints, and a severity classification—along with the prompt version under test. The model returns a coverage report with per-case pass/fail results, newly discovered failure modes, and a coverage gap analysis. This output must be parsed, validated, and compared against predefined coverage thresholds before the gate can pass.

To implement the harness, start by maintaining an edge case catalog as a versioned JSON or YAML file in your repository. Each entry should include an id, input, expected_behavior (described as constraints rather than exact outputs), severity (blocking, warning, or informational), and optional tags for categorization. The harness script reads this catalog, iterates over cases, and calls the LLM with the prompt template for each case or in batched groups. After receiving the model's coverage report, run a structural validator to confirm the JSON schema matches the expected output shape: a coverage_summary object with total_cases, passed, failed, coverage_percentage, and a cases array with per-case results. If the output fails schema validation, retry once with a repair prompt that includes the validation error; if it fails again, fail the gate with a malformed-output flag.

Coverage threshold enforcement is the core of the gate logic. Define minimum pass rates per severity level—for example, 100% pass required on blocking cases, 90% on warning cases. The harness must also detect newly discovered failure modes that were not in the original catalog and flag them for human review. Log every evaluation run with the prompt version hash, model version, timestamp, and full coverage report to an audit store. For high-risk domains, require a manual approval step when new failure modes are discovered or when coverage drops below threshold. Wire the harness into your CI/CD platform (GitHub Actions, GitLab CI, Jenkins) so that a failed gate blocks the merge or deployment. Finally, periodically review the edge case catalog itself—remove cases that no longer represent real risk, add cases discovered in production traces, and rebalance severity classifications to prevent gate fatigue.