Test Case Generation from Prompt Spec Prompt

This prompt is for QA engineers and prompt developers who need to build a regression test suite before a prompt ships to production. The job-to-be-done is converting a prompt template and its behavioral specification into a diverse, structured set of test cases that cover happy path, edge case, adversarial, and boundary inputs. You should use this when you have a stable prompt spec—meaning the system instructions, tool definitions, output schema, and constraints are documented—and you need to prove that future changes won't silently break expected behavior. The ideal user has read the prompt spec, understands the intended use case, and can provide concrete examples of correct behavior to seed the generation.

This approach works best when the prompt spec is explicit about what the model should and shouldn't do. You'll need to supply the raw prompt template, a description of expected behaviors, known failure modes, and any output schema or tool contracts. The generator will produce test cases with inputs, expected behaviors, coverage rationale, and difficulty tags. Do not use this prompt when the spec is vague or still evolving—garbage specs produce garbage test cases. It's also inappropriate for prompts that rely heavily on real-time retrieval or external tool state that can't be mocked in a static test case. For those, pair this with a mock harness or use the Regression Test Suite Runner instead.

After generating test cases, you must review them for coverage gaps and correctness before adding them to a golden dataset. The generator can miss domain-specific edge cases or produce expected behaviors that contradict unwritten conventions. Treat the output as a strong first draft, not a final test suite. Next, feed the reviewed cases into the Golden Dataset Construction Prompt to structure them for automated regression runs, and use the Golden Dataset Coverage Gap Analysis Prompt to audit what you might have missed.

This template takes a target prompt specification—its instructions, constraints, output schema, and risk level—and produces a structured set of test cases. Each test case includes an input, expected behavior, coverage rationale, and difficulty classification. The goal is to generate a test suite that covers happy paths, edge cases, adversarial inputs, and boundary conditions before the target prompt ever reaches production.

codeCopy
You are a QA engineer designing regression tests for an AI prompt before deployment.

You will receive a prompt specification that includes the prompt template, its constraints, its expected output schema, and its risk level. Your job is to generate a diverse set of test cases that would catch regressions if the prompt behavior changes.

## INPUT

**Prompt Template:**
[PROMPT_TEMPLATE]

**Constraints:**
[CONSTRAINTS]

**Output Schema:**
[OUTPUT_SCHEMA]

**Risk Level:** [RISK_LEVEL]

**Additional Context:**
[CONTEXT]

## INSTRUCTIONS

Generate test cases across these categories:

1. **Happy Path (minimum 3 cases):** Standard inputs that should produce clean, correct outputs following all constraints.
2. **Edge Cases (minimum 5 cases):** Inputs at the boundaries of what the prompt should handle—empty fields, maximum lengths, ambiguous phrasing, near-boundary values.
3. **Adversarial Inputs (minimum 3 cases):** Inputs designed to trick the prompt into violating its constraints, leaking instructions, or producing harmful outputs.
4. **Boundary Conditions (minimum 3 cases):** Inputs that test the limits of the output schema—missing optional fields, extra fields, type mismatches, enum violations.
5. **Format Stress (minimum 2 cases):** Inputs that might cause malformed outputs, markdown leakage, or schema non-conformance.

For each test case, provide:
- **test_id:** A unique identifier (e.g., TC-001)
- **category:** One of the categories above
- **difficulty:** easy | medium | hard
- **input:** The exact input to feed into the prompt template
- **expected_behavior:** What the prompt should do with this input, referencing specific constraints
- **expected_output_shape:** Key fields or structure that should appear in a correct output
- **coverage_rationale:** Why this test case matters—what failure mode it catches
- **failure_indicators:** What to look for if this test case fails (e.g., missing field, hallucinated value, refusal when it should answer)

## OUTPUT FORMAT

Return a JSON object with a "test_cases" array and a "coverage_summary" object:

```json
{
  "test_cases": [
    {
      "test_id": "string",
      "category": "happy_path | edge_case | adversarial | boundary_condition | format_stress",
      "difficulty": "easy | medium | hard",
      "input": "string",
      "expected_behavior": "string",
      "expected_output_shape": ["field_name"],
      "coverage_rationale": "string",
      "failure_indicators": ["string"]
    }
  ],
  "coverage_summary": {
    "total_cases": 0,
    "categories_covered": ["string"],
    "notable_gaps": ["string"],
    "recommended_additions": ["string"]
  }
}

RULES

• Generate at least 15 test cases total.
• Do not duplicate test scenarios—each case should test a distinct failure mode.
• For RISK_LEVEL "high," include at least 5 adversarial cases and note any safety-critical gaps.
• For RISK_LEVEL "low," you may reduce adversarial cases to 2 but must still cover edge cases thoroughly.
• If the prompt template contains placeholders like [INPUT] or [CONTEXT], treat those as variables and generate concrete values for each test case.
• Flag any constraints that are untestable without human judgment in the coverage_summary.notable_gaps field.

Adapt this template by replacing the square-bracket placeholders with your actual prompt specification. The [PROMPT_TEMPLATE] should contain the full prompt you intend to test, including its own placeholders. The [CONSTRAINTS] field should list explicit rules, forbidden behaviors, format requirements, and policy boundaries. The [OUTPUT_SCHEMA] should describe the expected JSON structure, field types, and required fields. Set [RISK_LEVEL] to "low," "medium," or "high" to control the adversarial testing depth. Use [CONTEXT] to provide domain-specific information that helps the test generator understand what realistic inputs look like. After generating test cases, review the coverage_summary for gaps and iterate on the prompt specification if critical scenarios are missing. For high-risk prompts, always have a human reviewer validate the generated test cases before adding them to your regression suite.

The Test Case Generation from Prompt Spec Prompt is designed to run as a pre-deployment step in a prompt release pipeline. It takes a prompt template and its constraints as input and produces a structured set of test cases before the prompt ever touches production traffic. The harness should treat this as a batch generation job: one invocation per prompt version, with the output fed directly into a regression test suite runner. The generated test cases become the golden dataset that gates future prompt changes.

Wire the prompt into an application by building a thin orchestration layer that: (1) loads the prompt spec and constraint document from your version control system, (2) populates the [PROMPT_SPEC] and [CONSTRAINTS] placeholders, (3) calls the model with temperature set to 0.7 to encourage diversity across edge cases, and (4) parses the JSON output into a structured test case array. Validate each generated test case against a schema that checks for required fields (input, expected_behavior, category, difficulty, coverage_rationale). Reject and retry any case missing these fields. Log every generation run with the prompt version hash, model used, timestamp, and count of cases per category. For high-risk domains, insert a human review step where a QA engineer spot-checks adversarial and boundary cases before they enter the regression suite. Store approved test cases in a versioned test repository with a unique run ID that links back to the prompt version they were generated for.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid smaller models that may collapse edge case diversity into repetitive patterns. If generating more than 20 test cases, split into multiple calls with different seed prompts to prevent mode collapse. Implement a deduplication step after generation: compare new test cases against existing golden dataset entries using semantic similarity to avoid redundant tests. The harness should also track coverage metrics—count of happy path, edge case, adversarial, and boundary cases—and flag if any category falls below a minimum threshold. Finally, wire the output into your regression runner so that every generated test case becomes an enforceable gate in CI/CD. Do not deploy a prompt version until its generated test suite passes against the candidate prompt.