Pre-Freeze Regression Test Suite Generation Prompt

This prompt is designed for QA engineers and AI release managers who are preparing to freeze a system prompt and need a comprehensive, structured regression test suite. The primary job-to-be-done is to transform a candidate system prompt, its behavioral specification, and a list of known edge cases into a detailed test catalog. Each test case in the catalog must include specific input variants, clearly defined expected behaviors, and unambiguous pass/fail criteria. This process is critical for verifying that freezing the prompt will not introduce regressions into production behavior, ensuring that the system continues to handle both common and adversarial inputs as intended.

Use this prompt only when you have a stable, version-controlled candidate system prompt and a well-documented behavioral specification. The required inputs are precise: the full text of the candidate system prompt, a structured description of its intended behavior across different scenarios, and an explicit list of known edge cases or failure modes you've previously encountered. The output is not a simple list of ideas; it's a machine-readable test catalog, often formatted as JSON, that can be directly fed into an automated testing harness. For example, a test case for a refusal policy might include an adversarial user input, the expected refusal response containing a specific key phrase, and a pass criterion that checks for the absence of disallowed content. Do not use this prompt for initial prompt drafting, for testing non-deterministic creative outputs like story generation where a strict rubric is inappropriate, or when the behavioral specification is still in flux.

Before executing this prompt, ensure you have completed a production behavior baseline capture to compare against. The generated test suite should be treated as a living artifact; after initial generation, you must review it for completeness and adversarial coverage gaps. A common failure mode is generating tests that only verify the 'happy path' and miss edge cases that are obvious to a human reviewer. To mitigate this, always pair this prompt with a manual adversarial review and a coverage audit. The next step after generation is to run this test suite in your CI/CD pipeline and treat any failures as a blocking condition for the prompt freeze.

The following prompt template is designed to be dropped into your test generation pipeline. It instructs the model to act as a QA architect and produce a catalog of test cases, each with input variants, expected behaviors, and explicit pass/fail criteria. The template uses square-bracket placeholders that you must replace with your specific system prompt, domain context, and risk profile before execution.

codeCopy
You are a senior QA architect specializing in AI system behavior. Your task is to generate a comprehensive regression test suite for the system prompt provided below. This test suite will be used as a final quality gate before the system prompt version is frozen.

**SYSTEM PROMPT UNDER TEST:**
[SYSTEM_PROMPT_TEXT]

**DOMAIN AND RISK CONTEXT:**
[DOMAIN_CONTEXT]

**KNOWN EDGE CASES AND FAILURE MODES:**
[KNOWN_EDGE_CASES]

**OUTPUT SCHEMA:**
Generate a JSON array of test case objects. Each object must conform to this structure:
{
  "test_id": "string, unique identifier",
  "category": "string, one of: [CATEGORY_LIST]",
  "description": "string, what this test validates",
  "input_variant": "string, the exact user input to test",
  "expected_behavior": "string, the required model response characteristics",
  "pass_criteria": ["string, specific, verifiable condition"],
  "failure_severity": "string, one of: 'critical', 'high', 'medium', 'low'",
  "adversarial_flag": "boolean, true if this is an injection or red-team test"
}

**CONSTRAINTS:**
- Generate at least [MIN_TEST_COUNT] test cases.
- Ensure coverage across all categories in [CATEGORY_LIST].
- Include adversarial tests for prompt injection, role escape, and policy bypass.
- Include boundary tests for empty input, maximum length input, and special characters.
- For each test, the pass_criteria must be specific enough for an automated eval or a human reviewer to judge consistently.
- Do not include tests that are duplicates or trivial variations.

To adapt this template, start by replacing [SYSTEM_PROMPT_TEXT] with the exact system instructions you intend to freeze. The [DOMAIN_CONTEXT] placeholder should be filled with a brief description of the product, its users, and the risk tolerance (e.g., 'healthcare appointment scheduling, high risk'). Populate [KNOWN_EDGE_CASES] with any failure patterns your team has already observed in development. The [CATEGORY_LIST] should be a set of strings relevant to your system, such as ["role_adherence", "refusal_policy", "tool_calling", "output_format", "safety", "injection_defense"]. Set [MIN_TEST_COUNT] to a number that reflects the complexity of your system prompt; a starting point is 30-50 for a moderately complex assistant. After generating the suite, you must review the output for completeness before it enters your test harness.

This prompt is designed to be called programmatically as part of a CI/CD or release-gating workflow, not as a one-off chat interaction. The application layer is responsible for supplying the frozen system prompt under test, a structured specification of expected behaviors, and any known edge-case categories. The model returns a structured test catalog that downstream test runners parse and execute. Because the output directly gates a production freeze decision, the harness must validate the generated test suite before any tests are run against the live system.

Wire the prompt into a Python or TypeScript service that constructs the [SYSTEM_PROMPT_UNDER_TEST], [BEHAVIORAL_SPEC], and [EDGE_CASE_CATEGORIES] inputs from your prompt registry and spec repository. Call the model with response_format set to a strict JSON schema matching the expected output shape—this prevents free-text drift that would break downstream parsers. On response, run a structural validator that checks: (1) every test case has a unique ID, input, expected behavior, and pass/fail criteria; (2) adversarial coverage categories from [EDGE_CASE_CATEGORIES] each have at least one corresponding test; (3) no test case duplicates another's input and expected behavior pair. If validation fails, retry once with the validation errors appended as [PREVIOUS_VALIDATION_ERRORS] in the prompt context. After two failures, escalate to a human QA lead for manual review rather than looping indefinitely. Log every generation attempt, validation result, and retry decision to your prompt observability system for post-freeze audit trails.

For model choice, prefer a model with strong instruction-following and structured output reliability—GPT-4o or Claude 3.5 Sonnet are appropriate defaults. Avoid smaller or older models that may drop adversarial categories or produce inconsistent JSON under the schema constraint. The prompt does not require tool use or retrieval-augmented generation because all necessary context is supplied inline. Do not treat the generated test suite as final; it is a starting point that a QA engineer should review for blind spots, especially around novel failure modes the spec didn't anticipate. The harness should flag any test suite where more than 20% of generated tests are flagged as low-confidence by the model, and it should block the freeze gate if the adversarial coverage check reveals gaps the retry loop couldn't close.