Prompt Smoke Test Suite Generator Prompt Template

This prompt is designed for DevOps and MLOps engineers who need a fast, automated gate that runs on every commit to a prompt repository. The primary job-to-be-done is catching catastrophic regressions—such as malformed JSON output, complete instruction disregard, or refusal spikes—before they consume expensive CI minutes in a full evaluation run. The ideal user is integrating prompt QA into an existing GitHub Actions, GitLab CI, or Jenkins pipeline and needs a lightweight test generator that can analyze a git diff between the current and proposed prompt versions, cross-reference a golden dataset, and select the minimal subset of test cases most likely to detect breakage. Required context includes access to the raw prompt diff, a labeled golden dataset with expected outputs, and a defined output schema or contract that the application expects downstream.

Use this prompt when the cost of running a full regression suite on every push is prohibitive, but the risk of a broken prompt reaching a staging environment is unacceptable. The smoke test suite it generates should execute in under two minutes and serve as the first quality gate in a multi-stage pipeline. It is particularly effective for teams managing multiple prompt variants or those practicing trunk-based development where prompt changes are frequent and small. The prompt assumes you already have a curated golden dataset; it does not help you build one from scratch. It also assumes your application has a well-defined output contract—if your prompt outputs free-form text with subjective quality criteria, a semantic evaluation step will still be required after the smoke test passes.

Do not use this prompt as a replacement for a comprehensive regression suite or canary analysis. It is a filter, not a guarantee. If the prompt diff touches safety-critical instructions, introduces new tool definitions, or modifies the system prompt's behavioral boundaries, you should always follow a passing smoke test with a full evaluation run. Similarly, if your golden dataset has known coverage gaps—such as missing edge cases for a newly added feature—the smoke test may produce false confidence. In these scenarios, treat a passing smoke test as a prerequisite for further testing, not a release approval. After generating the suite, wire the output directly into your eval harness and configure your CI pipeline to block merges on smoke test failure, routing failures to the prompt author for immediate investigation.

This prompt template is designed to be dropped directly into your CI/CD pipeline's evaluation stage. It takes a prompt version diff and a curated golden dataset as inputs and produces a structured smoke test execution report. The goal is not exhaustive coverage but high-signal regression detection: identifying the smallest set of tests most likely to break given the specific changes introduced. You provide the diff, the reference data, and your risk tolerance; the prompt returns a ready-to-execute test plan with pass/fail thresholds and a schema for reporting results back to your gate logic.

codeCopy
You are a prompt regression test architect. Your task is to generate a minimal, high-signal smoke test suite from a prompt version diff and a golden dataset.

## INPUTS

### PROMPT VERSION DIFF
[DOCUMENT_DIFF]

### GOLDEN DATASET
[DATASET]

### CONSTRAINTS
- Maximum test cases: [MAX_TESTS]
- Risk tolerance: [RISK_TOLERANCE] (low, medium, high)
- Required coverage areas: [COVERAGE_AREAS]

## OUTPUT SCHEMA

Return a JSON object with the following structure:

{
  "test_suite_id": "string",
  "generated_at": "ISO8601 timestamp",
  "diff_summary": {
    "changed_sections": ["string"],
    "instruction_additions": ["string"],
    "instruction_removals": ["string"],
    "constraint_modifications": ["string"],
    "risk_assessment": "low | medium | high"
  },
  "test_cases": [
    {
      "test_id": "string",
      "test_name": "string",
      "input": "object | string",
      "expected_output_shape": "object",
      "pass_criteria": [
        {
          "criterion_type": "exact_match | schema_match | semantic_similarity | refusal_check | tool_call_match | content_assertion",
          "threshold": "number | string",
          "description": "string"
        }
      ],
      "priority": "critical | high | medium",
      "regression_target": "string describing what specific change this test guards against",
      "source_row_indices": ["number"]
    }
  ],
  "execution_config": {
    "parallel_execution": true,
    "timeout_seconds": [TIMEOUT],
    "retry_on_failure": [RETRIES],
    "stop_on_first_critical_failure": true
  },
  "pass_thresholds": {
    "critical_pass_rate": [CRITICAL_THRESHOLD],
    "overall_pass_rate": [OVERALL_THRESHOLD],
    "max_allowed_regression_flags": [MAX_REGRESSION_FLAGS]
  }
}

## INSTRUCTIONS

1. Parse the diff to identify every meaningful change: added instructions, removed constraints, modified output schemas, changed tool definitions, altered refusal policies, and reordered sections.
2. For each change category, select the minimum number of golden dataset rows that exercise that specific behavior. Prefer rows that historically caused failures or edge cases.
3. For each selected row, define pass criteria that would detect if the change broke expected behavior. Use exact_match for deterministic outputs, schema_match for structured outputs, semantic_similarity for free text, refusal_check for safety changes, and tool_call_match for function-calling prompts.
4. Assign priority based on blast radius: critical if the change affects all outputs or safety behavior, high if it affects a specific output field or tool, medium if it affects phrasing or non-functional behavior.
5. Set pass thresholds based on the declared risk tolerance. Low tolerance demands 100% critical pass rate and >=95% overall. Medium tolerance allows 95% critical and >=90% overall. High tolerance allows 90% critical and >=85% overall.
6. If the diff introduces no meaningful behavioral changes, return an empty test suite with a note explaining why no tests are needed.

Return ONLY valid JSON. No markdown fences, no commentary.

To adapt this template, start by replacing the input placeholders. [DOCUMENT_DIFF] should contain a unified diff or side-by-side comparison of the old and new prompt versions. [DATASET] should be a JSON array of input-output pairs from your golden dataset, each with a row index for traceability. [MAX_TESTS] controls the suite size—start with 10-15 for a fast smoke test, increase to 25-30 for higher confidence. [RISK_TOLERANCE] directly adjusts the pass/fail thresholds in the output. [COVERAGE_AREAS] lets you scope the suite to specific concerns like tool calling, refusal behavior, or output formatting. The output schema is designed to be machine-readable by your CI/CD gate logic: parse the JSON, iterate over test_cases, execute each against the new prompt version, and compare results against the pass_criteria. If any critical test fails or overall pass rates drop below thresholds, block the deployment. Wire the execution_config into your test runner to handle timeouts and retries automatically.

The Prompt Smoke Test Suite Generator is designed to be called programmatically inside a CI/CD step, not run by hand. You feed it a structured diff of the prompt change and a reference to your golden dataset, and it returns a machine-readable test suite definition. The goal is to produce a minimal, high-signal set of tests that gate the change before it reaches staging or production. This prompt is the first stage in a multi-step gate: it generates the test plan, and downstream harnesses execute those tests against the new prompt version and report pass/fail results.

To wire this into a pipeline, wrap the prompt in a script or containerized step that accepts three inputs: a prompt diff (the output of diff between the current and proposed prompt versions), a golden dataset path (a file or database reference containing input-output pairs with expected behaviors), and a risk profile (a JSON object specifying the blast radius, affected workflows, and any known brittle areas). The script should call the model with these inputs, parse the JSON response against a strict schema, and validate that the returned test suite contains required fields: test_cases (array of input, expected_behavior, and assertion_type), pass_threshold (minimum pass rate, typically 0.95 or higher), and execution_report_schema (the shape of the report the downstream harness must produce). If the model returns malformed JSON or missing fields, retry once with a repair prompt that includes the validation error. After two failures, fail the pipeline step and alert the on-call engineer.

The output of this prompt becomes the input to your test execution harness. Store the generated test suite as a versioned artifact alongside the prompt diff and commit SHA. This artifact is what your downstream eval runner consumes. Do not skip validation: a malformed test suite that passes silently is worse than a loud failure. For high-risk prompt changes (those touching safety policies, refusal behavior, or regulated outputs), require a human to review the generated test suite before execution proceeds. The review step should confirm that the generated tests cover the blast radius and that the pass threshold is appropriate. After review, the pipeline should execute the tests, compare results against the threshold, and produce a structured report matching the execution_report_schema. Wire that report into your promotion gate: if the pass rate meets the threshold, proceed; if not, block the change and route the failure report to the prompt engineer for debugging.