Regression Test Suite Runner Prompt Template

This prompt is designed for CI/CD pipeline owners and evaluation engineers who need to run a batch of test cases against a specific prompt version and receive structured, machine-readable pass/fail results. The core job-to-be-done is automated regression detection: you have a golden dataset of known inputs and expected behaviors, and you need to verify that a prompt change hasn't introduced regressions before shipping to production. The ideal user is someone who already maintains a test suite and needs a reliable executor that produces per-case verdicts with failure categorization, not someone designing tests from scratch or evaluating a single output in isolation.

Use this prompt when you have a versioned prompt under test, a golden dataset with at least 20–50 test cases covering happy path, edge case, and known failure modes, and a defined evaluation rubric or pass/fail criteria. The prompt acts as the test executor: it receives the prompt under test, the set of test cases, and the evaluation criteria, then returns per-case verdicts with failure categorization such as instruction misinterpretation, format drift, hallucination, or over-refusal. This is a batch execution prompt, not a single-case evaluator. It assumes your test suite is already constructed and validated—if you're still building your golden dataset, start with the Golden Dataset Construction Prompt Template or Test Case Generation from Prompt Spec Prompt instead.

Do not use this prompt for designing evaluation rubrics, comparing two prompt versions side by side, or making release decisions. Those workflows have dedicated prompts in this content group: Prompt Version Comparison Evaluation Prompt for diff reports, Prompt Release Gate Decision Prompt for go/no-go automation, and Rubric Design and Scoring Prompts for building the evaluation criteria this runner consumes. Also avoid this prompt when you need semantic equivalence grading rather than strict pass/fail—use the Semantic Equivalence Evaluation Prompt Template for that. If you're running tests across multiple models simultaneously, the Multi-Model Regression Comparison Prompt is a better fit. This prompt is the execution engine, not the strategy layer. Wire it into your CI/CD pipeline after you've established your test suite, defined your criteria, and configured your timeout and retry handling in the application harness.

This template is the core instruction set for a regression test runner. It expects a batch of test cases—each containing an input, expected behavior, and metadata—along with the prompt version under test. The model's job is to execute each case, compare the output against the expected behavior using the provided evaluation criteria, and return a structured result for every test case. The output is designed to be consumed by a CI/CD pipeline, so consistency and machine-readability are non-negotiable.

textCopy
You are a regression test executor. Your task is to run a batch of test cases against a prompt version and produce per-case pass/fail results with failure categorization.

## PROMPT UNDER TEST
[PROMPT_VERSION]

## TEST CASES
[TEST_CASES]

## EVALUATION CRITERIA
[EVALUATION_CRITERIA]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "run_id": "[RUN_ID]",
  "prompt_version": "[PROMPT_VERSION_ID]",
  "executed_at": "[TIMESTAMP]",
  "summary": {
    "total": <integer>,
    "passed": <integer>,
    "failed": <integer>,
    "skipped": <integer>,
    "pass_rate": <float between 0 and 1>
  },
  "results": [
    {
      "case_id": "<string>",
      "status": "passed | failed | skipped | error",
      "input": "<original test input>",
      "actual_output": "<output produced by prompt under test>",
      "expected_behavior": "<expected behavior from test case>",
      "failure_category": "<null if passed, otherwise one of: instruction_misinterpretation | format_drift | hallucination | over_refusal | context_mishandling | constraint_violation | output_omission | other>",
      "failure_detail": "<null if passed, otherwise specific explanation of what went wrong>",
      "severity": "<null if passed, otherwise: critical | high | medium | low>",
      "execution_time_ms": <integer>
    }
  ],
  "execution_metadata": {
    "timeout_cases": ["<case_ids that timed out>"],
    "retry_cases": ["<case_ids that required retries>"],
    "execution_order": "<sequential | parallel>",
    "total_execution_time_ms": <integer>
  }
}

## CONSTRAINTS
- Execute every test case exactly once unless a retry is explicitly warranted due to a transient error.
- If a test case input is malformed or missing required fields, mark it as "skipped" and include the reason in failure_detail.
- If the prompt under test produces no output or times out, mark the case as "error" with failure_category "other".
- Do not skip a case because the expected behavior is ambiguous. Instead, mark it as "failed" with failure_category "other" and explain the ambiguity.
- For each failure, assign exactly one failure_category. Choose the most specific category that applies.
- Severity must be assigned based on: critical (data corruption, safety violation, or irreversible action), high (incorrect output that would cause user-visible harm), medium (degraded quality but not harmful), low (cosmetic or minor deviation).
- Return ONLY valid JSON. No markdown fences, no commentary, no additional text.

## EXECUTION INSTRUCTIONS
1. For each test case in [TEST_CASES], run the input through [PROMPT_VERSION] and capture the output.
2. Compare the actual output against the expected_behavior using [EVALUATION_CRITERIA].
3. Classify any failure using the failure_category taxonomy.
4. Assign severity to each failure.
5. Aggregate results into the summary object.
6. Return the complete JSON object.

To adapt this template, replace the square-bracket placeholders programmatically before each run. [PROMPT_VERSION] should contain the full prompt text under test, not just a version identifier. [TEST_CASES] should be a JSON array of test case objects, each with at minimum case_id, input, and expected_behavior fields. [EVALUATION_CRITERIA] should define how to judge pass/fail—this could be an exact-match rule, a semantic equivalence threshold, or a rubric. If your pipeline uses a separate LLM judge for evaluation, wire that judge's output into this template rather than asking the runner to perform both execution and evaluation simultaneously. For high-stakes pipelines, always log the raw output of this template before parsing, and implement a validation step that checks the JSON schema before the results enter your release gate logic.

The Regression Test Suite Runner prompt is designed to be called programmatically, not used in a chat interface. In a production harness, you iterate over a batch of test cases, render the prompt template with each case's [INPUT], [EXPECTED_BEHAVIOR], and [EVALUATION_CRITERIA], send the request to the model, and collect structured pass/fail results with failure categories. The harness is responsible for execution order, timeout handling, concurrency control, and aggregating per-case results into a suite-level report. Treat each test case as an independent evaluation unit—do not batch multiple test cases into a single prompt call, as that degrades failure attribution and makes retries coarse-grained.

Wire the prompt into your test runner with a thin orchestration layer that handles: timeout enforcement (set a per-request timeout of 30–60 seconds; if exceeded, mark the case as TIMEOUT and continue), structured output parsing (expect a JSON object with pass, failure_category, and explanation fields; validate the schema before accepting the result), retry logic (retry once on parse failures or model errors with exponential backoff; do not retry on clear FAIL results as that wastes compute and masks non-determinism), and result aggregation (accumulate per-case results into a summary with pass rate, failure category distribution, and a list of failed case IDs for triage). For CI/CD integration, emit a machine-readable JSON report and set a non-zero exit code when the pass rate drops below a configured threshold or when any CRITICAL severity regression is detected. Log the full prompt, raw model response, and parsed result for every test case so that debugging a specific failure does not require re-running the suite.

Model choice matters for this harness. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller or older models that may drift on the evaluation criteria or produce malformed JSON under batch load. If your test suite exceeds 100 cases, implement concurrency with a semaphore (e.g., 5–10 parallel requests) and track rate limits. For high-stakes release gates, add a human review step for cases categorized as CRITICAL or UNCLEAR before blocking deployment. The harness should never auto-approve a release when critical regressions exist; it should surface them for operator judgment. Store test run artifacts—prompt version, model identifier, timestamp, per-case results, and aggregate report—in a versioned artifact store so that every release decision is auditable.