Regression Test Report Generation Prompt

Engineering leads and QA managers need to communicate regression test results to stakeholders who do not read raw test logs. This prompt converts structured regression run output into a multi-section report with an executive summary, pass/fail breakdown, regression highlights, severity classification, and trend comparison against prior runs. Use it when you have raw test results from a CI/CD pipeline or evaluation harness and need a decision-ready document for release gates, sprint reviews, or compliance audits.

This prompt assumes the input data is already aggregated and structured—it does not execute tests, adjudicate ambiguous failures, or replace human judgment for high-severity regressions that require root cause analysis. The ideal input is a JSON object containing test case results with fields like test_id, status, severity, category, previous_status, and duration_ms. The output is a markdown report with clearly labeled sections that a VP of Engineering or compliance auditor can scan in under two minutes. Do not use this prompt when you need real-time test execution, when the input data is raw and unstructured (use a separate normalization step first), or when the report must include root cause analysis that requires access to source code, logs, or system telemetry beyond the test results themselves.

Before wiring this into a release pipeline, define your severity taxonomy explicitly in the [CONSTRAINTS] block—for example, critical means user-facing data loss, high means core functionality broken, medium means non-blocking deviation, and low means cosmetic. The prompt uses these definitions to classify and prioritize regressions. For compliance-sensitive workflows, always route the generated report through a human review step before it reaches external stakeholders. The prompt produces a decision-ready artifact, not a decision itself.

This prompt template transforms raw regression test results into a structured report suitable for engineering leads, QA managers, and release stakeholders. It expects pre-processed test run data—not raw log files—and produces a report with an executive summary, pass/fail breakdown, regression highlights, severity classification, and trend comparison against a prior baseline. The template uses square-bracket placeholders that your orchestration layer must populate before sending to the model.

textCopy
You are a QA engineering lead writing a regression test report for stakeholders.

Generate a structured report from the regression test run data below. Follow the output schema exactly. Do not invent test cases, scores, or trends not present in the input data.

## INPUT DATA

### Current Run
[TEST_RUN_DATA]

### Prior Baseline Run (for trend comparison)
[BASELINE_RUN_DATA]

### Test Suite Metadata
- Suite name: [SUITE_NAME]
- Prompt version under test: [PROMPT_VERSION]
- Model: [MODEL_ID]
- Execution timestamp: [RUN_TIMESTAMP]
- Total test cases executed: [TOTAL_CASES]

## OUTPUT SCHEMA

Return valid JSON with this structure:
{
  "report_metadata": {
    "suite_name": "string",
    "prompt_version": "string",
    "model": "string",
    "run_timestamp": "string",
    "generated_at": "ISO-8601 timestamp"
  },
  "executive_summary": {
    "verdict": "PASS | CONDITIONAL_PASS | FAIL",
    "summary": "2-4 sentence plain-English summary of results",
    "risk_statement": "1-2 sentences on deployment risk if applicable"
  },
  "pass_fail_summary": {
    "total_cases": "integer",
    "passed": "integer",
    "failed": "integer",
    "pass_rate_percent": "float",
    "baseline_pass_rate_percent": "float or null if no baseline",
    "pass_rate_delta": "float (current minus baseline)"
  },
  "regression_highlights": [
    {
      "test_case_id": "string",
      "test_name": "string",
      "category": "string",
      "severity": "CRITICAL | HIGH | MEDIUM | LOW",
      "current_result": "PASS | FAIL",
      "baseline_result": "PASS | FAIL | NOT_PRESENT",
      "failure_summary": "1-sentence description of what broke",
      "suspected_cause": "instruction_misinterpretation | format_drift | hallucination | over_refusal | context_mishandling | unknown"
    }
  ],
  "severity_breakdown": {
    "critical": "integer",
    "high": "integer",
    "medium": "integer",
    "low": "integer"
  },
  "category_breakdown": {
    "[category_name]": { "passed": "integer", "failed": "integer" }
  },
  "trend_comparison": {
    "new_regressions": "integer (cases that passed in baseline but fail now)",
    "fixed_regressions": "integer (cases that failed in baseline but pass now)",
    "unchanged_passing": "integer",
    "unchanged_failing": "integer",
    "new_cases": "integer (cases not present in baseline)",
    "trend_direction": "IMPROVING | STABLE | DEGRADING",
    "trend_narrative": "1-2 sentence interpretation of the trend"
  },
  "drill_down_sections": [
    {
      "section_title": "string",
      "focus": "string (e.g., 'CRITICAL failures', 'format_drift regressions', 'category X breakdown')",
      "findings": ["bullet-point finding"],
      "recommended_actions": ["actionable recommendation"]
    }
  ],
  "release_recommendation": {
    "recommendation": "PROCEED | PROCEED_WITH_CAUTION | HOLD | ROLLBACK",
    "rationale": "2-3 sentences explaining the recommendation",
    "blocking_issues": ["issue description if HOLD or ROLLBACK"]
  }
}

## CONSTRAINTS
- Only report on test cases present in the input data.
- If no baseline data is provided, set baseline fields to null and skip trend comparison.
- Severity must be derived from the test case metadata and failure impact, not invented.
- The executive summary verdict must be PASS only if zero CRITICAL or HIGH failures exist.
- Use CONDITIONAL_PASS when only MEDIUM or LOW failures exist and no blocking regressions.
- Use FAIL when any CRITICAL or HIGH regression exists.
- Do not hallucinate drill-down sections; generate them only for meaningful failure clusters.
- All counts must reconcile: pass + fail = total; severity counts must sum to total failures.

To adapt this template, replace each square-bracket placeholder with actual data from your test runner. The [TEST_RUN_DATA] and [BASELINE_RUN_DATA] placeholders expect pre-processed JSON arrays of test case results—each containing at minimum test_case_id, test_name, category, result, and any failure metadata your runner produces. If you don't have a baseline run yet, pass an empty array for [BASELINE_RUN_DATA] and the model will skip trend comparison. The output schema is strict JSON; validate it against the schema before forwarding the report to stakeholders. For high-stakes releases, add a human review step that verifies the severity classifications and release recommendation against the raw test data before the report leaves the engineering team.

The regression test report generation prompt is designed to sit at the end of an automated evaluation pipeline, consuming structured test results and producing a stakeholder-ready report. The harness should invoke this prompt only after all test cases have executed and raw per-case results are available. Do not feed raw model outputs directly into this prompt—pre-process them into a structured results payload that includes test case IDs, expected behaviors, actual outcomes, pass/fail status, failure categories, severity scores, and prior-run comparisons. The prompt expects this data in a specific schema, so the harness must validate the input structure before calling the model. If the input is malformed or missing required fields, the harness should abort early with a clear error rather than generating a misleading report.

Wire the prompt into your pipeline as a post-execution step with the following guardrails. First, validate the input payload against a JSON schema that enforces required fields: run_id, timestamp, total_cases, passed, failed, skipped, cases (array of objects with id, status, severity, category, prior_status). Second, implement a retry loop with exponential backoff (max 3 attempts) for model call failures, but do not retry on validation failures—those indicate a harness bug, not a transient error. Third, log every invocation with the run_id, model version, prompt version, token counts, and latency for observability. Fourth, if the report will be shared with stakeholders outside the engineering team, route the generated report through a human review queue before distribution. The review step should flag any severity misclassifications, missing regression highlights, or trend comparison errors. For model choice, use a model with strong structured output capabilities and a context window large enough to hold the full results payload plus the prompt—GPT-4o or Claude 3.5 Sonnet are appropriate defaults. Avoid small or local models for this task because the report requires consistent formatting, accurate arithmetic on pass/fail counts, and faithful representation of the input data without hallucination.

The most common production failure mode is the model summarizing results incorrectly—swapping pass/fail counts, omitting high-severity regressions, or fabricating trend comparisons when prior-run data is missing. To catch this, implement a post-generation validation step that extracts key numbers from the generated report (pass count, fail count, regression count) and compares them against the input payload. If the numbers don't match within a tolerance of zero, flag the report for human review and log the discrepancy. A secondary failure mode is the model producing a report that is technically correct but omits the executive summary or drill-down sections requested in the output schema. Use a structural validator that checks for the presence of required sections by heading text or markdown structure. If sections are missing, retry once with a stronger constraint instruction, then escalate to human review. Do not ship regression reports directly to stakeholders without at least automated numeric validation—stakeholders make decisions based on these reports, and a hallucinated pass rate can erode trust in the entire evaluation pipeline.