Test Report Summarization Prompt

Engineering leads and QA managers use this prompt when they need to extract a consistent, actionable signal from a wall of unstructured test run output. The ideal user is someone responsible for a release decision, a CI pipeline health dashboard, or an incident response channel who receives thousands of lines of logs, stack traces, and duration metrics from tools like Pytest, JUnit, or Go test. The job-to-be-done is not to replace a dedicated test analytics platform, but to act as the reliable extraction layer that turns raw, noisy output into the structured input those platforms, dashboards, and notification systems require. You should use this prompt when you need a machine-readable JSON report that highlights failures, identifies flaky tests, quantifies coverage changes, and flags trend anomalies against a known baseline.

This prompt is designed for post-execution summarization, not for real-time streaming analysis or as a substitute for a test runner's native reporting. It works best when you provide the complete, unredacted test output as [INPUT] along with a [BASELINE] of previous run statistics to enable trend detection. The prompt requires a defined [OUTPUT_SCHEMA] so that the generated report can be ingested programmatically. For example, you might specify a schema that includes failed_tests as an array of objects with name, error_message, and file_path, and flaky_tests as an array of objects with name and consecutive_failure_count. Without a strict schema, the model may produce a human-readable summary that is difficult to parse in a CI pipeline. You should also provide [CONSTRAINTS] such as a maximum token length for error messages or a rule to group failures by module.

Do not use this prompt when the raw test output contains sensitive data that has not been redacted, as the prompt sends the full text to a model. It is also not a replacement for a human on-call engineer when a critical production incident requires immediate, context-rich triage; the prompt summarizes what happened, but it cannot replace the debugging intuition of an experienced developer. Finally, avoid using this prompt for real-time log tailing or for test suites that generate more than the model's context window, unless you implement a pre-processing step to chunk and aggregate the output. For high-risk release gates, always pair the generated summary with a human review step before blocking or promoting a build.

This prompt template is designed to be pasted directly into your AI orchestration layer, test runner, or CI pipeline. It accepts raw test output—such as JUnit XML, pytest logs, or custom JSON—and produces a structured summary that engineering leads and QA managers can act on. The prompt is built to extract signal from noise: it identifies failures, flags flaky tests, notes coverage changes, and detects trend anomalies without burying the reader in passing-test logs.

textCopy
You are a test report analyst. Your task is to read the raw test run output provided in [TEST_OUTPUT] and produce a structured summary in the exact JSON format specified in [OUTPUT_SCHEMA].

Follow these rules strictly:
- Report only actionable findings. Do not list passing tests unless they are part of a trend anomaly.
- If a test has failed in the last [FLAKE_WINDOW] runs, flag it as flaky and include its failure rate.
- Compare coverage metrics against the baseline in [BASELINE_COVERAGE] and highlight any regression greater than [COVERAGE_THRESHOLD]%.
- If [TREND_DATA] is provided, identify any statistically significant deviations in failure rate, duration, or flakiness.
- For every failure, include the test name, suite, error message, and the file path if available.
- If the output contains more than [MAX_FAILURES] failures, group them by error type and report only the top [MAX_FAILURES] distinct root causes.
- Do not hallucinate fixes. Do not guess at root causes beyond what the error messages and stack traces provide.
- If the test output is empty or unparseable, return a valid JSON object with an error key set to a description of the problem.

[TEST_OUTPUT]:
[PLACEHOLDER]

[OUTPUT_SCHEMA]:
[PLACEHOLDER]

[BASELINE_COVERAGE]:
[PLACEHOLDER]

[TREND_DATA]:
[PLACEHOLDER]

[FLAKE_WINDOW]: [PLACEHOLDER]
[COVERAGE_THRESHOLD]: [PLACEHOLDER]
[MAX_FAILURES]: [PLACEHOLDER]

To adapt this template for your environment, replace each square-bracket placeholder with data from your test runner and CI context. The [TEST_OUTPUT] placeholder should receive the raw, unprocessed output from your test execution step—do not pre-filter or summarize it. The [OUTPUT_SCHEMA] placeholder must contain a strict JSON Schema definition that your downstream systems expect; this is your primary defense against malformed summaries. The [BASELINE_COVERAGE] and [TREND_DATA] placeholders are optional but strongly recommended for teams that want coverage regression alerts and anomaly detection. If you omit them, remove the corresponding instructions from the prompt to avoid confusing the model. The numeric placeholders [FLAKE_WINDOW], [COVERAGE_THRESHOLD], and [MAX_FAILURES] should be tuned to your team's tolerance for noise: set [MAX_FAILURES] low enough to keep summaries scannable, and set [FLAKE_WINDOW] to match your CI history retention.

Before deploying this prompt into a production pipeline, validate its output against a golden dataset of known test runs. At minimum, confirm that the model correctly identifies every failure in a run with known failures, correctly reports zero failures for a clean run, and produces valid JSON that passes your schema validator even when the input is malformed. If the summary will be surfaced in an incident response workflow or a release gate, add a human review step before the summary is treated as authoritative. A model can miss a subtle failure mode or misattribute a flaky test, and the cost of a bad summary in a high-stakes release decision justifies the review overhead.

The Test Report Summarization Prompt is designed to be called programmatically after test execution completes in a CI pipeline or test orchestration system. The prompt expects raw test output—typically JUnit XML, JSON test result files, or structured log output—as its primary input. Before calling the model, the application layer must parse the raw test results, extract the fields required by the prompt template (test suite name, total count, pass/fail/skip counts, failure details, duration, and any flaky test history), and assemble them into the [TEST_RUN_OUTPUT] placeholder. Do not pass raw, unparsed megabytes of test output directly to the model; pre-process into the structured summary fields the prompt expects to keep token usage predictable and avoid overwhelming the context window with noise.

Wire the prompt into a post-test hook in your CI system (GitHub Actions, GitLab CI, Jenkins, CircleCI) or a test result aggregator (ReportPortal, Allure, BuildPulse). After test execution, the harness should: (1) collect test result files from the test runner, (2) parse and normalize the results into the prompt's input schema, (3) call the LLM with the prompt template and structured input, (4) validate the output against the expected JSON schema (fields: summary, failures[], flaky_tests[], coverage_change, trend_anomalies[], recommended_actions[]), and (5) on validation failure, retry once with a repair prompt that includes the schema violation details. Log every invocation—input hash, model, latency, token count, output, and validation result—to your prompt observability store for debugging and trend analysis. For high-risk releases (production deployment gates, compliance-tested systems), route the generated summary to a human reviewer via a Slack notification, PR comment, or review queue before the pipeline proceeds. The human approval step should be a configurable gate: enable it for release branches, disable it for feature branch CI where latency matters more than review.

Model selection matters for this workflow. For routine CI runs where cost and speed dominate, use a fast, cheaper model (Claude Haiku, GPT-4o-mini, or a fine-tuned small model) with the structured output mode enabled. For release-candidate runs where accuracy and anomaly detection quality are critical, escalate to a more capable model (Claude Sonnet, GPT-4o) and consider providing additional context: the last three test run summaries for trend comparison, the git diff since the last passing run, and any known flaky test annotations from your test management system. Avoid calling the model when the test run is trivially green (zero failures, zero flaky tests, no coverage change); short-circuit with a pre-generated summary to save cost and latency. The harness should also enforce a timeout and token budget: if the model call exceeds 30 seconds or the output exceeds 2000 tokens, log a warning and fall back to a template-based summary generated from the parsed test results without the LLM.

Common failure modes to instrument for: the model hallucinates a failure that doesn't exist in the input (validate that every failure.test_name appears in the parsed input), the model misses a real failure (compare failures[].length against the parsed failure count and flag discrepancies), the model produces vague trend language like 'coverage slightly changed' without the actual delta (enforce numeric fields in coverage_change), or the model recommends actions that don't match the failure signatures (cross-check recommended_actions against failures[].error_type for plausibility). Build these checks into your validation layer as post-processing assertions, not as additional prompt instructions, to keep the prompt focused and the validation testable. When validation fails, log the specific check that failed and the raw model output for debugging—never silently accept a summary that fails structural or semantic validation.