Test Failure Triage Prompt Template

This prompt is designed for CI/CD pipeline operators and engineering teams who need to classify test failures by root cause category at scale. Instead of writing brittle rule-based classifiers or manually inspecting every failed job, you provide labeled examples of failure signatures—stack traces, error messages, and log snippets—and the model learns to categorize new failures into buckets like flaky test, environment issue, code regression, or dependency breakage. The primary job-to-be-done is reducing mean-time-to-diagnosis by routing failures to the right owner before a human opens the logs.

Use this prompt when you have a stream of test failures from automated pipelines and you need consistent, explainable triage decisions. It works best when you can supply 8–15 diverse examples covering your most common failure categories, including edge cases like timeout-induced flakes versus assertion-driven regressions. The prompt is not a replacement for flaky test detection algorithms or deterministic log parsers—it complements them by handling ambiguous failures where pattern matching alone fails. Do not use this prompt for real-time production incident response where latency is measured in seconds; it is designed for post-build triage workflows where a few seconds of inference time is acceptable.

Before deploying, you must validate the prompt against a golden set of 50–100 labeled failures to measure category-level precision and recall. Pay special attention to flakiness misclassification—the most common production failure mode is labeling a flaky infrastructure timeout as a code regression. If your pipeline produces failures that don't match any known category, the prompt should route them to an 'unclassified' bucket for human review, not force a wrong label. Wire the output into your alerting or ticketing system only after you've confirmed that ownership routing hints map to real teams and that the eval pass rate exceeds your organization's triage accuracy baseline.

This prompt template is designed to be copied directly into your application or testing harness. It uses square-bracket placeholders for all dynamic inputs, allowing you to swap in the specific test failure log, your team's categories, and custom routing rules without rewriting the core instruction set. The template relies on a dense set of few-shot examples to teach the model the difference between an infrastructure flake, a product code regression, a test environment issue, and a genuine product bug, which is often more reliable than a long list of abstract rules.

codeCopy
You are a test failure triage specialist. Your task is to analyze a CI/CD test failure log and classify it into one of the predefined root cause categories. You must also assess the likelihood of flakiness and suggest the correct engineering owner. Use only the provided examples to guide your classification.

INPUT:
[TEST_FAILURE_LOG]

OUTPUT_SCHEMA:
{
  "classification": "[ROOT_CAUSE_CATEGORY]",
  "confidence": [CONFIDENCE_SCORE_BETWEEN_0_AND_1],
  "is_flaky": [TRUE_OR_FALSE],
  "flakiness_rationale": "[BRIEF_EVIDENCE_FROM_LOG]",
  "suggested_owner": "[TEAM_OR_INDIVIDUAL]",
  "key_evidence": "[QUOTED_LOG_LINE_OR_SUMMARY]"
}

CONSTRAINTS:
- Classification must be one of: [LIST_OF_CATEGORIES]
- If confidence is below [CONFIDENCE_THRESHOLD], set classification to "NEEDS_HUMAN_TRIAGE".
- Do not invent evidence. Quote directly from the log.

EXAMPLES:
[FEW_SHOT_EXAMPLES]

---

Now, analyze the INPUT and return only the JSON object.

To adapt this template, start by replacing [LIST_OF_CATEGORIES] with your team's actual labels, such as INFRA_FLAKE, PRODUCT_REGRESSION, TEST_BUG, and ENVIRONMENT_MISCONFIG. The [FEW_SHOT_EXAMPLES] placeholder is critical; you should insert at least 3-5 contrasting examples that show the model how to distinguish a timeout caused by a network blip (a flake) from a timeout caused by a new database index (a regression). For high-risk deployments, add a [RISK_LEVEL] constraint that forces a human review if the failure occurs on a release branch. The final step is to validate the model's JSON output against the OUTPUT_SCHEMA before routing the result to your notification system.

This prompt is designed to be called as a step in your CI/CD pipeline immediately after a test suite run produces a failure log. The harness should extract the raw failure output from your test runner (e.g., JUnit XML, pytest logs, or Go test output), truncate it to fit the model's context window, and inject it into the [FAILURE_LOG] placeholder. You must also provide a [TEST_ENVIRONMENT] string (e.g., 'staging', 'production', 'PR #452') to give the model operational context. The prompt returns a structured JSON classification, not free text, so your harness must parse this JSON to decide the next step: auto-assign an owner, open a ticket, or flag for manual review.

Validation and retries are critical because a malformed classification can silently drop a failure. Implement a JSON schema validator in your harness that checks for the required fields: root_cause_category, confidence_score, is_flaky, recommended_owner, and evidence_summary. If the model's output fails validation, retry once with the same prompt but append the validation error to the [FAILURE_LOG] as a prefix: 'Previous output was invalid. Ensure your response matches the required JSON schema.' If the second attempt also fails, log the raw output and escalate to the on-call channel with a 'Triage classification failed' alert. Do not silently fall back to a default category.

Model choice and latency matter here. Use a fast, low-latency model (e.g., GPT-4o-mini, Claude 3.5 Haiku, or a fine-tuned open-weight model) because this prompt runs on every test failure in your pipeline. The few-shot examples are tuned for instruction-following models; if you switch to a smaller model, validate that it still respects the confidence_score field and doesn't hallucinate categories not present in the examples. For high-risk deployment pipelines (e.g., production release gates), consider routing low-confidence classifications (confidence_score < 0.7) to a human triage queue rather than auto-assigning. Log every classification with the prompt version, model, failure signature, and assigned owner to build a feedback loop for example drift detection.