Flaky Test Root Cause Categorization Prompt

This prompt is built for platform teams and QA engineers who need to move beyond simple test retries and build a systematic understanding of why tests fail intermittently. The core job-to-be-done is turning raw failure artifacts—logs, stack traces, and test source code—into a structured, machine-readable root cause category. The ideal user is someone maintaining CI pipeline health across multiple repositories who needs to detect systemic patterns, such as a specific service causing environment-level failures or a shared library introducing concurrency bugs. The required context includes the test's failure output and the test implementation code; without both, the categorization will be shallow guesswork.

Use this prompt when you have a batch of flaky test failures and need to build a taxonomy for pattern analysis. It is designed for offline analysis and taxonomy building, not for real-time CI/CD gating decisions. Do not use this prompt to decide whether to block a release or to trigger an automatic retry. Its primary value is enabling cross-repository pattern detection—for example, identifying that 40% of flaky failures across your organization stem from a single environment provisioning race condition. The output is a structured category label with supporting evidence, which you can aggregate to prioritize infrastructure investments or library fixes.

Before running this prompt, ensure you have stripped sensitive data from logs and test code. The prompt works best with a defined taxonomy you want to map to; if you don't have one, start with the standard categories: environment, concurrency, data, assertion, infrastructure, and external dependency. Avoid using this prompt for tests that fail consistently—those are deterministic bugs, not flaky tests, and require a standard debugging workflow. The next step after categorization is to feed the structured output into a database or dashboard that tracks flakiness trends over time, enabling data-driven decisions about where to invest remediation effort.

This is the core prompt for categorizing a flaky test failure. It is designed to be pasted into your AI tool, orchestration pipeline, or evaluation harness. The prompt instructs the model to act as a test reliability engineer, analyze provided failure evidence against a defined taxonomy, and return a structured JSON object. Every variable that changes between runs is enclosed in square brackets, such as [TAXONOMY], [FAILURE_LOG], and [TEST_CODE]. Replace these placeholders with your actual data before sending the request.

textCopy
You are a test reliability engineer specializing in CI/CD pipeline health. Your task is to analyze a flaky test failure and categorize its root cause using a structured taxonomy.

## TAXONOMY
Use the following categories and subcategories. Only assign a category if there is sufficient evidence in the provided log and code.

[TAXONOMY]

## INPUT
### Failure Log
[FAILURE_LOG]

### Test Code
[TEST_CODE]

### Additional Context (optional)
[ADDITIONAL_CONTEXT]

## INSTRUCTIONS
1. Identify the single most likely root cause category and subcategory from the taxonomy.
2. Provide a confidence score between 0.0 and 1.0.
3. Quote the specific lines from the failure log or test code that serve as evidence.
4. If the evidence is insufficient to categorize, set the category to "Uncategorized/Insufficient Data" and confidence to 0.0.
5. Do not invent reasons. If the failure is ambiguous, state that explicitly.

## OUTPUT SCHEMA
Return ONLY a valid JSON object with this exact structure:
{
  "category": "string",
  "subcategory": "string | null",
  "confidence": "number",
  "evidence": ["string"],
  "rationale": "string"
}

To adapt this template, start by defining your taxonomy in the [TAXONOMY] placeholder. A common starting point includes top-level categories like Environment, Concurrency, Test Data, Assertion Logic, Infrastructure, and External Dependency, each with specific subcategories. The [FAILURE_LOG] should contain the raw console output, stack trace, and any error codes. The [TEST_CODE] placeholder expects the source code of the failing test and any relevant helper functions. For high-risk release pipelines, always route outputs with a confidence score below a defined threshold (e.g., 0.7) to a human reviewer for manual triage before accepting the categorization.

This prompt is designed for batch processing, not synchronous request-response loops. Ingest failure logs from your CI/CD platform (e.g., GitHub Actions, Jenkins, Buildkite) via webhook or log exporter. For each failed test, assemble the prompt variables—[FAILURE_LOG], [TEST_CODE], [TEST_NAME], and [LANGUAGE]—and send them to the model API. The output is a JSON object with a category field drawn from the defined taxonomy (environment, concurrency, data, assertion, infrastructure, external_dependency, unknown) and a confidence score. Store the structured output in a dedicated database table or data warehouse (e.g., flaky_test_categories) with a timestamp and commit SHA. This allows you to run aggregate queries like SELECT category, COUNT(*) FROM flaky_test_categories WHERE date > now() - INTERVAL 30 DAYS GROUP BY category to power dashboards that track flakiness trends across repositories.

Implement a retry wrapper around the model call that catches JSON parse errors. If the model fails to return valid JSON after two retries, log the raw output and flag the record for human review rather than silently dropping it. For high-volume pipelines, consider using a cheaper, faster model (e.g., GPT-4o-mini, Claude Haiku) for initial categorization and escalating only low-confidence or unknown results to a more capable model. This tiered approach keeps costs manageable while preserving accuracy on ambiguous cases. Include a validation step that rejects any output where the category field does not match the allowed enum values or where confidence falls outside the 0.0–1.0 range. Invalid outputs should trigger the retry path immediately.

Wire the prompt into your pipeline after test execution and log collection, not during the test run itself. This keeps latency off the critical path. If you are processing thousands of failures per day, batch requests where your provider supports it and implement rate limiting to avoid quota exhaustion. Finally, expose the categorized results to your team through a dashboard or alerting system—flaky test categories that spike in frequency (e.g., a sudden increase in infrastructure failures) often signal an underlying platform issue that needs immediate attention. Avoid using this prompt for real-time pass/fail decisions; it is a diagnostic and analytical tool, not a gate.