Flaky Test Classification and Severity Triage Prompt

Engineering managers and platform teams use this prompt to stop flaky tests from being a diffuse, unmanaged drag on delivery velocity. The prompt ingests failure logs, test code, and CI metadata to classify each flaky failure into a defined category (environmental, timing, data, concurrency, infrastructure) and assign a severity score based on failure frequency, blast radius, and release impact. The output is a structured triage report that feeds directly into a remediation backlog, replacing ad-hoc Slack threads and spreadsheets with a repeatable, evidence-backed prioritization process.

Use this prompt when you have a growing list of flaky tests and need to decide which ones to fix first, quarantine, or monitor. It is designed for batch triage of known flaky failures where you have accumulated failure logs, test code, and CI metadata across multiple runs. The prompt works best when you can provide at least 10-20 failure instances per test to establish reliable frequency patterns. For each test under review, you should supply the test file path, failure stack traces from recent CI runs, the test's pass/fail history over the last 30 days, and any known blast radius metadata such as which services or release pipelines depend on this test passing. The severity scoring logic weighs three factors: failure frequency (how often it fails relative to total runs), blast radius (how many teams, pipelines, or releases are blocked when it fails), and release impact (whether failures have historically correlated with production regressions). Tests that fail frequently, block critical release paths, and correlate with past incidents receive the highest severity scores and should be fixed or quarantined first.

Do not use this prompt for real-time incident response or for diagnosing a single test failure in isolation. It is not designed for root cause analysis of a novel failure mode; for that, use the Flaky Test Failure Log Analysis Prompt or Race Condition Hypothesis Generation Prompt. This prompt also assumes you have already identified which tests are flaky—it does not detect flakiness from raw CI data. If you need to identify flaky tests first, pair this with a test suite stability regression detection workflow. Finally, the severity scores are recommendations, not automated decisions. Always review high-severity classifications with the owning team before quarantining tests or blocking releases, especially for tests covering security-critical or compliance-relevant code paths.

This prompt template is designed to be copied directly into your AI orchestration layer, test harness, or CI/CD pipeline. It expects structured input data about flaky test failures and returns a classification with severity triage. All square-bracket placeholders must be replaced with real data before execution. The prompt enforces a strict output schema so downstream systems can parse the classification deterministically.

textCopy
You are a test reliability engineer analyzing flaky test failures.

Your task is to classify each flaky test failure by its root cause category and assign a severity score based on failure frequency, blast radius, and release impact.

## INPUT
[FAILURE_REPORTS]

## CLASSIFICATION TAXONOMY
Classify each failure into exactly one of these categories:
- `environmental`: Infrastructure, network, CI worker, disk, or external service issues outside the test code.
- `timing`: Race conditions, fixed waits, missing await conditions, or timeout-induced failures.
- `data`: Database state contamination, missing cleanup, implicit data ordering, or seed-dependent data.
- `concurrency`: Parallel execution conflicts, shared mutable state, port conflicts, or test worker interference.
- `assertion`: Non-deterministic assertions on random seeds, UUIDs, timestamps, unordered collections, or floating-point precision.
- `infrastructure`: CI configuration drift, missing environment variables, or resource exhaustion.
- `external_dependency`: Mock/stub mismatch, third-party service flakiness, or API contract drift.

## SEVERITY SCORING RULES
Assign a severity score from 1 (lowest) to 5 (highest) using these criteria:
- **Frequency**: How often does this test fail? (1: <1% of runs, 3: 1-5%, 5: >5%)
- **Blast Radius**: How many other tests or pipelines are blocked? (1: isolated, 3: blocks a test suite, 5: blocks multiple pipelines or release)
- **Release Impact**: Does this failure block production releases? (1: no impact, 3: delays release, 5: blocks release entirely)
- **Investigation Cost**: Average developer hours wasted per failure investigation. (1: <15 min, 3: 1-4 hours, 5: >4 hours)

Calculate the final severity as: `round((frequency + blast_radius + release_impact + investigation_cost) / 4)`

## OUTPUT_SCHEMA
Return a valid JSON object with this exact structure:
{
  "classifications": [
    {
      "test_name": "string",
      "test_file": "string",
      "category": "environmental|timing|data|concurrency|assertion|infrastructure|external_dependency",
      "confidence": 0.0-1.0,
      "evidence_summary": "string (max 200 chars, cite specific log lines or stack trace elements)",
      "severity": {
        "score": 1-5,
        "frequency": 1-5,
        "blast_radius": 1-5,
        "release_impact": 1-5,
        "investigation_cost": 1-5,
        "rationale": "string (max 150 chars)"
      },
      "recommended_action": "quarantine|immediate_fix|monitor|investigate_further",
      "action_rationale": "string (max 150 chars)"
    }
  ],
  "summary": {
    "total_failures": "integer",
    "category_distribution": {"category": "count"},
    "average_severity": "float",
    "tests_recommended_for_quarantine": ["test_name"],
    "tests_requiring_immediate_fix": ["test_name"]
  }
}

## CONSTRAINTS
- Do not invent failure data. Only classify failures present in [FAILURE_REPORTS].
- If a failure cannot be confidently classified, set confidence below 0.5 and use category `unknown`.
- Do not recommend quarantine for tests covering critical path code (authentication, payment, data integrity) without flagging the risk.
- If severity score is 5, recommended_action must be `immediate_fix` or `quarantine`.
- Cite specific evidence from logs or stack traces in evidence_summary. Do not speculate.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK_LEVEL
[HIGH] - Classification errors can cause incorrect quarantine decisions or mask real regressions. Human review is required before applying quarantine actions to critical-path tests.

Adaptation guidance: Replace [FAILURE_REPORTS] with structured failure data including test names, log excerpts, stack traces, execution duration, and CI context. Replace [FEW_SHOT_EXAMPLES] with 2-3 annotated examples showing correct classification and severity scoring for your test suite's common failure patterns. The output schema is designed for direct ingestion into test management systems or quarantine automation. Always validate the JSON response against the schema before acting on quarantine or fix recommendations. For critical-path tests, route the classification to a human reviewer before automated action.

This prompt is designed to be called programmatically after a test failure event, not as a one-off chat interaction. The implementation harness should collect the required inputs—failure logs, test code, execution context, and historical failure data—from your CI system (e.g., GitHub Actions, Jenkins, CircleCI) and assemble them into the structured [INPUT] block before invoking the model. The output is a JSON classification and severity object, which your harness must validate against the expected [OUTPUT_SCHEMA] before routing the result to an issue tracker, a quarantine system, or a team notification channel.

Wire the prompt into a post-failure hook in your CI pipeline. When a test fails with a flaky signature (e.g., it passed on retry, or its failure history shows intermittent patterns), the harness should: (1) fetch the last N failure logs and stack traces from your test reporting tool; (2) retrieve the test source code from the repository at the relevant commit; (3) query your test history database for failure frequency, quarantine status, and affected code paths; (4) assemble these into the [INPUT] block with clear section delimiters; (5) call the model with a moderate temperature (0.1–0.2) to balance consistency with nuanced classification; (6) validate the returned JSON against the expected schema, checking that category is one of the allowed enum values and severity is an integer between 1 and 5; (7) on validation failure, retry once with the error message appended to the prompt as a correction hint; (8) log the full prompt, response, and validation result for auditability. For high-severity classifications (4–5), route to a human triage queue rather than auto-quarantining.

Avoid calling this prompt on every test failure—only trigger it when the failure matches a flaky pattern (intermittent, passes on retry, or flagged by your flakiness detection system). Do not use this prompt's output to automatically skip or delete tests without human review for severity levels 3 and above. Store the classification results in your test health database to build a historical record that improves future triage accuracy and feeds into your Flaky Test Budget and Threshold Enforcement dashboards. The next step after classification is to route the result: low-severity environmental failures may go to a platform team backlog, while high-severity concurrency failures should trigger an immediate developer alert with the generated hypothesis and suggested remediation path.