Test Suite De-Flaking Prioritization Prompt

This prompt is built for engineering managers and platform teams who have already collected flaky test metrics from their CI systems and now face a resource allocation problem: which flaky tests should the team fix first? The job-to-be-done is transforming raw per-test data—failure frequency, developer investigation hours lost, release pipeline blockage counts, and estimated fix complexity—into a ranked remediation backlog that maximizes business impact per engineering hour spent. The ideal user has access to a dataset containing these fields for each flaky test and needs a defensible prioritization order, not a root cause diagnosis or a code fix.

Use this prompt when you have a list of 10–200 flaky tests and need to decide where to allocate limited remediation capacity across a sprint or quarter. The prompt assumes you can provide structured input for each test: a unique identifier, failure rate over the last N runs, average time wasted per false-positive investigation, number of times the test blocked a release in the last 30 days, and a complexity estimate (e.g., Low/Medium/High or story points). Do not use this prompt for diagnosing a single flaky failure from logs, generating a fix, or classifying failure types—those are separate workflows covered by sibling prompts like Flaky Test Failure Log Analysis or Flaky Test Classification and Severity Triage. This prompt also does not replace human judgment about which code paths are most critical; if business-criticality data is available, include it as an additional input field.

The output is a prioritized backlog with effort estimates, not a guarantee of fix difficulty. Before acting on the ranking, validate that the input data is recent (ideally from the last two weeks of CI runs) and that complexity estimates come from engineers familiar with the affected code. If the prompt ranks a test highly but the team knows the fix requires a risky architectural change, override the ranking. The next step after generating the backlog is to load the top 3–5 items into your team's planning tool and assign a spike for each to validate the complexity estimate before committing to the fix.

This prompt template is designed to be dropped into an engineering manager's workflow when the team needs to decide which flaky tests to fix first. It expects structured input data about each flaky test and produces a prioritized backlog with effort estimates. The template uses square-bracket placeholders for all variable inputs, making it straightforward to wire into a CI dashboard, a test analytics tool, or a manual triage spreadsheet.

textCopy
You are a test reliability engineer helping an engineering manager prioritize flaky test remediation.

You will receive a list of flaky tests with metadata. For each test, produce a priority score from 1 (lowest) to 10 (highest) based on:
- Failure frequency (higher = higher priority)
- Developer hours wasted per week on investigation and reruns (higher = higher priority)
- Number of releases blocked or delayed in the past 90 days (higher = higher priority)
- Estimated fix complexity in person-hours (lower = higher priority, because quick wins matter)

Then rank all tests from highest to lowest priority. For each test, include:
- The priority score
- A one-sentence justification referencing the input data
- An estimated fix effort tier: Low (1-4 hours), Medium (5-20 hours), High (21+ hours)
- A recommended action: Fix Now, Schedule This Sprint, Backlog, or Monitor

Input data:
[FLAKY_TEST_LIST]

Output as a JSON array of objects with keys: test_name, priority_score, justification, fix_effort_tier, recommended_action. Sort by priority_score descending.

[CONSTRAINTS]

To adapt this prompt, replace [FLAKY_TEST_LIST] with a structured array of test objects, each containing fields like test_name, failure_rate_last_30_days, avg_dev_hours_wasted_per_week, releases_blocked_last_90_days, and estimated_fix_hours. The [CONSTRAINTS] placeholder can hold additional rules such as team capacity limits, sprint boundaries, or mandatory fix thresholds. If your organization tracks different cost metrics—such as customer-facing impact or SLA violations—swap those into the scoring criteria. Always validate the output JSON against a schema before feeding it into a backlog tool or dashboard. For high-stakes release decisions, route the ranked output to a human engineering manager for final approval rather than auto-assigning remediation work.

This prompt is designed to be called programmatically, not used as a one-off chat interaction. The primary integration point is a CI observability system or test reporting database that aggregates flaky test metadata across multiple runs, services, and teams. Before invoking the prompt, assemble the required inputs from your data sources: failure frequency from your test reporting tool (e.g., Buildkite Analytics, Test Analytics in Datadog, or a custom ELK stack), developer time wasted from incident response or flaky test quarantine logs, release blockage count from your deployment pipeline (e.g., LaunchDarkly flag evaluations or Spinnaker stage history), and fix complexity estimates derived from historical remediation time for similar failure categories. The prompt expects these as structured data in the [FLAKY_TEST_INVENTORY] placeholder—pass a JSON array of test objects, each containing test_name, failure_frequency_last_30_days, avg_investigation_minutes, release_blocks_last_quarter, and estimated_fix_complexity (one of low, medium, high).

After the model returns a prioritized backlog, validate the output before it reaches a human decision-maker or a ticketing system. Implement a schema validator that checks every returned object contains rank, test_name, priority_score, effort_estimate, and remediation_rationale. Reject any output where priority_score falls outside 0–100 or where effort_estimate is not one of the expected enum values. If validation fails, retry once with the same input and an appended constraint: [CONSTRAINTS]: Your previous output failed schema validation. Ensure every entry includes rank, test_name, priority_score (0-100), effort_estimate (low|medium|high), and remediation_rationale. Log both the failed and successful responses for prompt debugging. For high-risk contexts—such as when the prompt recommends deferring remediation on a test that blocks releases—route the output to a human engineering manager for review before any automated ticket creation. Use a model with strong structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent) and set temperature=0 to maximize ranking consistency across runs.

Do not use this prompt as a real-time gate in your CI pipeline. The prioritization exercise is a periodic batch operation—weekly or per-sprint—not a per-commit decision. Wire the output into your backlog management system (Jira, Linear, or a custom internal tool) by mapping each ranked entry to a ticket with priority set from the score and effort set from the estimate. Include the remediation_rationale in the ticket description so the assigned engineer understands why this test matters. Avoid the temptation to auto-assign tickets based solely on the model's output; team context, current workload, and code ownership should still drive assignment. Finally, track whether the prioritized remediation order actually reduces flakiness over time—feed actual fix outcomes back into your input data to close the loop and improve future prioritization accuracy.