Flaky Test Impact Assessment Prompt for Release Readiness

This prompt is designed for test infrastructure engineers and release coordinators who must evaluate flaky test results before a release decision. Its job is to distinguish environmental noise and test instability from genuine code regressions. The output is a structured flakiness impact report that assigns confidence scores to each failing test, recommends quarantines for unreliable tests, and quantifies the residual risk of masked defects. Use this prompt when your CI pipeline produces a mix of consistent and inconsistent failures and you need an evidence-based assessment of whether the test signal is trustworthy enough to support a go/no-go recommendation.

This prompt assumes you already have test execution logs, failure patterns, and environment metadata. It does not replace root cause analysis or test repair workflows. The ideal input includes multiple runs of the same test suite across different commits or environments, timestamps, exception messages, and any known environment instability events. If you only have a single run or no historical failure data, this prompt will produce low-confidence output because it cannot distinguish flakiness from a one-time anomaly. Wire this prompt into your release gate pipeline after test execution completes but before the final go/no-go decision meeting.

Do not use this prompt when you need a definitive root cause for a specific test failure—that requires a debugging workflow with code-level analysis. Do not use it when all failures are consistent and reproducible; in that case, a standard regression report is more appropriate. Avoid using this prompt as a substitute for fixing flaky tests. Its value is in making the release decision safer despite known test instability, not in eliminating the instability itself. After running this prompt, feed the quarantine recommendations into your test exclusion configuration and the residual risk summary into your release readiness memo.

The prompt below is designed to be copied directly into your AI harness, test runner, or evaluation pipeline. It forces the model to adopt the role of a test reliability analyst and to reason strictly from the provided execution data. Replace every square-bracket placeholder with your actual test run context before execution. The prompt is self-contained: it defines the role, the required output schema, explicit constraints, and a confidence-scoring rubric so that the model's behavior is predictable across different runs and models.

textCopy
You are a test reliability analyst. Your task is to assess the impact of flaky tests on release readiness using only the provided execution data.

## INPUT DATA
[EXECUTION_DATA]

## OUTPUT SCHEMA
Return a single JSON object with the following structure:
{
  "assessment_id": "string",
  "generated_at": "ISO-8601 timestamp",
  "summary": {
    "total_tests": "integer",
    "passed": "integer",
    "failed": "integer",
    "flaky_candidates": "integer",
    "quarantine_recommendations": "integer"
  },
  "flaky_analysis": [
    {
      "test_id": "string",
      "test_name": "string",
      "failure_count": "integer",
      "pass_count": "integer",
      "flakiness_confidence": "float between 0.0 and 1.0",
      "likely_cause": "ENVIRONMENT | TIMING | DATA | UNKNOWN",
      "regression_risk": "LOW | MEDIUM | HIGH",
      "evidence": "string explaining the pattern observed",
      "recommended_action": "QUARANTINE | INVESTIGATE | IGNORE"
    }
  ],
  "residual_risk": {
    "masked_defect_risk": "LOW | MEDIUM | HIGH",
    "explanation": "string describing what real failures might be hidden by flaky results",
    "confidence_in_assessment": "float between 0.0 and 1.0"
  },
  "release_readiness_impact": {
    "blocks_release": "boolean",
    "recommendation": "string summarizing the go/no-go impact of flaky tests",
    "conditions_for_proceeding": ["string array of conditions if release can proceed"]
  }
}

## CONSTRAINTS
- Classify a test as a flaky candidate only if it has both passing and failing runs in the provided data.
- Set flakiness_confidence based on the ratio of inconsistent results to total runs. A test that fails 50% of the time across 10 runs should have higher confidence than one that failed once in 100 runs.
- If no flaky tests are detected, return an empty flaky_analysis array and set residual_risk.masked_defect_risk to LOW.
- Do not invent test names, failure counts, or patterns not present in the input data.
- If the input data is insufficient to make a determination, set confidence_in_assessment below 0.5 and explain why in the residual_risk.explanation field.
- Never recommend QUARANTINE for a test with flakiness_confidence below 0.6 unless the regression_risk is HIGH.

## RISK LEVEL DEFINITIONS
- LOW regression_risk: failure pattern is clearly environmental (e.g., network timeout, test order dependency) and unrelated to application logic.
- MEDIUM regression_risk: failure pattern is ambiguous; could be environment or could mask a real defect in non-critical paths.
- HIGH regression_risk: failure occurs in security, data integrity, payment, or authentication paths; or failure pattern correlates with recent code changes in the same area.

After copying the template, replace [EXECUTION_DATA] with your actual test execution records. The data should include at minimum: test identifiers, pass/fail status per run, timestamps, and environment metadata. If your test runner produces JSON output, pass it directly. If you have logs or CI artifacts, preprocess them into a structured format before insertion. The prompt expects enough signal to distinguish genuine flakiness from one-off environment failures. For high-risk domains such as payments or health data, add a human review step after the model returns its assessment—do not auto-quarantine tests in CI without operator confirmation.

This prompt is designed to run as a post-execution analysis step in your CI pipeline, triggered after a test suite completes with failures. The harness should collect the raw test output (JUnit XML, JSON, or console logs), environment metadata (CI run ID, branch, commit SHA, infrastructure provider), and recent test history (last N runs for the same suite) before assembling the prompt. Do not run this prompt on every passing suite—only invoke it when the failure count exceeds a configurable threshold or when specific high-signal tests fail. The prompt expects structured inputs; feeding it unstructured console dumps without preprocessing will degrade output quality.

Validation and output contract: Parse the model response against a strict JSON schema before accepting it. Required fields include flaky_tests (array of objects with test_name, flakiness_confidence, evidence, recommended_action), genuine_regressions (same shape), quarantine_recommendations (array of test names with justification), and residual_risk (object with masked_defect_likelihood, confidence_erosion, summary). If the model returns invalid JSON, retry once with the validation error message appended to the prompt as a correction hint. If the second attempt also fails schema validation, log the raw output and escalate for human review rather than silently proceeding. For high-risk release contexts (production deployments, regulated systems), always require a human to approve quarantine actions before they are applied to the test suite configuration.

Model choice and tool integration: Use a model with strong structured output capabilities and a context window large enough to hold multiple historical test runs. For most teams, GPT-4o or Claude 3.5 Sonnet with JSON mode enabled works well. Wire the validated output into your test management system: automatically tag quarantined tests in TestRail, Xray, or a similar tool; create Jira tickets for genuine regressions with the evidence block attached; and post the residual risk summary to your release readiness dashboard or Slack channel. Store the full prompt, response, and validation result as an artifact on the CI run for auditability. If your test infrastructure already tracks flakiness scores (e.g., via a flaky test detection service), pass those scores into the [HISTORICAL_FLAKINESS_DATA] placeholder to ground the model's assessment in existing signals rather than asking it to infer patterns from raw logs alone.

Failure modes to instrument: The most common production failure is the model misclassifying a new flaky test as a genuine regression because it lacks sufficient history. Mitigate this by setting a minimum-history threshold: if a failing test has fewer than 5 prior runs, flag it as insufficient_data and route it for manual triage rather than trusting the model's classification. Another failure mode is environmental noise masquerading as flakiness—when a CI infrastructure problem (network partition, disk pressure, exhausted cloud quota) causes widespread failures that look flaky but are actually systemic. Add a pre-check that counts the proportion of failing tests; if more than 40% of the suite fails, skip the prompt and surface an infrastructure alert instead. Finally, monitor for confidence overstatement: if the model assigns high confidence to classifications that later prove wrong, adjust the prompt's [CONSTRAINTS] to require explicit enumeration of alternative explanations before committing to a classification.