Regression Test Flakiness Analysis Prompt

This prompt is built for infrastructure and QA teams who run the same LLM evaluation suite repeatedly and get inconsistent pass/fail results. The core job-to-be-done is differential diagnosis: you have multiple run logs for the same test cases, you know the prompt and model version are identical, yet results flip between passes and failures. The prompt analyzes these run logs and attributes flakiness to a specific root cause—temperature sensitivity, model non-determinism, ambiguous evaluation criteria, or input instability—rather than leaving you to guess.

Use this prompt when you can provide at least two complete run logs for the same test case, including the input, the model output, the evaluation rubric applied, and the resulting score or pass/fail judgment. The prompt requires structured evidence: run IDs, timestamps, model configuration (especially temperature and seed settings), and the exact evaluation criteria used. It works best when flakiness is reproducible enough that you have multiple examples of the same test case flipping. Do not use this prompt for single-run test failures, for debugging prompt logic errors where the output is consistently wrong, or for cases where the model version or prompt text actually changed between runs—those are regression bugs, not flakiness.

The output is a structured root cause attribution with supporting evidence from your logs. Expect a classification into one of four categories: temperature sensitivity (the model samples different tokens at non-zero temperature), model non-determinism (the model produces different outputs even at temperature zero due to infrastructure-level variation), ambiguous evaluation criteria (the rubric or pass/fail rule is too vague to produce consistent judgments), or input instability (the test input itself contains dynamic elements like timestamps or random seeds). Each attribution includes the specific evidence from your run logs that supports the diagnosis and a recommended remediation—such as setting temperature to zero, tightening rubric language, or stabilizing test inputs. Before acting on the diagnosis, validate it by running the suggested remediation on a subset of flaky cases and confirming the flakiness rate drops. If the prompt cannot confidently attribute the flakiness to a single cause, it will tell you that too, which is a signal that you need more run logs or a deeper investigation outside the scope of automated analysis.

This prompt template is designed to be pasted directly into your evaluation harness or LLM workbench. It expects a structured input containing repeated test run results, the evaluation criteria used, and the model configuration. Replace every square-bracket placeholder with your actual data before execution. The prompt instructs the model to identify which test cases are flaky, categorize the likely root cause, and recommend specific remediation steps—such as lowering temperature, rewriting evaluation rubrics, or adding explicit constraints.

textCopy
You are a test reliability engineer analyzing non-deterministic failures in an AI evaluation pipeline.

You will receive a set of test cases that were executed multiple times against the same prompt and model. Some test cases passed inconsistently across runs. Your job is to identify which test cases are flaky, determine the most likely cause of flakiness, and recommend a concrete fix.

## INPUT

[TEST_RUN_DATA]

## EVALUATION CRITERIA USED

[EVALUATION_CRITERIA]

## MODEL CONFIGURATION

[MODEL_CONFIG]

## INSTRUCTIONS

1. Identify every test case where the pass/fail result changed across runs. These are the flaky cases.
2. For each flaky case, classify the root cause into exactly one of the following categories:
   - **TEMPERATURE_SENSITIVITY**: The output varies significantly with sampling, suggesting the prompt allows multiple valid but semantically different responses that the evaluator treats inconsistently.
   - **MODEL_NON_DETERMINISM**: The model produces different outputs even at temperature=0, indicating infrastructure-level non-determinism or model-side variability.
   - **AMBIGUOUS_EVALUATION_CRITERIA**: The evaluation rubric or pass/fail criteria are too vague, allowing the same output to be scored differently across runs.
   - **BORDERLINE_CASE**: The test case sits exactly on the decision boundary, where tiny output variations flip the result.
   - **CONTEXT_WINDOW_TRUNCATION**: Long inputs or outputs are truncated inconsistently, changing what the evaluator sees.
3. For each flaky case, provide:
   - The test case identifier
   - The root cause category
   - A one-sentence evidence statement quoting the specific output variation or scoring inconsistency
   - A concrete, actionable remediation recommendation
4. Produce a summary table with counts per root cause category.
5. Rank the flaky cases by severity, where severity is defined as:
   - **HIGH**: Flakiness occurs in more than 50% of runs
   - **MEDIUM**: Flakiness occurs in 20-50% of runs
   - **LOW**: Flakiness occurs in less than 20% of runs

## OUTPUT FORMAT

Return a JSON object with this exact schema:

{
  "total_test_cases": <integer>,
  "total_runs_per_case": <integer>,
  "flaky_cases": [
    {
      "test_case_id": "<string>",
      "pass_rate": <float between 0 and 1>,
      "root_cause": "TEMPERATURE_SENSITIVITY | MODEL_NON_DETERMINISM | AMBIGUOUS_EVALUATION_CRITERIA | BORDERLINE_CASE | CONTEXT_WINDOW_TRUNCATION",
      "evidence": "<one sentence quoting the variation>",
      "severity": "HIGH | MEDIUM | LOW",
      "recommendation": "<concrete, actionable fix>"
    }
  ],
  "summary_by_cause": {
    "TEMPERATURE_SENSITIVITY": <count>,
    "MODEL_NON_DETERMINISM": <count>,
    "AMBIGUOUS_EVALUATION_CRITERIA": <count>,
    "BORDERLINE_CASE": <count>,
    "CONTEXT_WINDOW_TRUNCATION": <count>
  },
  "overall_assessment": "<2-3 sentence summary of flakiness health and top priority fix>"
}

## CONSTRAINTS

- Do not invent test cases not present in the input.
- If a test case passed consistently, do not include it in flaky_cases.
- If you cannot determine the root cause with confidence, use AMBIGUOUS_EVALUATION_CRITERIA as the default and note the uncertainty.
- Recommendations must be specific: name the exact parameter to change, the rubric line to rewrite, or the infrastructure check to perform.

Adapting the template: Replace [TEST_RUN_DATA] with a structured array of test cases, each containing an identifier, the input, the output from each run, and the pass/fail result per run. [EVALUATION_CRITERIA] should contain the exact rubric, pass/fail rules, or scoring function used by your judge. [MODEL_CONFIG] must include the model name, temperature setting, top_p, and any seed or infrastructure details. If your test harness doesn't capture per-run outputs, you'll need to add output logging before this prompt becomes useful—without output-level evidence, root cause classification degrades to guesswork.

Validation and safety: The output JSON must be validated against the schema before ingestion into your CI/CD pipeline. If the model returns malformed JSON, retry with a stricter schema reminder or use a repair prompt from the Output Repair and Validation Prompts pillar. This analysis is diagnostic, not autonomous—remediation recommendations should be reviewed by a human before changing evaluation thresholds, model parameters, or test definitions. For production pipelines, store the flakiness report alongside your regression run artifacts so you can track whether flakiness is increasing over time.

This prompt is designed to run as a post-execution analysis step in a regression testing pipeline, not as a real-time classifier. After your test runner produces a results file with repeated runs per test case, feed the aggregated results into this prompt to identify flaky tests and attribute the flakiness to temperature sensitivity, model non-determinism, or ambiguous evaluation criteria. The prompt expects structured input—typically a JSON array of test cases with their run histories—so your harness must collect and format that data before invocation.

Wire the prompt into your CI/CD workflow as a conditional analysis job that triggers when the overall pass rate falls below a threshold or when individual test cases show inconsistent results across runs. Use a model with low temperature (0.0–0.1) for the analysis itself to maximize reproducibility of the flakiness diagnosis. Implement a validation layer that checks the output schema before accepting the analysis: each identified flaky test must include a test_id, flakiness_score (0.0–1.0), attributed_cause from the allowed enum (temperature_sensitivity, model_non_determinism, ambiguous_criteria, unknown), and evidence referencing specific run outcomes. Reject and retry outputs that don't conform to this schema, or that attribute flakiness without citing concrete run data.

Log every analysis invocation with the input run data, the raw model output, the validated result, and the model version used. This trace is essential for diagnosing when the flakiness analyzer itself becomes unreliable. For high-stakes pipelines where flaky tests could mask real regressions, route analyses with flakiness_score > 0.7 or attributed_cause: unknown to a human reviewer before accepting the diagnosis. Avoid running this prompt on single-run test results—it needs repeated runs to distinguish flakiness from consistent failure. If your test infrastructure can't support multiple runs per commit, consider running this analysis weekly against a dedicated flakiness detection suite that does.