Flaky Evaluation Detection Prompt Template

This prompt is for platform engineers and AI quality teams who observe inconsistent scores from their LLM-as-a-judge evaluation harness. When the same prompt and test case produce different scores across multiple runs, the evaluation pipeline itself is flaky. This prompt template ingests raw eval run logs, identifies non-deterministic judges, quantifies score variance patterns, and generates root-cause hypotheses. Use it before you trust any eval gate to block a bad release or before you waste hours manually diffing eval traces.

The ideal user has access to eval run logs containing at least three repeated measurements per test case, with timestamps, judge identifiers, and raw scores. Required context includes the eval rubric used, the model serving as judge, and any sampling parameters (temperature, top_p) applied during evaluation. This prompt is strictly for debugging the measurement instrument—the judge itself—not the prompt-under-test. Do not use it when you have fewer than three runs per case, when the judge is deterministic by design (temperature=0 with fixed seed), or when you are trying to improve the prompt being evaluated rather than the evaluation harness.

Before running this prompt, ensure your eval logs are structured with consistent fields: test_case_id, run_id, judge_model, judge_temperature, raw_score, timestamp, and any per-axis breakdowns. The prompt expects variance to be measurable—if your judge always returns identical scores, you have a different problem (likely a calibration or rubric issue, not flakiness). After receiving the flakiness report, prioritize judges with coefficient of variation above 0.15 or those showing bimodal score distributions, as these are the most likely to cause false-positive or false-negative release gate decisions.

This prompt template is designed to be pasted directly into your debugging workspace. It instructs the model to act as an evaluation forensics analyst, comparing multiple eval run outputs for the same input to identify flaky judges, score variance patterns, and root-cause hypotheses. Replace every square-bracket placeholder with data from your actual eval pipeline before running.

textCopy
You are an evaluation forensics analyst. Your task is to analyze multiple evaluation runs for the same input and determine whether the evaluation pipeline is flaky (non-deterministic).

## INPUT DATA
- Original User Input: [USER_INPUT]
- Model Output Under Evaluation: [MODEL_OUTPUT]
- Eval Run Results (JSON array of run objects): [EVAL_RUNS]
  Each run object contains:
  - run_id: string
  - judge_model: string (e.g., "gpt-4o", "claude-3.5-sonnet")
  - score: number or string
  - justification: string
  - timestamp: ISO 8601 string

## EVALUATION CONTEXT
- Metric Being Evaluated: [METRIC_NAME]
- Expected Score Range: [MIN_SCORE] to [MAX_SCORE]
- Pass Threshold: [PASS_THRESHOLD]
- Number of Runs Provided: [RUN_COUNT]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "flakiness_detected": boolean,
  "flakiness_severity": "none" | "low" | "medium" | "high" | "critical",
  "score_variance": {
    "min_score": number,
    "max_score": number,
    "mean_score": number,
    "std_dev": number,
    "range": number
  },
  "pass_fail_disagreement": boolean,
  "disagreement_count": number,
  "judge_consistency": {
    "same_judge_variance": number | null,
    "cross_judge_variance": number | null,
    "notes": string
  },
  "root_cause_hypotheses": [
    {
      "hypothesis": string,
      "confidence": "low" | "medium" | "high",
      "evidence": string,
      "recommended_test": string
    }
  ],
  "repeatability_test": {
    "description": string,
    "expected_stable_output": string,
    "rerun_count_recommendation": number
  },
  "mitigation_recommendations": [string],
  "requires_human_review": boolean
}

## CONSTRAINTS
- Do not hallucinate run data. Only analyze the provided [EVAL_RUNS].
- If fewer than 3 runs are provided, set flakiness_severity to "low" and note insufficient data.
- Flag requires_human_review as true if flakiness_severity is "high" or "critical".
- For cross-judge variance, only compute if multiple judge_model values are present.
- Justifications must be compared semantically, not via exact string match.
- If score_variance cannot be computed (non-numeric scores), explain why in judge_consistency.notes.

## ANALYSIS STEPS
1. Parse all eval runs and extract scores and justifications.
2. Compute score variance statistics.
3. Check for pass/fail disagreements relative to [PASS_THRESHOLD].
4. Compare justifications across runs for semantic drift.
5. Identify whether variance is driven by judge model differences, temperature, ambiguous rubric, or input sensitivity.
6. Generate root-cause hypotheses with testable predictions.
7. Propose a repeatability test that would confirm or rule out flakiness.
8. Recommend concrete mitigations.

Adaptation guidance: Replace [EVAL_RUNS] with a JSON array of your actual eval run records. If your eval pipeline uses non-numeric scores (e.g., categorical labels), the model will note this in the output and skip numeric variance calculations. For high-stakes pipelines where flaky evals could allow regressions into production, always set requires_human_review to true when severity reaches "high" or "critical" and route the output to a human reviewer before accepting mitigation recommendations.

After running this prompt, feed the recommended_test field back into your eval harness to confirm or rule out each hypothesis. Do not treat the model's root-cause analysis as ground truth—it is a structured starting point for your own investigation. Log the full output alongside your eval traces for auditability.

This prompt is designed to be called programmatically as part of an automated eval health check, not as a one-off manual investigation. The harness should feed the prompt a structured payload containing eval run logs, score distributions, and judge identifiers, then parse the resulting flakiness report into a machine-readable format for downstream alerting and dashboards. The primary integration point is a scheduled CI/CD job or a monitoring cron that triggers whenever eval score variance exceeds a predefined threshold, such as a coefficient of variation above 0.15 across the last N runs on a static golden dataset.

Implement the call with a strict retry and validation wrapper. First, validate the output against a JSON schema that requires the flaky_judges array, variance_patterns object, and root_cause_hypotheses array to be present and non-null. If schema validation fails, retry once with a higher temperature (e.g., 0.4) and an explicit instruction appended to the prompt: 'The previous output failed schema validation. Ensure the response is valid JSON matching the required structure.' If the second attempt also fails, log the raw output and raise a non-blocking alert for manual review. For model choice, use a model with strong instruction-following and JSON mode support, such as gpt-4o or claude-3.5-sonnet, and set response_format to json_object where the API supports it. Log the full prompt, response, parse status, and any detected flaky judges to your observability platform (e.g., LangSmith, Datadog, or custom logging) with the eval run ID as the trace identifier.

Do not treat this prompt's output as the final word on whether an eval is production-ready. The flakiness report is a diagnostic signal, not an automated gate. Use the mitigation_recommendations field to generate tickets or runbook items, but require a human to approve changes to judge prompts, thresholds, or model versions. The highest-risk failure mode is a false negative where the prompt reports low flakiness but a judge is silently drifting. Mitigate this by pairing this detection prompt with a periodic inter-rater reliability check against a held-out human-labeled calibration set, and by tracking the stability of the flakiness report itself over time—if the report's conclusions oscillate wildly between runs, the detection harness itself may need recalibration.