Judge Prompt Sensitivity Analysis Template

This prompt is for prompt engineers and evaluation platform teams who need to understand how small changes in judge instructions affect scoring behavior. The job-to-be-done is rigorous: you have a baseline judge prompt that produces scores, and you need to test multiple instruction variants to see which phrasing is fragile, which changes cause statistically significant score shifts, and which wording is most robust against irrelevant input variation. This is not a prompt for casual A/B testing or vibe checks—it is for teams shipping automated evaluation pipelines where score consistency is a product requirement.

Use this prompt when you are finalizing a judge prompt for production and need evidence that your instruction wording is stable. It is appropriate when you have a reference set of scored examples (human-verified or golden dataset) and a list of instruction variants to test—such as reworded criteria, different scale anchors, added constraints, or changed output formats. The prompt expects you to provide the baseline prompt, the variants, the test inputs, and the reference scores. It produces a sensitivity report with per-variant score distributions, statistical tests for significant shifts, and a fragility ranking that identifies which instruction phrases cause the most score variance.

Do not use this prompt when you are still designing the initial rubric or when you lack a stable reference dataset. It is not a substitute for calibration against human ratings—use the Judge Calibration prompt for that. It is also not designed for detecting score drift over time; use the Score Drift Detection prompt instead. This prompt assumes you already have a working judge and need to harden its instructions. If your variants include fundamentally different scoring dimensions rather than wording changes, you are testing rubric redesign, not sensitivity, and should use a rubric comparison workflow. Finally, this prompt requires statistical literacy from the reader—the output includes p-values, effect sizes, and confidence intervals that demand interpretation before acting on the results.

This template is the core instruction set you will send to an LLM judge to perform sensitivity analysis on another judge prompt. It asks the model to systematically vary instruction phrasing, apply each variant to a held-out set of scored examples, measure score shifts, and identify fragile wording. The output is a structured sensitivity report, not a single score. Use this template when you are iterating on an evaluation prompt and need to know which phrases are load-bearing and which are cosmetic.

textCopy
You are an evaluation engineer performing prompt sensitivity analysis on a judge prompt.

Your task is to test how variations in judge instructions affect scoring behavior and produce a sensitivity report.

## JUDGE PROMPT UNDER TEST
[JUDGE_PROMPT]

## SCORED EXAMPLES
[SCORED_EXAMPLES]

## INSTRUCTION PHRASES TO VARY
[PHRASES_TO_VARY]

## VARIATION STRATEGY
[VARIATION_STRATEGY]

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "baseline_summary": {
    "mean_score": number,
    "score_distribution": { "score_value": count },
    "example_count": number
  },
  "variants": [
    {
      "variant_id": "string",
      "original_phrase": "string",
      "variant_phrase": "string",
      "variant_scores": [number],
      "mean_shift": number,
      "score_change_distribution": { "example_id": "old_score -> new_score" },
      "statistical_significance": {
        "test_used": "string",
        "p_value": number,
        "significant_at_0_05": boolean
      },
      "fragility_rating": "low" | "medium" | "high",
      "affected_dimensions": ["string"]
    }
  ],
  "fragile_phrases": [
    {
      "phrase": "string",
      "fragility_rating": "low" | "medium" | "high",
      "max_score_shift": number,
      "recommendation": "string"
    }
  ],
  "robustness_recommendations": [
    {
      "original_phrase": "string",
      "recommended_replacement": "string",
      "rationale": "string",
      "expected_stability_improvement": "low" | "medium" | "high"
    }
  ]
}

## CONSTRAINTS
- Apply each variant independently; do not combine phrase variations unless [VARIATION_STRATEGY] specifies combinatorial testing.
- Use the same set of [SCORED_EXAMPLES] for every variant.
- Report statistical significance using [SIGNIFICANCE_TEST] with a threshold of [SIGNIFICANCE_THRESHOLD].
- Flag any phrase as "high" fragility if the mean score shift exceeds [FRAGILITY_THRESHOLD] or if significance is below [SIGNIFICANCE_THRESHOLD].
- If a variant causes the judge to refuse, produce an error, or output an unparseable score, record the failure in variant_scores as null and set fragility_rating to "high".
- Do not invent examples or scores. Work only from the provided [SCORED_EXAMPLES].

## EVALUATION NOTES
- A phrase is "fragile" if small wording changes cause large, statistically significant score shifts.
- A phrase is "robust" if score distributions remain stable across reasonable paraphrases.
- Recommendations should prefer concrete replacement text over vague advice like "be clearer."

Adaptation guidance: Replace [JUDGE_PROMPT] with the full text of the evaluation prompt you are testing. [SCORED_EXAMPLES] should contain at least 20–50 examples with known scores, ideally from human raters or a stable baseline judge. [PHRASES_TO_VARY] is a list of specific instruction fragments you suspect may be brittle—such as severity adverbs, constraint wording, or rubric descriptors. [VARIATION_STRATEGY] controls whether you test single-phrase substitutions or combinatorial pairs; start with single-phrase for tractability. [SIGNIFICANCE_TEST] should be set to a method like "paired t-test" or "Wilcoxon signed-rank" depending on your score distribution. [FRAGILITY_THRESHOLD] and [SIGNIFICANCE_THRESHOLD] let you tune sensitivity—start with 0.5 points mean shift and p < 0.05.

Before running this at scale, validate the output schema against a single variant to confirm your judge model produces parseable JSON. If the judge model is the same model family as the judge under test, be aware of self-consistency biases—consider using a different model for the sensitivity analysis itself. For high-stakes evaluation pipelines where score drift could affect product decisions, always review the fragile_phrases and robustness_recommendations with a human before rewriting the judge prompt. Store each sensitivity report alongside the judge prompt version it analyzed so you can track which phrases caused instability over time.

The sensitivity analysis prompt is not a one-shot query; it is a batch evaluation harness that compares scoring behavior across multiple judge instruction variants against a fixed, curated test set. The harness must iterate over a list of prompt variants, run each variant against the same set of test cases, collect structured score outputs, and then feed those results into the analysis prompt. This means the application layer is responsible for variant management, parallel execution, result aggregation, and statistical computation—the LLM prompt itself only performs the final analysis and report generation.

Start by defining a test case dataset of at least 30–50 examples that span your score range and include known edge cases. Each test case needs an id, the input to be judged, and optionally a reference_score if you have human labels. Next, define your variant list as an array of objects, each containing a variant_id and the full judge_instructions string. Common variants to test include: reworded criteria, changed scale anchors, added or removed constraints, and different output format instructions. Run each variant against every test case, capturing the raw score and justification. Store results in a structured format (JSON Lines or a database table) with fields: test_case_id, variant_id, score, justification, and timestamp. For high-stakes evaluation pipelines, log the full prompt and response for auditability.

Before calling the analysis prompt, precompute the statistical comparisons in application code. For each variant pair, calculate the mean score difference, the percentage of test cases where scores changed, and a simple significance test (e.g., Wilcoxon signed-rank for paired ordinal data). Package these precomputed statistics alongside the raw score matrix into the [SCORE_DATA] placeholder. This reduces hallucination risk in the LLM's statistical reasoning and ensures the prompt focuses on interpretation—identifying fragile phrases, explaining why certain variants caused shifts, and recommending robust wording. Validate the analysis output against your precomputed numbers; if the LLM reports a significant shift where your code found none, flag it for human review. For production evaluation pipelines, run this sensitivity analysis as a pre-release gate whenever judge instructions change, and store the report alongside your prompt version history.