Few-Shot Example Consistency Check Prompt Template

This playbook is for evaluation infrastructure teams who need to verify that their LLM judges produce stable scores regardless of which few-shot examples are included or how they are ordered. If your judge prompt includes examples to calibrate scoring behavior, you must test whether those examples introduce bias. This prompt template produces a consistency report that compares scoring behavior across multiple example sets, detects example-induced drift, and flags sensitivity to example ordering. Use this before deploying a new judge prompt, after changing your example pool, or when inter-rater agreement drops unexpectedly.

Do not use this for testing the judge's accuracy against ground truth; that requires a separate alignment workflow. This prompt assumes you already have a functioning judge prompt with few-shot examples and a set of evaluation items to score. The ideal user is an evaluation engineer or ML platform developer who owns the judge prompt lifecycle and needs evidence that example selection is not silently degrading score reliability. You should have at least 20–50 evaluation items ready, a pool of candidate few-shot examples, and a baseline judge prompt whose instructions remain constant while only the examples vary.

Before running this consistency check, confirm that your judge prompt's core instructions, output schema, and scoring rubric are version-locked. If you change instructions and examples simultaneously, you won't be able to attribute drift to examples alone. After generating the consistency report, use the findings to either prune unstable examples, randomize example ordering in production, or increase the example pool size until scores stabilize. If the report reveals high sensitivity to example ordering, consider switching to zero-shot judging with a stricter rubric or implementing example selection strategies that sample diverse, representative cases rather than fixed few-shot sets.

This template is designed to be pasted directly into your evaluation harness. It instructs an LLM judge to analyze scoring consistency when few-shot examples are varied, reordered, or replaced. The prompt detects example-induced bias, sensitivity to example ordering, and overfitting signals that can silently degrade judge reliability in production. Replace each square-bracket placeholder with your actual data before running the check.

textCopy
You are an evaluation auditor analyzing the consistency of an LLM judge's scoring behavior when few-shot examples are modified.

## INPUT

**Judge System Prompt (without examples):**
[JUDGE_SYSTEM_PROMPT]

**Evaluation Rubric:**
[RUBRIC_TEXT]

**Test Items (items being scored):**
[TEST_ITEMS]

**Example Sets (multiple sets of few-shot examples to test):**
[EXAMPLE_SETS]

**Example Orderings (permutations tested per set):**
[EXAMPLE_ORDERINGS]

**Scores Produced (score matrix: example_set x ordering x test_item):**
[SCORE_MATRIX]

**Judge Rationales (optional, for deeper analysis):**
[JUDGE_RATIONALES]

## OUTPUT_SCHEMA

Return a JSON object with this exact structure:
{
  "consistency_summary": {
    "overall_stability_score": number (0-1, where 1 = perfectly consistent across all example variations),
    "stability_classification": "STABLE" | "MODERATELY_STABLE" | "UNSTABLE" | "HIGHLY_UNSTABLE",
    "items_affected_count": number,
    "total_items_count": number
  },
  "per_item_analysis": [
    {
      "item_id": string,
      "score_range": { "min": number, "max": number },
      "score_variance": number,
      "stability": "STABLE" | "UNSTABLE",
      "example_set_sensitivity": "NONE" | "LOW" | "MODERATE" | "HIGH",
      "ordering_sensitivity": "NONE" | "LOW" | "MODERATE" | "HIGH",
      "affected_by_example_set": [string],
      "affected_by_ordering": [string]
    }
  ],
  "example_set_analysis": [
    {
      "example_set_id": string,
      "mean_score_shift_vs_baseline": number,
      "bias_direction": "LENIENCY" | "SEVERITY" | "NONE",
      "items_with_shift_count": number,
      "representativeness_flag": "REPRESENTATIVE" | "POTENTIALLY_BIASED" | "UNREPRESENTATIVE"
    }
  ],
  "ordering_effects": [
    {
      "ordering_id": string,
      "primacy_effect_detected": boolean,
      "recency_effect_detected": boolean,
      "position_bias_magnitude": number (0-1),
      "affected_item_positions": [number]
    }
  ],
  "overfitting_signals": [
    {
      "signal_type": "EXAMPLE_MEMORIZATION" | "SPURIOUS_CORRELATION" | "OVERLY_SPECIFIC_PATTERN",
      "description": string,
      "severity": "LOW" | "MODERATE" | "HIGH",
      "evidence": string,
      "affected_example_set_ids": [string]
    }
  ],
  "recommendations": [
    {
      "priority": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
      "category": "EXAMPLE_REPLACEMENT" | "EXAMPLE_REORDERING" | "RUBRIC_CLARIFICATION" | "ADDITIONAL_EXAMPLES" | "HUMAN_REVIEW",
      "description": string,
      "rationale": string
    }
  ]
}

## CONSTRAINTS

- Compare scores across all example set and ordering combinations for each test item.
- Flag any item where the score changes by more than [SCORE_CHANGE_THRESHOLD] across example variations.
- Detect position bias: if scores shift systematically when examples are reordered, flag primacy or recency effects.
- Identify example sets that produce systematically higher or lower scores (leniency/severity bias).
- Check for overfitting: if certain examples cause the judge to over-apply narrow patterns, flag as overfitting risk.
- If example sets are meant to be representative of the evaluation domain, flag any set that appears unrepresentative.
- Base all findings on the provided score data. Do not speculate beyond what the data supports.
- If judge rationales are provided, use them to strengthen root-cause analysis.
- If [RISK_LEVEL] is "HIGH", require human review for any UNSTABLE or HIGHLY_UNSTABLE classification.

Adaptation guidance: Replace each placeholder with your actual data. The [SCORE_MATRIX] should be a structured mapping of example sets, orderings, and test items to their scores. If you don't have multiple orderings, set [EXAMPLE_ORDERINGS] to a single ordering and the ordering-effects analysis will return empty. Set [SCORE_CHANGE_THRESHOLD] based on your rubric's tolerance—0.5 is a reasonable default for 5-point scales. For high-stakes evaluation pipelines, set [RISK_LEVEL] to "HIGH" to enforce human review gates on unstable results. Run this check whenever you add, remove, or reorder few-shot examples in your judge prompts, and include it in your prompt release pipeline as a pre-merge gate.

This prompt is designed to be run as a meta-evaluation step within a broader judge calibration pipeline, not as a one-off manual check. The implementation harness must treat the consistency check as a batch job: it receives a set of few-shot example configurations, a target judge prompt, and a sample of test inputs, then produces a structured consistency report. The harness is responsible for iterating over example sets, collecting judge outputs, and feeding them into this prompt for analysis. Because the prompt itself performs statistical reasoning over score distributions, the harness must ensure that the input data is complete and correctly shaped before invoking the model.

The harness should be built with three layers: orchestration, validation, and logging. The orchestration layer runs the target judge prompt against a fixed test suite multiple times, each time with a different few-shot example configuration (varying example selection, ordering, and count). For each configuration, collect the judge's scores and rationales into a structured JSON array. The validation layer checks that each score entry contains the required fields (example_config_id, test_item_id, score, rationale) and that score values fall within the expected rubric range. If validation fails for a configuration, log the failure and exclude that configuration from the consistency analysis rather than passing malformed data to the model. The logging layer records the full input payload, the model's consistency report, and any validation errors to an append-only store for auditability.

Retry logic should be applied at two levels. First, if the target judge produces an unparseable output for a test item, retry that single item up to two times with a repair prompt before marking it as a failed evaluation. Second, if this consistency check prompt returns malformed JSON or fails schema validation, retry the entire analysis once with a stricter output format instruction. Implement a circuit breaker: if more than 10% of test items fail evaluation in any example configuration, abort the consistency check for that configuration and flag it for human review. The harness should also enforce a minimum sample size—reject any configuration with fewer than 20 completed test items, as the prompt's statistical claims about bias and sensitivity become unreliable with sparse data.

For model choice, use a model with strong reasoning capabilities and a large context window, as the prompt requires analyzing score matrices and detecting patterns across multiple example sets. The input payload can grow large when many configurations and test items are included. If the combined input exceeds the model's context limit, the harness should implement a windowing strategy: split configurations into batches of no more than five, run the consistency check on each batch, and then run a final aggregation pass that synthesizes the batch-level reports into a single summary. Each batch report must include the example_config_ids it covers so the aggregation pass can trace findings back to specific configurations.

Before deploying this harness to production, validate it against a golden consistency dataset: a set of example configurations with known bias patterns and ordering effects. The harness should correctly identify injected biases and flag configurations that produce unstable scores. Run this validation as part of your CI pipeline whenever the consistency check prompt or the target judge prompt changes. Finally, route any report where the severity field is high or where requires_human_review is true to a review queue. Do not automatically accept or reject example sets based solely on this prompt's output—use it as a diagnostic tool that informs human decisions about which few-shot examples to keep, reorder, or replace.