Outlier Judge Detection Prompt Template

This prompt is designed for evaluation infrastructure teams who need to monitor a fleet of LLM judges for anomalous scoring behavior. The core job-to-be-done is automated surveillance: you have multiple judges scoring the same or similar outputs, and you need to identify which judges are deviating from the fleet consensus in a statistically meaningful way. The ideal user is an MLOps engineer, evaluation platform lead, or AI reliability engineer who already has a pipeline producing judge scores and wants to add automated outlier detection before those scores feed into model training, regression gates, or production quality dashboards.

Use this prompt when you have a score matrix—multiple judges rating multiple items—and you need a structured report flagging outlier judges with deviation metrics, severity classifications, and root-cause hypotheses. The prompt expects inputs like judge IDs, item IDs, raw scores, and optional judge metadata (model type, prompt version, deployment date). It is not appropriate for single-judge evaluation setups, real-time scoring with no historical baseline, or cases where you lack at least three judges per item. Do not use this prompt when the scoring scale is purely categorical with no ordinal relationship, as deviation metrics like mean absolute error lose interpretability. For high-stakes decisions—such as removing a judge from a production pipeline or flagging a model for retraining—always pair this prompt's output with human review of the flagged judge's raw scores and rationales before taking action.

Before running this prompt, ensure you have a clean score matrix with consistent item-judge pairings, a defined agreement baseline (e.g., historical inter-rater reliability), and configurable severity thresholds. The prompt works best when embedded in a batch monitoring job that runs daily or weekly, comparing current judge behavior against a rolling window of recent performance. After receiving the output, validate flagged judges by spot-checking their score distributions against the fleet, reviewing their rationales for drift, and confirming that the detected deviation is not caused by a legitimate shift in item difficulty. If the prompt flags too many judges, recalibrate your deviation thresholds; if it flags none when you suspect drift, widen the detection window or lower the sensitivity.

This prompt template is designed to be dropped into an automated evaluation monitoring pipeline. It accepts a matrix of scores from multiple judges across multiple items, identifies which judges are producing anomalous scoring patterns, and returns a structured report. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to parameterize in code before sending to the model. Adapt the [JUDGE_SCORE_MATRIX] format to match your internal data structures, and adjust the [DEVIATION_THRESHOLDS] based on your fleet's historical agreement baselines.

textCopy
You are an evaluation auditor analyzing scoring patterns from a fleet of LLM judges. Your task is to detect outlier judges whose scoring behavior deviates significantly from the fleet consensus.

## INPUT DATA

### Judge Score Matrix
[JUDGE_SCORE_MATRIX]

Format: A table or JSON structure where rows are items, columns are judges, and cells contain numeric scores. Include judge identifiers and item identifiers.

### Historical Baseline (Optional)
[HISTORICAL_BASELINE]

If provided, use this to establish expected agreement ranges and per-judge reliability scores. If not provided, derive baselines from the current matrix.

## DETECTION PARAMETERS

- Deviation Thresholds: [DEVIATION_THRESHOLDS]
  - Specify thresholds for flagging (e.g., z-score > 2.0, mean absolute deviation > 1.5, agreement rate < 0.7)
- Minimum Items for Analysis: [MIN_ITEMS]
- Judge Grouping (Optional): [JUDGE_GROUPS]
  - If judges belong to groups (e.g., model family, prompt version), analyze within-group and cross-group patterns

## OUTPUT SCHEMA

Return a JSON object with this structure:
{
  "fleet_summary": {
    "total_judges": number,
    "total_items": number,
    "overall_agreement_rate": number,
    "mean_pairwise_correlation": number,
    "analysis_timestamp": string
  },
  "outlier_judges": [
    {
      "judge_id": string,
      "severity": "low" | "medium" | "high" | "critical",
      "deviation_metrics": {
        "mean_absolute_deviation_from_fleet_mean": number,
        "z_score_of_mean": number,
        "agreement_rate_with_fleet": number,
        "score_variance_ratio": number
      },
      "pattern_type": "leniency_skew" | "severity_skew" | "inconsistent" | "scale_compression" | "bimodal" | "item_specific",
      "affected_items": [string],
      "root_cause_hypotheses": [
        {
          "hypothesis": string,
          "confidence": "low" | "medium" | "high",
          "supporting_evidence": string
        }
      ],
      "recommended_action": string
    }
  ],
  "fleet_health_assessment": {
    "status": "healthy" | "warning" | "degraded" | "critical",
    "summary": string,
    "requires_human_review": boolean
  }
}

## CONSTRAINTS

[CONSTRAINTS]

Default constraints:
- Flag judges only when deviation is statistically meaningful given the number of items
- Distinguish between systematic bias (consistent skew) and random inconsistency
- For severity "critical" or "high", require at least two distinct deviation signals
- If fewer than [MIN_ITEMS] items are available, note low-confidence in the fleet_summary
- Do not flag judges for deviation that falls within expected noise ranges
- Include specific item IDs and score comparisons in supporting evidence

## ANALYSIS STEPS

1. Compute fleet-level agreement statistics (pairwise correlations, mean scores per item)
2. For each judge, compute deviation from fleet mean, agreement rate, and score distribution characteristics
3. Identify judges exceeding [DEVIATION_THRESHOLDS]
4. Classify pattern type by analyzing whether deviation is directional, item-specific, or random
5. Generate root-cause hypotheses by examining judge behavior patterns against known failure modes
6. Assign severity based on deviation magnitude, pattern consistency, and impact on downstream decisions
7. Produce fleet health assessment

## KNOWN FAILURE MODES TO CHECK

- Rubric misinterpretation (judge applies different standard)
- Context truncation (judge received incomplete item context)
- Scale misuse (judge compresses range or avoids extremes)
- Position bias (judge scores differently based on item order)
- Few-shot example contamination (judge overfits to examples)
- Model-specific behavior (underlying model produces different scoring distributions)

Return only the JSON object. Do not include explanatory text outside the JSON.

To adapt this template for your pipeline, replace the [JUDGE_SCORE_MATRIX] with your actual score data structure—this could be a JSON array of objects, a CSV-formatted string, or a markdown table depending on your model's context window and parsing preferences. Set [DEVIATION_THRESHOLDS] based on your fleet's historical performance; start with z-score > 2.0 and agreement rate < 0.7 as initial values, then calibrate against known-good and known-bad judge behavior from your own evaluation logs. If you maintain per-judge reliability scores from prior calibration runs, pass those in [HISTORICAL_BASELINE] to improve detection accuracy and reduce false positives on judges with naturally wider scoring distributions.

Before deploying this prompt into an automated pipeline, implement a validation layer that checks the output JSON against the expected schema. Pay particular attention to the severity field—a misclassification here can trigger unnecessary human review or, worse, silently ignore a degrading judge. For high-risk evaluation pipelines where judge scores gate model releases or content decisions, route all critical and high severity detections to a human reviewer before taking automated action such as removing a judge from the fleet. Log every outlier detection event with the full prompt input, model response, and validation result so you can audit false-positive rates and tune [DEVIATION_THRESHOLDS] over time.

The Outlier Judge Detection prompt is designed to operate as a scheduled or event-driven job within an evaluation infrastructure, not as a one-off manual query. It expects a structured payload of recent judge scores, historical performance baselines, and fleet-wide agreement statistics. The harness should assemble this context from a centralized evaluation store—such as a database of scored items with judge IDs, timestamps, and optional rationale traces—before invoking the model. Because the prompt produces a severity classification and root-cause hypotheses, the calling application must treat the output as a diagnostic signal, not a final verdict. The recommended model tier is a capable reasoning model (e.g., GPT-4, Claude 3.5 Sonnet) due to the multi-step analytical reasoning required; smaller models may conflate correlation with causation or miss subtle deviation patterns.

The implementation should enforce a strict input schema before the prompt is assembled. Required fields include: a judge identifier, a time-bounded score vector, a fleet-wide agreement matrix for the same period, and the judge's historical mean and variance. The harness should validate that the score vector length meets a minimum sample size (e.g., at least 20 scored items) to avoid spurious outlier flags from small samples. After the model returns its analysis, a post-processing validator must parse the structured output—ideally JSON with fields for outlier_flag, severity, deviation_metrics, and root_cause_hypotheses—and confirm that the reported deviation metrics are internally consistent with the input data. If the output fails schema validation or contains contradictory claims, the harness should retry once with a simplified prompt variant that omits root-cause analysis and focuses only on flagging and metrics. Log every invocation, including the input summary statistics, the raw model response, and the validation result, to an observability system for later audit and threshold calibration.

For high-stakes evaluation pipelines where judge scores gate model releases or regulatory evidence, this prompt's output must route through a human review queue before any automated action is taken. The harness should enrich the model's output with direct links to the specific scored items that contributed most to the deviation signal, enabling a human reviewer to spot-check the judge's behavior. Avoid using this prompt to automatically remove or down-weight judges without human confirmation; false-positive outlier flags can silently erode evaluation diversity. After initial deployment, run this prompt against a historical period with known judge behavior to calibrate the severity thresholds and measure the false-positive rate against ground-truth judge reliability data. This calibration step is essential before enabling any automated alerting or judge quarantine logic.