Inter-Rater Reliability Threshold Alert Prompt Template

This prompt is designed for evaluation infrastructure teams who need to automate monitoring of their LLM judge fleet. Use it when you have multiple judges scoring the same items and you need a system that generates a structured alert payload the moment inter-rater agreement drops below a configurable threshold. The primary job-to-be-done is not statistical analysis in isolation—it is the production of an actionable, machine-readable alert that can trigger paging, open an incident, or feed a monitoring dashboard. The ideal user is an MLOps engineer, evaluation platform developer, or AI reliability lead who already has a scoring pipeline producing judge scores and needs a reliable alerting contract on top of it.

Before using this prompt, you must have a matrix of scores from at least two judges across a batch of items, along with the agreement metric you intend to monitor (e.g., Cohen's Kappa, Fleiss' Kappa, or percent agreement). The prompt ingests this matrix, computes agreement metrics, identifies the most divergent judge pairs, surfaces sample items causing the disagreement, and recommends concrete actions. It is not a prompt for calculating agreement scores in isolation—it assumes your pipeline already computes the raw metrics. Instead, it acts as the alert-generation layer that triggers when reliability degrades. The output is a structured payload containing the alert severity, affected judge pairs, divergence samples, and recommended actions, ready for ingestion by your incident management system.

Do not use this prompt when you need a statistical tutorial on inter-rater reliability, when you are designing a new scoring rubric from scratch, or when you are performing a one-off manual review of judge quality. It is also unsuitable for real-time per-item alerting—this prompt operates on batches and is designed for periodic monitoring windows, not streaming individual score disagreements. If your goal is to debug why a specific judge scored a single item differently, use the Disagreement Root Cause Analysis Prompt Template instead. For continuous monitoring pipeline integration, pair this prompt with the Continuous Monitoring Pipeline Prompt Template to define sampling strategies and alert suppression windows. Always configure a minimum sample size threshold before triggering this prompt to avoid false alarms from small batches where agreement metrics are statistically unreliable.

This prompt template is designed to be called by an automated monitoring system when agreement metrics between LLM judges fall outside acceptable bounds. It consumes raw agreement data, affected judge pairs, and sample divergence items to produce a structured alert payload suitable for logging, dashboard display, or pager escalation. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to wire into a production harness that queries your evaluation database and feeds the results into the model.

textCopy
You are an evaluation reliability monitor. Your task is to generate a structured alert payload when inter-rater reliability between LLM judges drops below the configured threshold.

## INPUT DATA
- Agreement Metrics: [AGREEMENT_METRICS_JSON]
- Affected Judge Pairs: [AFFECTED_JUDGE_PAIRS]
- Sample Divergence Items: [DIVERGENCE_SAMPLES]
- Threshold Configuration: [THRESHOLD_CONFIG]
- Historical Baseline: [HISTORICAL_BASELINE]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "alert_id": "string",
  "severity": "critical|warning|info",
  "timestamp": "ISO-8601",
  "summary": "One-sentence description of the reliability breach",
  "metrics": {
    "current_agreement": "number",
    "threshold": "number",
    "metric_type": "cohens_kappa|fleiss_kappa|percent_agreement",
    "baseline_agreement": "number",
    "deviation_from_baseline": "number"
  },
  "affected_judges": [
    {
      "judge_pair": ["judge_id", "judge_id"],
      "pairwise_agreement": "number",
      "divergence_count": "number"
    }
  ],
  "divergence_samples": [
    {
      "item_id": "string",
      "judge_scores": {"judge_id": "score", "judge_id": "score"},
      "disagreement_type": "score_divergence|rubric_interpretation|missing_context",
      "excerpt": "string"
    }
  ],
  "recommended_actions": [
    {
      "action": "string",
      "priority": "immediate|short_term|investigate",
      "owner": "string"
    }
  ],
  "suppression_applicable": "boolean",
  "escalation_path": "string"
}

## CONSTRAINTS
- Only generate an alert if current agreement is below the configured threshold.
- Classify severity as "critical" if deviation from baseline exceeds 0.15, "warning" if between 0.05 and 0.15, and "info" otherwise.
- Include no more than 5 divergence samples, prioritizing the largest score gaps.
- If a suppression window is active per [THRESHOLD_CONFIG], set suppression_applicable to true and reduce severity by one level.
- Recommended actions must be specific to the disagreement type observed.
- Do not fabricate judge IDs, item IDs, or scores not present in the input data.
- If historical baseline is unavailable, note this in the summary and use the threshold as the sole comparison point.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[HUMAN_REVIEW_REQUIRED if severity is critical and suppression is not applicable]

To adapt this template, replace each placeholder with production data before sending it to the model. [AGREEMENT_METRICS_JSON] should contain the computed agreement scores from your evaluation pipeline. [AFFECTED_JUDGE_PAIRS] lists the specific judge pairs whose pairwise agreement fell below threshold. [DIVERGENCE_SAMPLES] provides the raw items where judges disagreed most, including their scores and the item content. [THRESHOLD_CONFIG] defines the acceptable agreement floor, suppression window rules, and escalation contacts. [HISTORICAL_BASELINE] supplies the expected agreement range from prior periods. [FEW_SHOT_EXAMPLES] should include 2-3 example alert payloads showing correct severity classification and action recommendation patterns. The [HUMAN_REVIEW_REQUIRED] flag should be set to true for critical alerts in production to ensure a human validates the alert before paging on-call engineers.

This prompt is designed to be invoked by an automated monitoring job, not a human chat interface. The primary integration point is a scheduled task—typically a cron job, Airflow DAG, or Temporal workflow—that runs after each batch evaluation cycle. The harness must collect the required inputs: the agreement matrix from your judge fleet, the current threshold configuration, and any active suppression windows. The prompt should only fire when preliminary statistical checks (e.g., a simple percent-agreement calculation) already suggest a potential breach, reducing unnecessary LLM calls and cost.

The implementation flow follows a strict sequence. First, the scheduler queries your evaluation database for the latest inter-rater agreement scores across all judge pairs. Second, a lightweight pre-check compares raw agreement values against the configured [ALERT_THRESHOLD]. If no breach is detected, the job exits. If a breach is possible, the harness assembles the prompt payload: the full agreement matrix, the list of affected judge pairs, a sample of divergent items with both scores and rationales, and the current [SUPPRESSION_WINDOW_CONFIG]. The LLM call is made with a low temperature (0.0–0.1) to ensure deterministic alert payloads. The output is parsed as JSON and validated against the expected schema before any alert is dispatched.

Validation and suppression logic is critical here. Before sending an alert, the harness must check: (1) Is the alert within an active suppression window? If a previous alert for the same judge pair was sent within the configured cooldown period, suppress and log. (2) Does the output JSON parse correctly and contain all required fields (alert_id, severity, affected_pairs, sample_divergence_items, recommended_actions)? If parsing fails, retry once with a stricter output schema instruction; if it fails again, escalate to the on-call channel with the raw output. (3) Are the recommended actions safe to automate? Actions like 'escalate to human review' should create a ticket; actions like 'recalibrate judge' should never be executed automatically. The harness must enforce a human-in-the-loop gate for any action beyond notification.

For model choice, use a model with strong JSON mode and instruction-following capabilities. GPT-4o or Claude 3.5 Sonnet are appropriate; avoid smaller models that may hallucinate metric values or skip required output fields. If cost is a concern, consider routing only high-severity potential breaches to a larger model and using a smaller model for low-severity checks, but always validate the output schema regardless of model. Log every invocation—including the prompt, the raw response, the parsed output, and the suppression decision—to your observability platform (e.g., Datadog, Grafana, or a custom trace store) for audit and debugging.

Finally, wire the validated alert payload into your incident management system. Map severity levels to PagerDuty severities or Slack channel routing rules. Include the sample_divergence_items directly in the notification so responders can immediately see the evidence without querying another system. Avoid sending raw agreement matrices to wide-audience channels; the alert payload should be the curated, actionable summary this prompt produces. Test the full harness end-to-end with known agreement drops before deploying to production, and set up a synthetic monitor that periodically injects a known-bad agreement matrix to verify the pipeline fires correctly.