Judge Fleet Health Dashboard Query Prompt Template

This prompt is designed for evaluation infrastructure and operations teams who need a structured, machine-readable health snapshot of their entire LLM judge fleet. The primary job-to-be-done is to query a time-series database of judge scores, agreement metrics, and drift indicators and produce a dashboard-ready summary. This summary must flag anomalies, track SLA compliance, and identify which judges or judge groups require immediate attention. The ideal user is an MLOps engineer or an evaluation platform lead who already has a running judge fleet producing metrics and needs to operationalize its monitoring.

Use this prompt when you have a pre-existing data source containing per-judge metrics over time. The prompt template is designed to accept a time range, a list of judge groups, and specific health indicators as input. It is not a prompt for scoring individual outputs or designing evaluation rubrics. It assumes you are querying aggregated operational data, not raw model outputs. The output is a structured JSON report containing sections for an executive summary, anomaly flags, SLA compliance status, and per-judge-group health details. This report is intended to be consumed by a monitoring dashboard or an automated alerting system, not read in isolation.

Do not use this prompt if you are trying to build the initial evaluation pipeline, calibrate a single judge, or diagnose a specific scoring error on one item. For those tasks, refer to the playbooks on Judge Calibration Set Generation or Disagreement Root Cause Analysis. This prompt is also unsuitable if your metrics are not yet centralized in a queryable database. The prompt's value is in synthesizing existing operational data into a coherent health report. Before using it, ensure your data pipeline reliably collects judge scores, agreement metrics, and drift indicators with consistent timestamps and judge identifiers. The next step after generating this report is to wire its structured output into your alerting platform, such as PagerDuty or a Grafana webhook, to automate incident creation for critical anomalies.

This prompt template is designed to be wired directly into an evaluation infrastructure pipeline. It instructs the model to act as an operations analyst, consuming raw judge fleet metrics and producing a structured JSON health report. The primary job is to flag SLA violations, calculate agreement trends, and classify anomaly severity—all within a specified time window. Use this when you need a consistent, automatable snapshot for dashboards or alerting systems, not a conversational summary.

textCopy
You are an evaluation infrastructure operations analyst. Your task is to generate a structured health snapshot for an LLM judge fleet based on the provided time-series metrics. The output must be machine-readable JSON suitable for dashboard ingestion and alerting pipelines.

Analyze the judge fleet data below and produce a health report covering the specified time window. Flag any judges, groups, or metrics that violate SLA thresholds. Include trend direction for agreement metrics and severity classifications for all anomalies.

[JUDGE_FLEET_METRICS_JSON]

[TIME_WINDOW_START]
[TIME_WINDOW_END]

[SLA_THRESHOLDS_JSON]

[ALERT_CONFIGURATION_JSON]

Produce a JSON health report following the output schema exactly. Include only metrics and judges with data in the specified time window. For any metric where data is missing or insufficient, set the status to "INSUFFICIENT_DATA" and include a reason. Do not fabricate data or extrapolate beyond the provided window.

To adapt this template, replace the square-bracket placeholders with your actual data payloads. [JUDGE_FLEET_METRICS_JSON] expects a structured object containing per-judge time-series data, including agreement scores, latency, and throughput. [SLA_THRESHOLDS_JSON] defines the acceptable bounds for each metric, such as a minimum Cohen's Kappa of 0.7. The [ALERT_CONFIGURATION_JSON] maps severity levels (e.g., WARNING, CRITICAL) to specific threshold breaches. The output schema, which you must provide alongside this prompt, should strictly define the expected JSON structure, including arrays for violations, trends, and judge_statuses. Do not rely on the model to invent a schema; provide it as part of the system context or a preceding user message.

Before deploying this prompt into a production pipeline, you must implement a validation layer. The model's JSON output must be parsed and validated against your defined schema. Any deviation—missing fields, incorrect types, or fabricated data—should trigger a retry or fallback. For high-stakes environments where a missed SLA violation could mask a degrading judge, route reports with CRITICAL severity anomalies for human review before they trigger automated paging. A common failure mode is the model hallucinating metric values for judges with sparse data; the explicit "INSUFFICIENT_DATA" instruction mitigates this, but your validator should still flag any report where a judge's status is "OK" but the underlying data points are fewer than your minimum sample threshold.

The Judge Fleet Health Dashboard Query prompt is designed to be called by an automated scheduler—such as a cron job, Airflow DAG, or Prefect flow—that runs at the same cadence as your evaluation dashboard refresh interval. Before invoking the LLM, your harness must assemble the JUDGE_FLEET_METRICS_JSON payload from your metrics store. This typically involves querying Prometheus, Datadog, a feature store, or an eval database to collect agreement trends, outlier counts, drift flags, and SLA compliance data for the specified time window. Validate that all required fields are present, timestamps fall within the expected range, and numeric fields contain valid values before passing the payload to the model.

After receiving the LLM response, validate the output JSON against the expected contract: a top-level health_report object containing fleet_summary, alert_items, and recommended_actions arrays, with each alert item including severity, metric, affected_judges, and evidence fields. If validation fails, retry once by appending the validation error message to the original prompt as additional context—this gives the model a chance to correct structural mistakes. If the retry also fails, log the full failure details and emit an UNHEALTHY status to your dashboard with the validation error as the reason. Route any alerts with critical severity to your incident management system (PagerDuty, Opsgenie, or similar) using the mappings defined in ALERT_CONFIGURATION_JSON, which should specify routing keys, severity thresholds, and suppression windows.

For high-stakes deployments—such as judge fleets used in regulated domains, safety-critical evaluations, or customer-facing quality gates—require human review of any report containing critical severity alerts before automated actions are taken. Log every query payload, model response, validation result, and routing decision for auditability. Store these logs in a queryable format (structured JSON in your observability platform) so you can trace disagreements back to specific prompt invocations. When selecting a model for this workflow, prefer one with strong JSON output reliability and low latency at your query volume; if cost is a concern, consider routing routine health checks to a faster, cheaper model and reserving a more capable model for reports that trigger alerts or require human review.