Continuous Monitoring Pipeline Prompt Template

This prompt is for evaluation infrastructure teams who have moved beyond ad-hoc spot checks and need a production-grade monitoring configuration for their LLM judge fleet. The primary job-to-be-done is generating a complete, actionable monitoring plan—not running a single evaluation, but defining how the evaluation infrastructure observes itself over time. You should use this prompt when you have an existing judge fleet, known evaluation volumes, and defined reliability targets (e.g., inter-rater agreement thresholds, maximum acceptable drift). The ideal user is a platform engineer or evaluation lead who understands their current judge composition, the types of evaluations being run, and the operational constraints of their pipeline.

Before using this prompt, you must assemble concrete context: the list of judges in your fleet with their model identifiers and roles, your typical daily or weekly evaluation volume, your service-level objectives for judge agreement (e.g., Cohen's Kappa > 0.75), and any existing alerting or dashboard infrastructure you need to integrate with. The prompt expects this context as structured inputs—it will not invent reasonable defaults for your specific operational environment. If you lack historical judge performance data, run the Inter-Rater Agreement Score Calculation or Judge Alignment with Human Gold Standard prompts first to establish baselines. This prompt is not a replacement for those foundational measurement steps; it is the configuration layer that sits on top of them.

Do not use this prompt when you are still prototyping your judge fleet, when you have fewer than three judges evaluating the same item type, or when your evaluation pipeline lacks persistent score storage. The monitoring configuration this prompt produces assumes you have a stream of scored items to sample from and a way to execute statistical process control checks. If you are in early-stage rubric design or judge selection, start with the Rubric Clarity and Judge Divergence Diagnostic or Judge Replacement Candidate Evaluation prompts instead. This prompt also does not replace human review for high-stakes evaluation domains—it tells you when to escalate to human review, but the review process itself must exist outside the generated configuration.

This template produces a complete monitoring configuration for an LLM judge fleet. It defines sampling strategies, statistical process control rules, regression test integration, and dashboard feed generation. The output is designed to be consumed by an automated pipeline orchestrator, not read as a standalone document. Copy the template, replace each square-bracket placeholder with your specific values, and wire the output into your evaluation infrastructure.

textCopy
You are a monitoring pipeline architect for LLM evaluation systems. Your task is to produce a complete continuous monitoring configuration for a fleet of LLM judges. The configuration must be actionable by an automated pipeline orchestrator and must include sampling strategies, alert rules, regression test integration, and dashboard feed specifications.

## INPUTS

### Judge Fleet Definition
[JUDGE_FLEET: List of judge identifiers, model versions, and assigned evaluation criteria. Example: judge-001 (gpt-4o, factuality), judge-002 (claude-sonnet, instruction-adherence)]

### Evaluation Criteria Monitored
[CRITERIA: List of evaluation dimensions being scored. Example: factuality, groundedness, instruction-adherence, tone-appropriateness]

### Historical Baseline Data
[BASELINE_DATA: Summary of historical agreement metrics, score distributions, and known failure modes. Include time range and sample sizes.]

### Pipeline Integration Points
[PIPELINE_HOOKS: Where in the existing evaluation pipeline monitoring checks should be inserted. Example: post-score-collection, pre-dashboard-publish, on-regression-test-completion]

### Alerting and Escalation Rules
[ALERT_CONFIG: Alert thresholds, notification channels, suppression windows, and escalation paths. Example: agreement-drop-below-0.7 triggers Slack alert to #eval-alerts; sustained-drift-3-windows escalates to PagerDuty]

### Dashboard Requirements
[DASHBOARD_SPEC: Required dashboard panels, refresh cadence, metric groupings, and stakeholder views. Example: fleet-health-overview (hourly), per-judge-drift-trend (daily), agreement-heatmap (real-time)]

### Regression Test Suite
[REGRESSION_SUITE: Description of existing regression test cases, golden datasets, and expected score ranges. Include test execution frequency.]

### Output Schema
[OUTPUT_SCHEMA: Expected structure for the monitoring configuration. Example: JSON with sections for sampling_rules, control_charts, alert_definitions, dashboard_queries, regression_integration]

### Constraints
[CONSTRAINTS: Specific limitations or requirements. Example: max-sampling-rate-100-per-minute, min-agreement-threshold-0.65, required-human-review-for-score-drops-exceeding-0.2]

## INSTRUCTIONS

1. Design a stratified sampling strategy that ensures coverage across judges, criteria, and time windows without exceeding rate limits.
2. Define statistical process control rules with upper and lower control limits based on historical baseline data. Specify which metrics trigger alerts and at what thresholds.
3. Create regression test integration rules that automatically run the regression suite when drift is detected and compare results against expected score ranges.
4. Generate dashboard feed queries that surface fleet health, agreement trends, outlier counts, and drift severity by stakeholder view.
5. For each alert definition, include: trigger condition, severity level, notification channel, suppression window, and recommended response action.
6. Flag any configuration gaps where insufficient baseline data prevents reliable control limit calculation.
7. Include a validation section that checks the configuration for internal consistency before deployment.

## OUTPUT FORMAT

Return a JSON object with the following structure:
{
  "monitoring_config": {
    "sampling_strategy": { ... },
    "control_rules": [ ... ],
    "alert_definitions": [ ... ],
    "regression_integration": { ... },
    "dashboard_feeds": [ ... ],
    "validation_checks": [ ... ],
    "gaps_and_risks": [ ... ]
  }
}

## RISK LEVEL
[RISK_LEVEL: high | medium | low]

If RISK_LEVEL is high, require human approval before any automated action beyond alerting. Include explicit approval gates in the configuration.

After copying the template, replace each placeholder with concrete values from your evaluation infrastructure. The JUDGE_FLEET placeholder expects a list of judge identifiers with their model versions and assigned criteria—this is the foundation that sampling and alert rules reference. The BASELINE_DATA placeholder is critical: without sufficient historical data, control limits will be unreliable, and the configuration should flag this gap rather than produce misleading thresholds. The PIPELINE_HOOKS placeholder determines where in your existing orchestration the monitoring checks execute; wire these to actual hook points in your pipeline code, not abstract descriptions. Before deploying the generated configuration, run the internal validation checks the prompt produces and verify that alert thresholds align with your team's acceptable false-positive rate. For high-risk evaluation pipelines where judge scores gate production deployments, always route the generated configuration through human review before activation.

The Continuous Monitoring Pipeline Prompt Template is designed to be called by an orchestration layer, not directly by a human. The prompt expects structured metadata about your judge fleet, evaluation volume, and existing infrastructure. The harness must assemble this context before invoking the model. The output is a machine-readable monitoring configuration (JSON or YAML) that downstream systems consume to configure sampling, statistical process control (SPC) rules, and dashboard feeds. Treat the generated configuration as a starting point that requires validation before deployment.

The implementation harness should enforce a strict contract around the prompt's input and output. Before calling the model, validate that the [JUDGE_FLEET_METADATA] includes judge IDs, model versions, and historical agreement baselines. The [EVALUATION_VOLUME] must specify items-per-day and peak throughput to allow the model to recommend appropriate sampling rates. After receiving the output, run a schema validator against the expected monitoring configuration shape: check that sampling_strategy contains a valid method (e.g., fixed_rate, adaptive, triggered), that spc_rules define control_limit_type and window_size, and that regression_test_suite maps to known test suite IDs in your system. Reject configurations that reference non-existent judges, test suites, or metrics. Log every generation attempt with the input hash, output hash, and validation result for auditability.

Wire the validated configuration into your pipeline's control plane. Sampling rules should feed into your evaluation scheduler to determine which items get scored by multiple judges for agreement checks. SPC rules should be converted into alert queries against your metrics store (e.g., Prometheus, Datadog) with thresholds for judge drift, agreement degradation, and outlier frequency. The dashboard_feed section should be translated into dashboard JSON or widget configurations for your observability platform. Implement a dry-run mode that simulates the configuration against historical data for one week before activating it in production. If the dry-run generates false alarms or misses known issues, route the configuration back through the prompt with the dry-run results as additional [CONTEXT] for refinement.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (GPT-4, Claude 3.5 Sonnet, or equivalent). Set temperature=0 to maximize determinism in configuration generation. Implement retries with exponential backoff if the output fails schema validation; after three failed attempts, escalate to a human reviewer with the failed outputs and validation errors. Store all generated configurations in a versioned registry with a rollback mechanism. Before promoting a configuration to production, run it through a regression test that verifies the pipeline still detects known judge reliability issues from your historical incident library. Never auto-apply a monitoring configuration without human approval if it modifies alert thresholds or regression test selection for high-stakes evaluation pipelines.