Router Health Check Prompt Template

This prompt is designed for AI operations teams and platform engineers who need to run periodic health assessments of a model router using production trace samples. The router is the decision layer that selects which model handles each request. When routing degrades, costs rise, latency spikes, and user experience suffers. This prompt evaluates a batch of production traces against defined health thresholds for latency, decision consistency, fallback rate, and error rate. Use it to produce a structured health status report suitable for dashboard integration, alerting pipelines, and operational reviews.

This is not a real-time monitoring query. It is a batch assessment tool for scheduled health checks, post-deployment validation, and incident follow-up. Before using this prompt, you must have access to a representative batch of production traces that include routing metadata such as the selected model, fallback activations, per-step latency, and error codes. The prompt expects these traces to be provided as structured input, along with your organization's specific health thresholds for each metric. Without this context, the assessment will be generic and may miss critical signals. Do not use this prompt for real-time alerting, single-trace debugging, or as a substitute for a metrics dashboard. It is a diagnostic tool for periodic review, not a continuous monitoring system.

After running this prompt, you will receive a structured health status report that classifies the router's state as healthy, degraded, or critical for each dimension. The report includes specific trace IDs for anomalies, making it actionable for follow-up investigation. Wire the output into your operational dashboards or alerting pipelines, but always pair it with a human review step before taking automated action on routing configuration changes. The next step is to adapt the prompt template with your specific thresholds and trace schema, then integrate it into a scheduled evaluation harness.

This prompt template is designed to be integrated into an automated observability pipeline. It ingests a batch of production trace segments and evaluates the router's performance against predefined health thresholds for latency, decision consistency, fallback rate, and error rate. The output is a structured health status report intended for dashboard consumption and automated alerting. Before using this template, ensure you have collected a representative sample of traces from your production monitoring system, including the original request context, the routing decision, and the outcome metadata.

textCopy
You are an AI operations auditor performing a health check on a model routing system.

Your task is to analyze a batch of production trace samples and produce a structured health status report. Evaluate the router's performance against the provided thresholds.

# INPUT DATA

You will receive a JSON array of trace objects. Each object contains:
- trace_id: string
- timestamp: ISO 8601 string
- request_summary: string (the user's original request)
- routing_decision: object
  - selected_model: string
  - decision_latency_ms: integer
  - fallback_activated: boolean
  - fallback_reason: string or null
- outcome: object
  - status: "success" | "error" | "timeout"
  - total_latency_ms: integer
  - error_code: string or null

# HEALTH THRESHOLDS

Evaluate the batch against these thresholds:
- [LATENCY_THRESHOLD_MS]: Maximum acceptable decision latency in milliseconds.
- [FALLBACK_RATE_THRESHOLD]: Maximum acceptable fallback activation rate (0.0 to 1.0).
- [ERROR_RATE_THRESHOLD]: Maximum acceptable error rate (0.0 to 1.0).
- [CONSISTENCY_CHECK_ENABLED]: If true, check for decision consistency on semantically similar requests.

# OUTPUT SCHEMA

Return a single JSON object conforming to this schema:
{
  "report_id": "string",
  "assessment_timestamp": "ISO 8601 string",
  "batch_summary": {
    "total_traces": integer,
    "time_window_start": "ISO 8601 string",
    "time_window_end": "ISO 8601 string"
  },
  "health_checks": [
    {
      "check_name": "string",
      "status": "healthy" | "degraded" | "unhealthy",
      "observed_value": number,
      "threshold": number,
      "breaching_trace_ids": ["string"],
      "details": "string"
    }
  ],
  "overall_status": "healthy" | "degraded" | "unhealthy",
  "anomalies": [
    {
      "trace_id": "string",
      "anomaly_type": "string",
      "description": "string"
    }
  ],
  "recommendations": ["string"]
}

# CONSTRAINTS

- Base all findings strictly on the provided trace data. Do not invent or assume data.
- If a threshold is not breached, set status to "healthy".
- If a threshold is breached by less than 10%, set status to "degraded".
- If a threshold is breached by 10% or more, set status to "unhealthy".
- For the consistency check, group requests by semantic similarity and flag groups where different models were selected without a clear differentiating factor.
- If [CONSISTENCY_CHECK_ENABLED] is false, omit the consistency health check.
- The "details" field must contain a concise, data-driven explanation of the finding.

# INPUT DATA

[TRACE_BATCH]

To adapt this template for your environment, replace the square-bracket placeholders with production data. [TRACE_BATCH] should be populated with a JSON array of trace objects from your observability store. [LATENCY_THRESHOLD_MS], [FALLBACK_RATE_THRESHOLD], and [ERROR_RATE_THRESHOLD] should be set to your team's SLO values. The [CONSISTENCY_CHECK_ENABLED] boolean allows you to toggle a more expensive semantic analysis. The output schema is designed to be ingested directly by a monitoring dashboard or alerting system. If any check returns an "unhealthy" status, the overall_status should trigger an alert. Always validate the model's JSON output against the schema before posting it to a dashboard, and log the raw response for debugging.

The Router Health Check prompt is designed to run as a periodic batch job, not a real-time per-request check. You feed it a representative sample of production traces from a fixed time window (e.g., the last 15 minutes) and receive a structured health report. The implementation harness must handle trace extraction, prompt assembly, response validation, and integration with your alerting and dashboarding systems. Treat the prompt output as a signal that can trigger an on-call page, not as a replacement for your metrics pipeline.

Wiring the workflow: 1) Trace extraction: Query your observability store (e.g., ClickHouse, Elasticsearch, Datadog) for routing events in the target window. Select a statistically meaningful sample—at least 200 traces, stratified by model if your router serves multiple models. 2) Prompt assembly: Inject the sampled traces into the [TRACE_SAMPLE] placeholder along with your [HEALTH_THRESHOLDS] (max acceptable fallback rate, p95 routing latency, error rate ceiling). 3) Model selection: Use a model with strong structured output capabilities and a context window large enough for your trace sample. GPT-4o or Claude 3.5 Sonnet are good defaults; avoid smaller models that may miscount or hallucinate statistics. 4) Validation: Parse the JSON output and validate that all required fields (overall_status, fallback_rate, routing_latency_p95, error_rate, decision_consistency_score, anomalies, recommendations) are present and within expected ranges. Reject reports where fallback_rate exceeds 1.0 or routing_latency_p95 is negative. 5) Retry logic: If validation fails, retry once with the same input. If the second attempt also fails, log the raw response and escalate to a human reviewer—do not silently discard a failed health check.

Alerting and dashboard integration: Map overall_status to your alerting severity: healthy clears any open router health alerts, degraded triggers a warning-level notification, and unhealthy fires a critical page. Push the numeric metrics (fallback_rate, routing_latency_p95, error_rate) to your time-series dashboard alongside your existing router metrics so you can correlate the LLM's assessment with your ground-truth instrumentation. What to avoid: Do not run this prompt on every trace—it is too expensive and slow for real-time routing decisions. Do not use the prompt output to automatically change router configuration; the health report informs a human or a gated deployment pipeline, it does not self-heal. If the prompt consistently reports unhealthy while your metrics dashboard shows normal operation, investigate whether your trace sample is biased or your thresholds are misconfigured before trusting the prompt output.