Model Endpoint Health Check Prompt

This prompt is designed for SRE teams and platform engineers who need to programmatically verify that an inference endpoint is healthy before an agent or automated workflow sends it a production workload. The core job-to-be-done is an operational readiness check: confirming that the endpoint is reachable, responding within acceptable latency thresholds, returning a healthy error rate, and serving the expected model version. It is a precondition gate, not a quality evaluation. Use it inside an agent harness, a CI/CD deployment pipeline, or a scheduled monitoring job to prevent cascading failures caused by routing traffic to a degraded or misconfigured endpoint.

The ideal user is an engineer building a production AI system who needs a structured, machine-readable health status report to drive automated decisions. Required context includes the target endpoint URL, expected model version identifier, and acceptable thresholds for latency (e.g., p95 < 2000ms) and error rate (e.g., < 1%). The prompt produces a report with fields for reachability, latency percentiles, error rate, model version consistency, and a top-level healthy: true/false verdict. Do not use this prompt for debugging model output quality, evaluating semantic correctness, or comparing response accuracy across models. It is strictly an operational signal, not a quality benchmark.

Before deploying this prompt in a production harness, define the concrete thresholds that constitute 'healthy' for your specific service level objectives (SLOs). Wire the prompt's structured output into a decision node: if healthy is false, the harness should block the downstream workflow, log the full health report for diagnosis, and potentially trigger a failover to a secondary endpoint. Avoid using this check in isolation for zero-downtime deployments; pair it with a canary analysis or gradual traffic shift to catch regressions that pass a simple health probe but degrade under load.

This template produces a structured health status report for a model inference endpoint. It is designed to be run as a pre-flight check before an agent orchestrator dispatches a workload, ensuring that latency, error rates, and model version consistency are within acceptable thresholds. The prompt assumes you have already collected raw telemetry data and are asking the model to interpret it against defined SLOs.

textCopy
You are an SRE assistant evaluating the health of a model inference endpoint before an agent workload is dispatched.

Analyze the provided telemetry data and produce a health status report.

## INPUT DATA

[TELEMETRY_DATA]

## CONSTRAINTS

- Latency SLO: p95 < [LATENCY_THRESHOLD_MS]ms
- Error Rate SLO: < [ERROR_RATE_THRESHOLD]%
- Model Version: must match [EXPECTED_MODEL_VERSION]
- Failover Readiness: [FAILOVER_ENDPOINT_STATUS]
- Data Freshness: telemetry must be no older than [MAX_DATA_AGE_MINUTES] minutes

## OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "overall_status": "healthy" | "degraded" | "unhealthy",
  "checks": {
    "latency": {
      "status": "pass" | "fail" | "no_data",
      "observed_p95_ms": number,
      "threshold_ms": number,
      "breach_magnitude_percent": number | null
    },
    "error_rate": {
      "status": "pass" | "fail" | "no_data",
      "observed_rate_percent": number,
      "threshold_percent": number,
      "breach_magnitude_percent": number | null
    },
    "model_version": {
      "status": "pass" | "fail" | "no_data",
      "observed_version": string,
      "expected_version": string
    },
    "failover_readiness": {
      "status": "pass" | "fail" | "not_configured",
      "details": string
    },
    "data_freshness": {
      "status": "pass" | "fail",
      "data_age_minutes": number,
      "max_allowed_minutes": number
    }
  },
  "recommendation": "proceed" | "proceed_with_caution" | "abort",
  "failover_recommended": boolean,
  "summary": string
}

## RULES

1. If any check has "no_data" status, set overall_status to "degraded" and recommendation to "proceed_with_caution".
2. If any critical check (latency, error_rate, model_version) fails, set overall_status to "unhealthy" and recommendation to "abort".
3. If only non-critical checks fail, set overall_status to "degraded" and recommendation to "proceed_with_caution".
4. If failover_readiness passes and overall_status is "unhealthy", set failover_recommended to true.
5. The summary must cite specific observed values and thresholds that were breached.
6. Do not invent data. If a metric is absent from [TELEMETRY_DATA], mark it as "no_data".

Adaptation notes: Replace each square-bracket placeholder with values from your monitoring stack before sending this prompt. The [TELEMETRY_DATA] placeholder should contain a structured dump of recent endpoint metrics—ideally from a monitoring API like Prometheus, Datadog, or CloudWatch. If your observability pipeline already evaluates SLOs, you can simplify the prompt by providing pre-computed pass/fail flags and asking the model only to produce the recommendation and summary. For high-risk production systems, always pair this prompt with a programmatic pre-check that validates the JSON schema before the orchestrator acts on the recommendation.

This prompt is designed to be called by a script or agent harness, not by a human operator. The harness is responsible for gathering the raw telemetry data—latency percentiles, error rates, model version strings, and endpoint status codes—from your observability stack (Prometheus, Datadog, CloudWatch, or equivalent) and injecting it into the prompt's [TELEMETRY_DATA] placeholder. The prompt itself acts as a structured reasoning step that interprets the data, identifies anomalies, and produces a health status report. It should never be the sole decision-maker for automated failover; instead, its output should feed into a deterministic circuit breaker or load balancer that enforces the actual routing decision.

The implementation flow should follow a strict sequence: (1) Query your metrics store for the last N minutes of data per endpoint. (2) Assemble the telemetry payload with explicit timestamps, endpoint identifiers, and raw metric values—do not pre-summarize or you risk hiding the signal the prompt needs. (3) Inject the payload into the prompt template and call the model with a low temperature (0.0–0.2) to maximize consistency. (4) Parse the JSON output and validate it against a strict schema that requires health_status (one of healthy, degraded, unhealthy), confidence_score (0.0–1.0), and a failover_recommendation boolean. If parsing fails, retry once with a repair prompt; if it fails again, escalate to the on-call channel and default to the last-known-good endpoint. (5) Log the full prompt, response, and validation result to your prompt observability store for later trace analysis and regression testing.

The most common production failure mode is stale telemetry. If your metrics pipeline has a 60-second ingestion lag, the prompt may be reasoning about data that no longer reflects reality. Mitigate this by attaching a data_freshness_seconds field to the telemetry payload and including a constraint in the prompt that requires the model to downgrade its confidence when data is older than your defined threshold. A second failure mode is model version inconsistency across endpoints behind a load balancer—the prompt should flag this explicitly, but your harness should also independently verify version headers before dispatching agent workloads. Finally, never allow this prompt's output to trigger an automated failover without a cooldown period; implement a minimum 30-second stabilization window in the harness to prevent flapping between endpoints during transient blips.

For model selection, prefer a fast, cost-efficient model (GPT-4o-mini, Claude Haiku, or equivalent) because this check runs on the critical path before every agent dispatch and latency adds directly to user-facing response time. If you need higher accuracy for high-stakes production traffic, consider running a cheaper model for the initial check and escalating to a more capable model only when the first pass returns degraded or unhealthy. Wire the harness to emit a health_check_duration_ms metric so you can track prompt latency separately from endpoint latency and set alerts if the check itself becomes a bottleneck.