Verification Pipeline Observability Prompt Template

This prompt is for SREs and platform engineers who own a multi-stage verification pipeline—claim extraction, evidence retrieval, matching, scoring, and reporting—and need to instrument it for production. The job-to-be-done is generating a concrete observability configuration that specifies what metrics to emit, what log events to capture, what traces to propagate, and what alert conditions to set per pipeline stage. You should use this prompt when you have a defined pipeline architecture (even if it's just a draft) and you need to move from 'we should monitor this' to a specific, implementable observability plan that your team can wire into Prometheus, OpenTelemetry, Datadog, or similar tooling.

Do not use this prompt if you are still designing the pipeline stages themselves or if you lack clarity on the input/output contracts between stages. The prompt assumes you can describe each stage's purpose, its inputs, its outputs, and its failure modes. It also assumes you have a target observability stack in mind. If you are in early exploration, start with the Verification Pipeline Harness Configuration Prompt Template to define the pipeline structure first, then return here to instrument it. This prompt is also not a substitute for runtime tracing implementation—it produces the specification, not the instrumentation code.

The ideal reader is someone who can describe a pipeline stage in operational terms: 'This stage calls an LLM with a 30-second timeout, processes up to 100 claims per batch, and can fail on malformed JSON or empty evidence sets.' The prompt will ask you to provide that context per stage, along with your latency budgets, error budgets, and existing alerting channels. The output is a structured observability plan that covers metrics (counters, gauges, histograms), log events (structured, with severity levels), trace spans (with parent-child relationships across stages), and alert rules (with thresholds, evaluation windows, and routing). Each element is tied to a specific failure mode or performance characteristic you've described.

High-risk verification pipelines—those feeding compliance reports, public-facing fact-checks, or regulated decisions—require additional scrutiny in the observability layer. The prompt includes placeholders for [RISK_LEVEL] and will generate audit-trail log events and human-review alert conditions when risk is elevated. After generating the configuration, you should validate it against your actual pipeline code: every metric and log event the prompt specifies must have a corresponding emission point in your implementation. Run a failure injection test against your instrumented pipeline and verify that the alerts fire as expected before considering the observability plan complete.

This prompt template is the core instruction set for generating an observability plan for your verification pipeline. It is designed to be copied directly into your AI harness, with square-bracket placeholders that you replace with your specific pipeline architecture, tooling, and operational requirements. The template instructs the model to act as an SRE architect and produce a structured configuration covering metrics, logs, traces, and alerts for each stage of your claim verification workflow.

codeCopy
You are an SRE architect specializing in observability for multi-stage AI verification pipelines. Your task is to produce a complete observability configuration for the pipeline described below.

## PIPELINE DEFINITION
[PIPELINE_STAGES]

## OBSERVABILITY REQUIREMENTS
- Metrics: Define key performance, accuracy, and operational metrics for each stage.
- Log Events: Specify structured log events that must be emitted at stage entry, exit, and on error.
- Traces: Describe trace context propagation rules and span attributes for distributed tracing.
- Alerts: Define alert conditions, severity levels, and notification channels for critical failures and performance degradation.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
[OUTPUT_SCHEMA]

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Generate the observability configuration.

To adapt this template, start by replacing [PIPELINE_STAGES] with a detailed description of your verification workflow. This should include the exact sequence of operations—such as claim extraction, evidence retrieval, matching, and scoring—along with the technologies used for each (e.g., a specific model endpoint, a vector database, a custom scoring function). For [CONSTRAINTS], specify your operational boundaries, such as a maximum latency budget of 500ms for the matching stage, a requirement for PII redaction in all logs, or a mandate to use only your existing Prometheus/Grafana stack. The [OUTPUT_SCHEMA] placeholder should be replaced with a strict JSON schema or a detailed markdown structure you expect the model to follow, ensuring the output is machine-readable for direct integration into your infrastructure-as-code or monitoring dashboards. Provide a few concrete [EXAMPLES] of good and bad metric definitions or alert rules to calibrate the model's output. Finally, set the [RISK_LEVEL] to high if the pipeline handles regulated content, which will instruct the model to include more conservative alerting thresholds and mandatory human-review triggers. After generating the configuration, always validate it against your schema and run a manual review to ensure no critical failure modes, such as a silent evidence retrieval failure, are left unmonitored.

This prompt template is designed to be called programmatically at pipeline configuration time, not as a one-off chat interaction. The primary integration point is a CI/CD or infrastructure-as-code pipeline that generates or updates observability configurations (OpenTelemetry, Prometheus rules, log schemas) before deploying a new verification workflow version. The prompt expects a structured [PIPELINE_DEFINITION] as input—typically a JSON or YAML object describing each stage, its inputs, outputs, retry policies, and expected latency budgets. The output is a machine-readable observability configuration that your deployment system can apply directly, so treat this prompt as a code generator with a strict output contract.

Integration pattern: Wrap the prompt in a thin service or script that reads your pipeline definition from a version-controlled file, calls the model with this template, validates the output against a JSON Schema, and writes the result to your observability stack's configuration store. Use a model with strong JSON mode and structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature=0 to maximize determinism. Implement a retry loop with exponential backoff (3 attempts max) if the output fails schema validation or if the model returns a refusal. Log every invocation—input hash, model, latency, token count, validation pass/fail, and the generated config—to your pipeline's audit store. This log becomes critical evidence when debugging why an alert didn't fire or a metric was missing during an incident.

Validation and safety gates: Before applying the generated observability config, run automated checks: (1) schema conformance—every metric, log event, trace span, and alert rule must match your organization's OpenTelemetry or monitoring schema; (2) coverage completeness—verify that each pipeline stage defined in the input has at least one corresponding metric, log event, and alert condition in the output; (3) alert threshold sanity—check that latency and error-rate thresholds are within your SLO bounds (e.g., no alert for <100ms latency if your budget is 500ms); (4) absence of hallucinated stage names or metric labels not present in the input. Fail the pipeline if any check fails. For high-risk verification pipelines (finance, healthcare, legal), require a human to approve the diff before the config reaches production.

Model choice and cost considerations: This is a configuration-generation task, not a high-frequency runtime call. You'll run it once per pipeline version change, so latency is less critical than correctness. Use your most capable model. If processing very large pipeline definitions (>50 stages), chunk the input by stage group and aggregate results, or use a model with a 128k+ context window. Cache the prompt prefix (everything before [PIPELINE_DEFINITION]) to reduce token costs across pipeline versions. The observability config itself is small relative to the input, so output token costs are negligible.

What to avoid: Do not call this prompt at runtime inside the verification pipeline itself—it's a design-time tool. Do not skip the validation step; a hallucinated metric name in production means silent failures. Do not use this prompt to generate runbooks or incident response procedures—it produces monitoring configuration, not operational documentation. If your pipeline includes stages that call external APIs or databases, ensure the generated observability config captures those dependency latencies and error codes, not just internal stage boundaries.