Observability Pipeline Config Change Analysis Prompt

This prompt is designed for Site Reliability Engineers (SREs) and platform engineers who are reviewing proposed changes to observability pipeline configurations—specifically for tools like OpenTelemetry Collector, Fluentd, or Vector—as part of a pull request or change management process. The primary job-to-be-done is to prevent production incidents caused by misconfigured telemetry pipelines. The ideal user has a deep understanding of their system's SLOs but needs a structured, automated second pair of eyes to catch subtle regressions in sampling logic, log redaction, metric cardinality, and pipeline throughput that are easy to miss during manual review. The required context is a unified or side-by-side diff of the configuration file changes, along with the relevant service's SLO definitions and any known high-priority alerts that depend on the telemetry data being modified.

You should use this prompt when a configuration change touches any part of the pipeline that transforms, filters, routes, or samples telemetry data before it reaches its final destination. This includes modifying processors in an OpenTelemetry Collector pipeline, changing <filter> or <match> directives in Fluentd, or altering transforms and sinks in a Vector configuration. The prompt is specifically valuable when the change is not a simple endpoint URL update but a structural or behavioral modification. Do not use this prompt for reviewing changes to the destination backends themselves (e.g., an Elasticsearch index template) unless those changes are embedded in the pipeline configuration. It is also not a replacement for testing the configuration in a staging environment; it is a pre-merge risk analysis tool.

Before relying on this prompt, you must ensure the diff is clean and represents the intended change. The prompt's analysis is only as good as the input context. A common failure mode is providing a diff that includes unrelated whitespace or comment changes, which can distract the model. Another constraint is that the prompt requires explicit SLO and alert definitions to assess impact accurately; without them, the analysis will be generic. The output is a structured risk report, not a binary pass/fail. The next step after receiving the analysis is for a human to review the high and critical severity findings, correlate them against the staging environment's behavior, and approve or reject the change. Do not use this prompt to auto-approve or auto-merge configuration changes without human review, as a false negative in risk assessment could lead to a silent loss of observability during a real incident.

This template is designed to be dropped into an automated review harness or used manually by an SRE before approving a configuration change. It forces the model to reason about the change's impact on alert accuracy, metric cost, log quality, and pipeline performance. All placeholders are in square brackets and must be replaced with real data from your diff, pipeline topology, and SLO definitions before execution.

textCopy
You are an SRE reviewing a configuration change to an observability data pipeline.
Your goal is to identify risks that could cause production impact, including missed alerts, metric cardinality explosions, log data leaks, and pipeline backpressure.

## INPUT
[PIPELINE_CONFIG_DIFF]

## CONTEXT
- Pipeline type: [PIPELINE_TYPE] (e.g., OpenTelemetry Collector, Fluentd, Vector)
- Affected data types: [AFFECTED_SIGNALS] (e.g., traces, metrics, logs)
- Downstream consumers: [DOWNSTREAM_CONSUMERS] (e.g., Prometheus, Elasticsearch, Datadog)
- Relevant SLOs: [RELEVANT_SLOS]

## CONSTRAINTS
- Do not flag cosmetic or formatting-only changes.
- For each finding, cite the specific line or block in the diff that triggered it.
- If a finding is based on an assumption about runtime behavior, state the assumption explicitly.
- Do not recommend a change without explaining the risk of the current configuration.

## OUTPUT_SCHEMA
Return a valid JSON object with the following structure:
{
  "summary": "A one-paragraph summary of the change's intent and overall risk.",
  "findings": [
    {
      "severity": "CRITICAL | HIGH | MEDIUM | LOW",
      "category": "SAMPLING | REDACTION | CARDINALITY | PERFORMANCE | COVERAGE | OTHER",
      "title": "Short finding title",
      "location": "Specific line or block reference from the diff",
      "description": "Detailed explanation of the risk",
      "affected_slos": ["List of SLO names from the context that are impacted"],
      "assumptions": ["Any assumptions made about runtime behavior"],
      "recommendation": "Specific, actionable fix or validation step"
    }
  ],
  "overall_risk_assessment": "LOW | MEDIUM | HIGH | CRITICAL",
  "recommended_actions_before_merge": ["List of pre-merge actions"]
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK_LEVEL
[HUMAN_REVIEW_REQUIRED] (true/false)

Adaptation guidance: Replace [PIPELINE_CONFIG_DIFF] with the unified diff of the configuration change. Populate [PIPELINE_TYPE] and [AFFECTED_SIGNALS] to constrain the model's reasoning to the correct domain. The [RELEVANT_SLOS] field is critical—without it, the model cannot assess alerting impact. For [FEW_SHOT_EXAMPLES], provide 2-3 examples of correct findings for your specific pipeline type to calibrate severity and category selection. If the change touches production infrastructure, set [HUMAN_REVIEW_REQUIRED] to true and route the output to an approval queue rather than auto-applying.

This prompt is designed to operate inside a CI/CD pipeline or a change management webhook that fires whenever an observability pipeline configuration (OpenTelemetry Collector, Fluentd, Vector) is modified in a pull request or a deployment branch. The harness should extract the raw config diff, the full previous and proposed config files, and any relevant SLO definitions or alert rule metadata before assembling the prompt. The model call should be treated as a structured analysis step, not a final decision. Its output must be parsed into a machine-readable risk report that downstream systems can act on.

Implement the harness as an asynchronous job with a strict JSON output contract. After calling the model, validate the response against a schema that requires findings (array of objects with severity, category, affected_component, description, slo_impact, and remediation fields) and a top-level overall_risk_score (enum: LOW, MEDIUM, HIGH, CRITICAL). If validation fails, retry once with a repair prompt that includes the schema and the raw invalid output. Log both attempts. For any finding with severity CRITICAL or category sampling_rate_change that affects an active SLO, block the CI pipeline and require a human approval comment on the PR before the config can be merged. For MEDIUM findings, post a non-blocking review comment. For LOW, log and proceed.

Model choice matters here. Use a model with strong structured output support and a context window large enough to hold the full config files plus SLO definitions. If configs exceed the context window, preprocess them by extracting only the changed sections and their immediate dependencies (e.g., pipeline receivers, processors, and exporters connected to the modified block). Do not truncate SLO definitions. Store the full prompt, response, and validation result as an artifact attached to the deployment or PR for auditability. Avoid wiring this prompt directly to an auto-merge action without the human approval gate for destructive changes.