OpenTelemetry Collector Configuration Prompt

Platform engineers and SREs should use this prompt when they need a complete, standards-compliant first draft of an OpenTelemetry Collector configuration. The job-to-be-done is translating a set of high-level observability requirements—such as signal types, sampling rules, PII redaction policies, and export destinations—into a valid YAML pipeline definition. The ideal user is someone who understands the Collector's component model (receivers, processors, exporters, connectors, and service pipelines) but wants to accelerate the configuration authoring process and reduce the risk of misconfigurations that violate data governance or signal integrity constraints.

This prompt is most effective when you provide concrete constraints. For example, specify that the pipeline must accept OTLP and Prometheus receiver inputs, apply tail-based sampling for traces, redact email and ip_address attributes with a redaction processor, and export to both a vendor SaaS backend and a local Jaeger instance. The prompt will generate the corresponding receivers, processors, exporters, and service.pipelines stanzas, ensuring the data flow respects the order of operations you defined. It also validates the output against vendor-agnostic standards, such as ensuring the batch processor is placed before the exporter queue and that PII redaction occurs before any export.

Do not use this prompt for runtime debugging of an already-deployed Collector instance, as it is designed for greenfield configuration generation, not log analysis or metric interpretation. It is also not a substitute for generating application-level instrumentation code; it focuses exclusively on the Collector's configuration surface. If your goal is to troubleshoot a broken pipeline, investigate dropped spans, or write OpenTelemetry SDK code in a specific language, you need a different workflow. After generating the configuration, always review the output for environment-specific details like TLS certificate paths, authentication headers, and endpoint URLs, and test it in a staging Collector instance before production deployment.

The following prompt template is designed to be pasted directly into your AI workflow. It instructs the model to act as a platform engineering architect and produce a complete, vendor-agnostic OpenTelemetry Collector configuration. The output is not just YAML; it includes a validation rationale, a data flow explanation, and a governance checklist. Before using it, gather your specific requirements for receivers, processors, exporters, and any data governance constraints. The more precise your input in the placeholders, the more production-ready the generated configuration will be.

markdownCopy
You are a senior platform engineer specializing in observability infrastructure. Your task is to generate a production-grade OpenTelemetry Collector configuration in YAML based on the specific requirements provided.

# INPUT REQUIREMENTS
- **Telemetry Sources:** [LIST_RECEIVERS, e.g., OTLP gRPC on port 4317, Prometheus scraping on port 8888, filelog receiver for /var/log/app/*.log]
- **Processing Needs:** [LIST_PROCESSORS, e.g., batch, memory_limiter, attributes for adding 'environment: production', filter to drop health-check spans]
- **Data Destinations:** [LIST_EXPORTERS, e.g., otlp to https://my-otlp-backend:4318, debug for local testing, prometheusremotewrite to https://my-metrics-backend/api/v1/write]
- **Pipeline Structure:** [DESCRIBE_PIPELINES, e.g., a traces pipeline with OTLP input, a metrics pipeline with Prometheus input, and a logs pipeline with filelog input]
- **Governance Constraints:** [LIST_CONSTRAINTS, e.g., must redact all PII from log bodies, must not export to any non-approved endpoint, must enforce a 10-second timeout on all exports]

# OUTPUT SCHEMA
Your response must be a JSON object with the following structure:
{
  "collector_config_yaml": "The complete, valid YAML configuration as a single string.",
  "data_flow_summary": "A paragraph explaining how data moves through the configured pipelines.",
  "validation_rationale": "A list of checks you performed against the governance constraints and best practices.",
  "governance_checklist": "A list of boolean pass/fail items confirming each constraint from the input is met."
}

# CONSTRAINTS
- The YAML must be syntactically valid and follow the official OpenTelemetry Collector configuration schema.
- Do not use any deprecated components.
- If a governance constraint cannot be met with the specified components, explicitly state the failure in the `governance_checklist` and suggest an alternative.
- Assume a standard `contrib` distribution of the collector.

To adapt this template, replace each square-bracket placeholder with your concrete requirements. For example, instead of [LIST_RECEIVERS], provide a specific list like otlp receiver on port 4317, hostmetrics receiver for CPU and memory. The model's performance is directly tied to the specificity of these inputs. After generating the configuration, always run it through a YAML validator and test it in a non-production environment. The governance_checklist in the output is your first line of defense against misconfigurations that could lead to data leaks or pipeline failures. For high-risk environments, treat this generated configuration as a first draft that requires human review and a security sign-off before deployment.

This prompt is designed to be called programmatically as part of a GitOps-managed observability stack. The typical integration point is a pull request workflow: a platform engineer proposes a new collector configuration or a change to an existing one, and the prompt generates a validated, vendor-agnostic configuration that is then committed to the observability-as-code repository. The prompt expects structured inputs—service topology, signal requirements, data governance constraints—and produces a YAML collector configuration. Because collector misconfiguration can cause silent telemetry loss or cost overruns, the harness must enforce validation before the output reaches production.

Wire the prompt into a CI job that runs on PR creation or update. The job should: (1) assemble the input context from repository files (service manifests, existing collector configs, data classification tags) and PR description; (2) call the prompt with the assembled [SERVICE_TOPOLOGY], [SIGNAL_REQUIREMENTS], [DATA_GOVERNANCE_CONSTRAINTS], and [EXISTING_COLLECTOR_CONFIG] placeholders; (3) parse the YAML output and validate it structurally using otelcol validate or a schema check against the OpenTelemetry Collector configuration schema; (4) run a semantic validation pass that checks for common failure modes—missing pipelines, undefined exporters, processors applied to wrong signal types, and PII leaking into attributes; (5) if validation fails, retry once with the validation errors injected into the [CONSTRAINTS] placeholder as explicit prohibitions; (6) post the generated config and validation report as a PR comment for human review. Use a model with strong YAML generation and instruction-following, such as claude-3-5-sonnet or gpt-4o. Set temperature=0 for deterministic output. Log the full prompt, response, validation results, and reviewer decision for auditability.

Do not auto-merge collector configuration changes without human approval. The review gate should require a platform engineer to confirm that the generated pipelines match the intended signal flow, that sampling and redaction rules are correct, and that the configuration does not introduce cardinality explosions or cost regressions. For high-risk environments, add a canary deployment step: apply the generated config to a staging collector, run synthetic traffic through it, and verify that expected metrics, traces, and logs appear in the observability backend before promoting to production. Avoid wiring this prompt into a fully automated pipeline that pushes configs directly to running collectors without review—the blast radius of a bad collector config includes data loss, runaway costs, and compliance violations.