Observability Signal Quality Rubric Prompt

This prompt is designed for platform teams and SREs who need a structured, repeatable method to audit whether their observability signals—metrics, traces, and logs—collectively support effective incident diagnosis. The primary job-to-be-done is to move beyond ad-hoc, manual reviews of telemetry data and produce a multi-dimensional quality score that is defensible and comparable over time. The ideal user is an engineer responsible for service reliability who can provide a representative sample of telemetry data from a specific service or request path. Required context includes the raw telemetry payloads and the expected topology or service map, so the model can identify gaps in correlation and attribute completeness.

Use this prompt when you need a pre-production readiness review for a new or refactored service, a post-incident gap analysis to determine why diagnosis was slow, or an ongoing observability health check as part of a regular operational audit. It is particularly effective when run against a controlled failure injection test, where the expected signals are known and the model can compare actual outputs against a defined standard. The output includes a red-flag checklist for common instrumentation gaps, such as missing correlation IDs, insufficient cardinality on key metrics, or broken parent-child relationships in traces. This makes the prompt suitable for generating actionable work items for engineering teams, not just a vague quality score.

Do not use this prompt when you need real-time alerting or automated gating in a CI/CD pipeline without human review. The rubric provides a qualitative assessment that requires engineering judgment to interpret and prioritize, especially when scoring trade-offs between cardinality and cost. It is also not a replacement for a full observability vendor assessment or a performance profiling tool; it focuses on the semantic completeness and correlation of signals, not on storage efficiency or query performance. For high-risk production changes, always have a human SRE review the rubric output and the raw data before accepting the score as a release gate.

This prompt template provides a structured framework for platform teams to assess whether their observability signals—metrics, traces, and logs—collectively support effective incident diagnosis. It produces a multi-dimensional quality score covering cardinality, sampling adequacy, attribute completeness, and cross-signal correlation. The template uses square-bracket placeholders that you replace with your specific service context, instrumentation standards, and evaluation criteria before sending to the model. This prompt is designed to be used as part of a regular observability health audit or as a gate in your CI/CD pipeline for new service deployments.

textCopy
You are an observability signal quality auditor. Your task is to evaluate the observability signals for [SERVICE_NAME] against a defined rubric and produce a structured quality assessment.

## INPUT

### Service Context
- Service Name: [SERVICE_NAME]
- Service Type: [SERVICE_TYPE, e.g., HTTP API, gRPC, Event-Driven, Batch]
- Criticality Tier: [TIER, e.g., Tier-1 Customer-Facing, Tier-2 Internal, Tier-3 Background]
- Expected Request Path: [REQUEST_PATH_DESCRIPTION]
- Upstream Dependencies: [UPSTREAM_SERVICES_LIST]
- Downstream Consumers: [DOWNSTREAM_SERVICES_LIST]

### Observability Data
#### Metrics Sample
[METRICS_SAMPLE, e.g., a snippet of Prometheus metrics output or Datadog metric summaries]

#### Trace Sample
[TRACE_SAMPLE, e.g., a representative distributed trace in JSON or visual format]

#### Log Sample
[LOG_SAMPLE, e.g., a set of log lines from a failure scenario]

### Instrumentation Standards
[INSTRUMENTATION_STANDARDS, e.g., your team's OpenTelemetry attribute conventions, required log fields, alert thresholds]

## EVALUATION RUBRIC

Score each dimension from 1 (critical gap) to 5 (excellent). Provide specific evidence and a recommendation for each score below 4.

### 1. Cardinality Management (1-5)
- Are metric label cardinalities bounded and documented?
- Are high-cardinality attributes (e.g., user IDs, session IDs) excluded from metric labels and placed in trace attributes or log context instead?
- Is there evidence of cardinality explosion risks in the provided metrics sample?

### 2. Sampling Adequacy (1-5)
- Is the trace sampling strategy documented and appropriate for the service's criticality?
- Are error traces and high-latency traces retained regardless of sampling rate?
- Is the sampling decision recorded as a span attribute for transparency?

### 3. Attribute Completeness (1-5)
- Do spans include required resource attributes per [INSTRUMENTATION_STANDARDS]?
- Are correlation IDs, request IDs, or trace IDs present in all log lines?
- Do logs include service name, environment, and version identifiers?

### 4. Cross-Signal Correlation (1-5)
- Can you navigate from a metric anomaly to relevant traces and logs without manual identifier reconstruction?
- Are exemplars attached to metrics where appropriate?
- Do error responses include a trace ID that appears in logs?

### 5. Failure Mode Coverage (1-5)
- Do the provided log samples capture upstream dependency failures with sufficient context?
- Are timeout, circuit-breaker, and rate-limit events logged with diagnostic detail?
- Do health check endpoints return actionable error bodies?

### 6. Semantic Convention Compliance (1-5)
- Do span names, attribute keys, and metric names follow [INSTRUMENTATION_STANDARDS] or OpenTelemetry semantic conventions?
- Are HTTP/gRPC/DB client calls instrumented with standard attributes?

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "service_name": "string",
  "evaluation_timestamp": "ISO 8601",
  "overall_score": number (average of dimension scores, rounded to 1 decimal),
  "dimensions": [
    {
      "name": "string",
      "score": number (1-5),
      "evidence": ["string"],
      "gaps": ["string"],
      "recommendation": "string"
    }
  ],
  "red_flags": [
    {
      "severity": "critical" | "high" | "medium",
      "finding": "string",
      "impact": "string",
      "remediation": "string"
    }
  ],
  "improvement_priority": [
    {
      "rank": number,
      "action": "string",
      "effort_estimate": "low" | "medium" | "high",
      "expected_impact": "string"
    }
  ]
}

## CONSTRAINTS
- Base all scores on the provided evidence. Do not assume instrumentation exists unless visible in the samples.
- If a dimension cannot be evaluated due to missing data, score it 1 and note "insufficient data" as the primary gap.
- Flag any PII or secrets visible in the log or trace samples as a critical red flag.
- Prioritize recommendations by impact on incident diagnosis speed.

To adapt this template for your environment, replace each square-bracket placeholder with concrete data from your service. The [INSTRUMENTATION_STANDARDS] placeholder should reference your team's actual conventions—if you don't have documented standards, use the OpenTelemetry semantic conventions as a baseline and note the gap. For the observability data samples, provide representative data from a recent failure scenario rather than happy-path traffic, as failure mode coverage is a key dimension. If you're evaluating multiple services, run this prompt once per service rather than batching, as the context window can become saturated and scores may bleed across services. The output JSON is designed to be ingested by a dashboard or tracking system—store results over time to measure observability improvement trends.

This prompt is designed to be called programmatically as part of a scheduled or event-driven observability audit. The typical integration point is a CI/CD pipeline, a periodic SRE audit job, or a post-incident review workflow. The application layer is responsible for gathering the required inputs—telemetry samples, service topology metadata, and team instrumentation standards—and assembling them into the [CONTEXT] block before invoking the model. The prompt itself should be treated as a stateless function: it receives a complete evidence packet and returns a structured quality assessment. Do not rely on the model to fetch live telemetry; all data must be pre-collected and provided in the prompt.

Validation and Output Contract: The prompt instructs the model to return a JSON object matching a specific schema. Your harness must validate this output before accepting it. At minimum, confirm that: (1) the response is valid JSON, (2) all required top-level keys (overall_score, dimension_scores, red_flags, recommendations) are present, (3) overall_score is a number between 0 and 100, (4) each entry in red_flags contains a severity field with a valid enum value (critical, high, medium, low), and (5) dimension_scores includes all expected dimensions (cardinality, sampling, attribute_completeness, correlation). If validation fails, retry with the same prompt once. If it fails again, log the raw output and escalate for human review rather than silently accepting a malformed assessment.

Model Choice and Tool Use: This is a structured reasoning task that benefits from models with strong instruction-following and JSON output capabilities. Use a model that supports strict JSON mode or structured output constraints (e.g., GPT-4o with response_format, Claude 3.5 Sonnet with tool use for the output schema). The prompt does not require external tool calls, RAG retrieval, or function calling during execution—all evidence is provided inline. However, you may choose to wrap this prompt in a tool definition where the output schema is enforced by the platform rather than by prompt instructions alone. This reduces format drift risk.

Logging and Audit Trail: Every invocation should log: the prompt version hash, the model and temperature used, the input context sources (with timestamps), the raw model response, the validation result, and the final accepted or rejected output. This audit trail is critical because the rubric's recommendations may drive instrumentation changes or incident review actions. Store these records in a queryable format so that SRE teams can trace any quality score back to the evidence that produced it.

Human Review Gate: For any audit where the overall_score falls below a defined threshold (e.g., 60) or where critical severity red flags are present, route the output to a human reviewer before publishing results or opening automated tickets. The prompt's red-flag checklist is designed to catch systemic gaps, but remediation decisions—such as increasing sampling rates or adding span attributes—often require human judgment about cost, performance impact, and team priorities. The harness should support a review queue where an SRE can approve, amend, or reject the assessment before it drives downstream actions.