Inferensys

Prompt

SRE Golden Signals Dashboard Prompt Template

A practical prompt playbook for using the SRE Golden Signals Dashboard Prompt Template in production AI workflows. Generate a complete dashboard specification with panels for latency, traffic, errors, and saturation, validated against service-specific SLIs.
Wide-angle shot of a modern WeWork open floor plan with creative walls covered in AI system architecture diagrams, product team collaborating in standing desk area with industrial lighting.
PROMPT PLAYBOOK

When to Use This Prompt

Defines the job-to-be-done, ideal user, required context, and constraints for the SRE Golden Signals Dashboard prompt.

This prompt is designed for Site Reliability Engineers (SREs) and platform engineers who need to translate a service description into a structured, implementation-ready dashboard specification. The core job is to take a high-level understanding of a service—its user-facing interactions, internal dependencies, and known failure modes—and produce a consistent, repeatable monitoring plan centered on the four golden signals: latency, traffic, errors, and saturation. Use it when onboarding a new service, standardizing observability across a fleet of microservices, or redesigning an existing dashboard that has become cluttered with vanity metrics and lacks a clear signal hierarchy.

The prompt assumes you have a solid grasp of the service's architecture but need a framework to force disciplined thinking about what to measure and why. It is not a replacement for that architectural understanding; if you cannot describe the service's critical paths, its dependencies, and what 'normal' looks like, the output will be generic and low-value. The prompt works best when you provide a concrete service description that includes user journeys, upstream and downstream dependencies, and the specific technologies involved (e.g., HTTP APIs, message queues, databases). Do not use this prompt to generate raw PromQL, SQL, or vendor-specific query syntax. Its output is a human-readable specification that an engineer then implements in their observability tool of choice, such as Grafana, Datadog, or Honeycomb.

Before running this prompt, gather the service's key architectural details: a list of user-facing endpoints or operations, the critical internal and external dependencies, any existing SLIs or SLOs, and a summary of the last one or two incidents if available. The prompt will guide you through defining panels for each golden signal, specifying the data sources, and structuring the dashboard layout for rapid diagnosis during an incident. After generating the specification, validate it against your on-call diagnostic workflows: can an on-call engineer locate the affected signal within 30 seconds? Does the dashboard surface saturation before it causes user-facing errors? If the answer is no, iterate on the specification before implementation.

PRACTICAL GUARDRAILS

Use Case Fit

Where the SRE Golden Signals Dashboard prompt works well and where it introduces risk. Use these cards to decide if this prompt fits your current operational context.

01

Good Fit: Standard Service Topology

Use when: You have a well-defined service with known RPC or HTTP endpoints, and you need a structured dashboard specification for latency, traffic, errors, and saturation. Guardrail: Provide the service name, protocol, and existing SLI definitions in [SERVICE_CONTEXT] to ground the output in real telemetry sources.

02

Bad Fit: Undefined or Missing SLIs

Avoid when: The service has no defined Service Level Indicators, and the team cannot articulate what 'good' looks like for latency or error rate. Guardrail: Run the SLI Selection Prompt first to produce candidate SLIs before generating a dashboard specification. Without SLIs, the dashboard panels will be generic and operationally useless.

03

Required Input: Telemetry Source Mapping

Risk: The prompt produces panel definitions that reference metrics or log sources that do not exist in your observability stack. Guardrail: Always include [TELEMETRY_SOURCES] mapping SLIs to actual Prometheus metric names, log query fields, or trace attributes. Validate the output against your metrics registry before implementation.

04

Operational Risk: Dashboard Without Runbooks

Risk: A dashboard that shows signals but provides no guidance on what to do when a panel turns red increases on-call cognitive load and mean time to resolve. Guardrail: Pair this prompt with an alerting architecture or runbook prompt. Require each panel to include a runbook_link or troubleshooting_hint field in the output schema.

05

Operational Risk: Alert Storms from Thresholds

Risk: The prompt may suggest static thresholds that trigger alert storms during normal traffic patterns or deployment windows. Guardrail: Add [THRESHOLD_CONSTRAINTS] specifying burn-rate-based alerting, hysteresis windows, or deployment suppression rules. Validate suggested thresholds against historical data before activating alerts.

06

Bad Fit: Batch or Async-Only Workloads

Avoid when: The service is a batch processor, cron job, or async worker with no synchronous request path. Golden signals assume request-driven traffic patterns. Guardrail: For batch workloads, use the Metrics Collection Architecture Design Prompt instead, and focus on queue depth, processing lag, and error rate over time windows rather than latency percentiles.

PROMPT PLAYBOOK

Copy-Ready Prompt Template

A reusable prompt for generating a complete SRE Golden Signals dashboard specification from service context and operational requirements.

This prompt template produces a structured dashboard specification covering latency, traffic, errors, and saturation panels. It is designed for SREs and platform engineers who need to move from service context to a concrete, implementable dashboard design. The template forces explicit consideration of SLIs, data sources, panel types, thresholds, and on-call diagnostic workflows. Before using this prompt, you should have a clear understanding of the target service's architecture, its critical user journeys, and the telemetry systems already in place.

text
You are an SRE architect designing a Golden Signals dashboard for a production service. Your output must be a complete, implementation-ready dashboard specification.

## Service Context
- Service Name: [SERVICE_NAME]
- Service Description: [SERVICE_DESCRIPTION]
- Critical User Journey: [CRITICAL_USER_JOURNEY]
- Architecture Type: [ARCHITECTURE_TYPE] (e.g., monolith, microservice, serverless, event-driven)
- Protocol: [PROTOCOL] (e.g., HTTP/gRPC/async messaging)
- Dependencies: [DEPENDENCIES]

## Telemetry Context
- Metrics System: [METRICS_SYSTEM] (e.g., Prometheus, Datadog, CloudWatch, Grafana)
- Logging System: [LOGGING_SYSTEM]
- Tracing System: [TRACING_SYSTEM]
- Available Metric Namespaces: [AVAILABLE_METRIC_NAMESPACES]
- Dashboard Tool: [DASHBOARD_TOOL] (e.g., Grafana, Datadog, New Relic)

## Constraints
- [CONSTRAINTS] (e.g., cardinality limits, retention windows, cost budgets, compliance boundaries)

## Output Schema
Produce a JSON object with the following structure:
{
  "dashboard_title": "string",
  "service_identifier": "string",
  "panels": [
    {
      "signal_type": "latency|traffic|errors|saturation",
      "panel_title": "string",
      "description": "string explaining what this panel shows and why it matters",
      "query": "string (the actual PromQL/Datadog/CloudWatch query)",
      "visualization": "line|heatmap|stat|gauge|table",
      "thresholds": {
        "warning": "number or expression",
        "critical": "number or expression"
      },
      "slo_target": "string (the SLO this panel supports, or null)",
      "diagnostic_links": ["list of related panels or drill-down targets"]
    }
  ],
  "layout": {
    "rows": [
      {
        "row_title": "string",
        "panel_ids": ["list of panel indices from the panels array"]
      }
    ]
  },
  "on_call_workflow": {
    "initial_triage": "string describing which panels to check first when an alert fires",
    "diagnostic_path": ["ordered list of diagnostic steps"],
    "escalation_triggers": ["conditions that warrant escalation beyond first responder"]
  },
  "validation_notes": ["list of assumptions, known gaps, or data quality concerns"]
}

## Instructions
1. For each golden signal (latency, traffic, errors, saturation), design at least one panel.
2. Latency panels must distinguish between average, tail (p95/p99), and max latency where appropriate.
3. Traffic panels must show request rate, connection count, or message throughput relevant to the service protocol.
4. Error panels must separate client errors (4xx) from server errors (5xx) and include error rate as a percentage of total traffic.
5. Saturation panels must identify the most constrained resource (CPU, memory, I/O, connection pool, queue depth) and show utilization relative to capacity.
6. Every query must be syntactically valid for the specified metrics system.
7. Thresholds must be derived from SLO targets where available, or from reasonable operational defaults.
8. The on-call workflow must be actionable: a first responder should know exactly which panels to check and in what order.
9. If any required signal cannot be instrumented with available telemetry, note this explicitly in validation_notes and suggest a workaround or instrumentation gap.
10. Do not invent metric names. Use only the provided metric namespaces. If a metric is missing, state the gap.

After pasting this template, replace every square-bracket placeholder with concrete values from your service and telemetry environment. The most critical placeholders are [AVAILABLE_METRIC_NAMESPACES] and [CONSTRAINTS] because they determine whether the generated queries will actually execute. If you are unsure about available metrics, run a discovery query against your metrics system first and paste the results into the context. For high-risk production services, always have a second SRE review the generated dashboard specification before deploying it, and validate that every query returns expected data in a staging or pre-production environment.

IMPLEMENTATION TABLE

Prompt Variables

Inputs the prompt needs to generate a reliable SRE Golden Signals dashboard specification. Provide these with as much detail as possible to avoid generic or unactionable outputs.

PlaceholderPurposeExampleValidation Notes

[SERVICE_NAME]

Identifies the target service for the dashboard specification.

api-gateway-prod

Must be a non-empty string matching a known service identifier in your service catalog. Reject if null or generic.

[SLI_DEFINITIONS]

Provides the specific Service Level Indicators, their measurement methods, and data sources for the service.

Latency: p95 < 200ms measured by Prometheus histogram 'http_request_duration_seconds' at the load balancer.

Must be a structured list of at least 4 SLIs covering latency, traffic, errors, and saturation. Validate each SLI has a metric, threshold, and data source.

[SLO_TARGETS]

Defines the Service Level Objectives and error budgets that the dashboard must help track.

99.9% availability over a 30-day rolling window.

Must include a numerical target and a measurement window. Validate format as 'X% over Y period'. Reject if missing.

[ON_CALL_WORKFLOWS]

Describes the diagnostic steps an on-call engineer follows during an incident, which the dashboard must support.

  1. Check service latency panel. 2. Correlate with upstream error rates. 3. Drill down to instance-level CPU saturation.

Must be a numbered list of at least 3 steps. Validate that each step references a signal type (latency, traffic, error, saturation).

[DASHBOARD_CONSUMER]

Specifies the primary persona using the dashboard to tailor information density and layout.

On-call SRE during a SEV2 incident.

Must be a non-empty string. Validate against a predefined list of personas (e.g., 'On-call SRE', 'Service Owner', 'Engineering Manager') to control output tone and detail.

[EXISTING_TOOL_CONSTRAINTS]

Lists the observability tools and data sources the dashboard must integrate with, including query language limitations.

Data sources: Prometheus, Grafana. Query language: PromQL. No Elasticsearch access.

Must be a non-empty string. Validate that the generated dashboard spec only uses panels and queries compatible with the listed tools. Flag any incompatible suggestions.

[CRITICAL_DEPENDENCIES]

Lists upstream and downstream services whose health directly impacts the target service's golden signals.

Upstream: auth-service. Downstream: user-profile-service, payment-gateway.

Must be a list of service names. Validate that the generated dashboard includes panels for these dependencies' error rates and latency where appropriate.

PROMPT PLAYBOOK

Implementation Harness Notes

How to wire the SRE Golden Signals Dashboard Prompt into a design review workflow with validation, retries, and a human approval gate.

This prompt is a design-side tool, not a runtime component. It should be integrated into a service onboarding template or a pull request checklist within your internal developer platform. When a new service entry is added to your service catalog, trigger this prompt automatically to generate an initial dashboard specification. The output is a JSON specification, so your harness must validate the JSON structure before presenting it to the user. If the JSON is malformed, use a retry loop with a simplified prompt that asks the model to fix the specific parsing error reported by your validator.

Store the generated specification in version control alongside the service's code. This ensures the dashboard design evolves with the service and can be reviewed in the same pull request workflow. For automation, use a CI/CD pipeline step that runs the prompt, validates the output against a JSON Schema that defines the required fields for each panel type (latency, traffic, errors, saturation), and posts the result as a comment on the service onboarding ticket. The human review step is critical: an SRE must approve the specification before any dashboard implementation work begins. This approval should be captured as a code review approval or a ticketing system sign-off.

For model choice, use a model with strong JSON generation capabilities and a context window large enough to hold your service catalog entry, any relevant SLO definitions, and the prompt template. If your service catalog contains sensitive data, run this prompt in a private deployment or ensure your API provider meets your data handling requirements. Do not bypass the human review step for production services. The model can propose a dashboard layout, but only an on-call SRE can validate that the proposed panels will actually help them diagnose an incident at 3 AM.

IMPLEMENTATION TABLE

Expected Output Contract

Defines the strict JSON schema for the SRE Golden Signals Dashboard specification. Use this contract to validate the model's output before passing it to a dashboard-as-code pipeline or a manual review step.

Field or ElementType or FormatRequiredValidation Rule

dashboard_title

string

Must match regex ^[a-zA-Z0-9_- ]{3,100}$. No empty strings.

service_name

string

Must be a non-empty string matching the [SERVICE_NAME] input exactly.

panels

array of objects

Array must contain exactly 4 objects, one for each golden signal: latency, traffic, errors, saturation.

panels[].signal_type

enum string

Must be one of: 'latency', 'traffic', 'errors', 'saturation'. No duplicates allowed across the array.

panels[].title

string

Must be a non-empty string, max 80 characters. Should include the service name and signal type.

panels[].queries

array of strings

Array must contain at least 1 and no more than 5 non-empty strings. Each string must be a valid PromQL or vendor-agnostic metric query.

panels[].visualization

enum string

Must be one of: 'line_chart', 'heatmap', 'stat', 'gauge', 'table'. Recommendation: 'heatmap' for latency, 'line_chart' for traffic and errors, 'stat' or 'gauge' for saturation.

panels[].thresholds

array of objects

If present, each object must have 'value' (number), 'color' (string hex or name), and 'operator' (enum: '>', '<', '>=', '<='). Validate operator logic against SLO targets.

PRACTICAL GUARDRAILS

Common Failure Modes

What breaks first when generating SRE golden signals dashboards and how to guard against it.

01

Hallucinated Metrics and Data Sources

What to watch: The model invents metric names, PromQL queries, or data sources that don't exist in your observability stack. It may confidently reference http_requests_total when your services expose api_requests_count. Guardrail: Always provide a [METRICS_CATALOG] or [TELEMETRY_SCHEMA] as input. Add a post-generation validation step that checks every metric name and label against your actual metrics registry before accepting the dashboard specification.

02

Missing Saturation Signals

What to watch: The prompt produces latency, traffic, and error panels but omits saturation—the most commonly forgotten golden signal. Dashboards without CPU, memory, connection pool, or queue depth panels leave on-call engineers blind to resource exhaustion. Guardrail: Add a hard constraint in [CONSTRAINTS] requiring at least one saturation panel per service. Use a post-generation checklist that rejects dashboard specs missing saturation indicators.

03

Dashboard Noise and Low Signal-to-Noise Ratio

What to watch: The model generates too many panels, including vanity metrics, raw counters without rates, or single-service views that don't support diagnosis. On-call engineers drown in information during incidents. Guardrail: Constrain output to a maximum panel count (e.g., 12-16 panels). Require each panel to include a diagnostic_justification field explaining which on-call question it answers. Review for panels that don't map to a known failure mode.

04

Incorrect Threshold and SLO Alignment

What to watch: The model sets alert thresholds or dashboard warning lines that contradict your actual SLOs. A panel might show a 99.9% latency target when your error budget allows 99.5%, creating confusion during incidents. Guardrail: Provide [SLO_SPEC] as required input. Add a validation step that compares every threshold in the generated dashboard against the provided SLO document and flags mismatches for human review before deployment.

05

Service Boundary Confusion

What to watch: The model mixes metrics from different services into the same panels or creates dashboard sections that don't align with your actual service topology. An on-call engineer looking at the checkout service dashboard sees payment-gateway metrics without context. Guardrail: Provide a [SERVICE_DEPENDENCY_MAP] as input. Require the output to group panels by service boundary and explicitly label cross-service metrics with the source service name.

06

Static Dashboard Without Diagnostic Drill-Down

What to watch: The generated dashboard is a flat collection of charts with no guidance on how to use them during an incident. On-call engineers see a spike but don't know which panel to check next or what runbook to open. Guardrail: Require the output to include a diagnostic_workflow section that maps common alert patterns to a sequence of panels to inspect. Validate that each alert referenced in the workflow has a corresponding panel in the dashboard.

IMPLEMENTATION TABLE

Evaluation Rubric

Use this rubric to score the generated SRE Golden Signals dashboard specification on a 1-5 scale before shipping to production. Each criterion targets a specific failure mode common in AI-generated observability designs.

CriterionPass StandardFailure SignalTest Method

Signal Coverage

Spec includes panels for all four golden signals: latency, traffic, errors, and saturation.

Missing one or more golden signals; panels describe metrics without mapping to a specific signal.

Manual review: count distinct signal sections in the output.

SLI Mapping

Each panel references at least one concrete SLI from [SERVICE_SLIS] with the correct metric name and source.

Panels use generic metric names like 'cpu' or 'latency' without mapping to a specific SLI from the input.

Schema check: verify each panel object contains a non-null 'sli_reference' field matching an entry in [SERVICE_SLIS].

Diagnostic Workflow

Dashboard layout follows a top-down diagnostic flow: high-level SLO status first, then signal deep-dives, then drill-down links.

Panels are arranged alphabetically or by data source; no logical progression from alert to root cause.

Manual review: trace the panel order against the on-call diagnostic steps in [ONCALL_WORKFLOW].

Threshold Context

Visualizations include SLO threshold lines, error budget remaining, or burn rate indicators where applicable.

Charts show raw time-series data without any reference to acceptable operating ranges or SLO targets.

Schema check: verify at least one panel includes a 'thresholds' array with 'value' and 'label' fields.

Actionable Panel Titles

Every panel title describes what to investigate (e.g., 'P99 Latency by Endpoint vs SLO') rather than just the metric name.

Panel titles are metric names like 'cpu_usage' or 'http_requests_total' with no investigative framing.

Manual review: check that each title contains a verb, comparison, or question implied by the signal.

Saturation Specificity

Saturation panels target a constrained resource (e.g., connection pool, thread pool, disk IOPS) relevant to the service type.

Saturation panel defaults to generic CPU/memory without considering the service's bottleneck resource from [SERVICE_TYPE].

Manual review: verify the saturation panel metric matches known bottleneck patterns for [SERVICE_TYPE].

Granularity and Aggregation

Spec defines appropriate roll-up windows and aggregation functions for each signal (e.g., P95/P99 for latency, rate for errors).

All panels use the same aggregation (e.g., avg) or no aggregation logic is specified.

Schema check: verify each panel object contains a non-null 'aggregation' and 'window' field.

Runbook Linkage

Spec includes a section mapping each signal panel to a specific runbook entry or alert rule from [ALERT_RULES].

Dashboard spec has no references to runbooks, alert names, or incident response procedures.

Schema check: verify the output contains a non-empty 'runbook_mappings' array with 'panel_id' and 'runbook_ref' fields.

ADAPTATION OPTIONS

Adapt This Prompt

How to adapt

Use the base prompt with a single service and minimal SLI validation. Replace [SERVICE_NAME] with a known service, limit [DATA_SOURCES] to one metrics backend, and set [CONSTRAINTS] to '4 panels maximum, no alert links required.' Run against a frontier model with no schema enforcement.

Watch for

  • Panels that list metrics without actionable thresholds
  • Missing saturation signals (the hardest to define)
  • Dashboard specs that look correct but lack on-call diagnostic flow
Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.