Load Balancer 502/504 Error Investigation Prompt

This prompt is designed for Site Reliability Engineers (SREs) and on-call developers who are actively troubleshooting HTTP 502 Bad Gateway or 504 Gateway Timeout errors. The core job is to correlate load balancer access logs with backend service logs to pinpoint the root cause: an unhealthy upstream instance, a connection timeout, a connection reset, or a backend pool misconfiguration. Use this prompt when you have access to both log sources and need a structured, evidence-backed diagnosis that identifies the specific failing backend pool and instance. This is not a general-purpose error explainer; it requires concrete log data to function reliably.

The prompt expects two primary inputs: a sample of load balancer access log lines showing the 502/504 errors and the corresponding backend service logs from the same time window. The access logs must include fields like upstream_addr, upstream_status, upstream_connect_time, and upstream_response_time. The backend logs should capture connection events, request processing, and any error or timeout messages. Without both log sources, the prompt cannot perform correlation and will produce low-confidence or speculative output. The prompt is most effective when the error is reproducible or ongoing, allowing you to capture fresh, time-aligned log samples rather than relying on stale historical data.

Do not use this prompt for initial alert triage or for diagnosing client-side errors (4xx). It is not a replacement for your monitoring dashboard or a substitute for checking basic infrastructure state—verify that backend instances are running and network ACLs are open before invoking the prompt. The prompt also assumes you have already narrowed the problem to a specific load balancer and backend pool; it is not designed to scan multiple unrelated services simultaneously. If you lack access to backend logs or your load balancer logs do not include upstream timing fields, the prompt will be unable to produce a grounded diagnosis and you should resolve that observability gap first.

After using the prompt, validate its output by checking the identified backend instance directly: review its health check status, resource utilization at the time of the error, and any application-level exceptions. The prompt's diagnosis is a hypothesis, not a final verdict. Use it to focus your investigation, then confirm with direct instance inspection before applying a fix such as restarting an instance, adjusting timeout settings, or removing an unhealthy node from the pool.

This prompt template is designed to be pasted directly into your AI orchestration harness. It instructs the model to act as an SRE investigating a 502/504 error burst by correlating two distinct log sources: the load balancer's access logs and the backend service's application logs. The goal is to move beyond the generic gateway error and pinpoint whether the failure is due to an unhealthy upstream target, a connection timeout, a connection reset, or a specific failing backend pool member. The template uses square-bracket placeholders for all dynamic inputs, allowing you to swap in your own log data, time windows, and infrastructure context without modifying the core reasoning instructions.

textCopy
You are an SRE investigating a spike in 502/504 errors from a load balancer. Your task is to correlate the provided load balancer access logs with backend service logs to determine the root cause of the failures.

## INPUT DATA

### Load Balancer Access Logs (filtered for 502/504 errors within the incident window)
```json
[LOAD_BALANCER_ACCESS_LOGS]

Backend Service Logs (from all upstream pool members for the same time window)

jsonCopy
[BACKEND_SERVICE_LOGS]

CONTEXT

• Incident Time Window: [INCIDENT_TIME_WINDOW]
• Load Balancer Type: [LB_TYPE, e.g., AWS ALB, Nginx, HAProxy]
• Backend Pool Configuration: [BACKEND_POOL_CONFIG, e.g., list of healthy/unhealthy targets, timeout settings]
• Recent Changes: [RECENT_CHANGES, e.g., deployments, config updates, scaling events]

INVESTIGATION STEPS

1. Parse and Align Timestamps: Normalize all log timestamps to a single timezone (UTC). Identify the precise start and peak of the error burst from the load balancer logs.
2. Identify Failing Targets: From the load balancer logs, extract the specific backend target IPs or instance IDs that returned 502/504 errors. Calculate the error rate per target.
3. Correlate with Backend Logs: For the failing targets identified in step 2, examine their corresponding backend service logs during the same micro-window. Look for:
   • Unhealthy Targets: Evidence of failed health checks, crash loops, or OOMKills immediately before or during the error burst.
   • Timeouts: Requests in the backend logs that took longer than the load balancer's configured timeout to complete.
   • Connection Resets: Log entries indicating the backend process closed the connection prematurely (e.g., connection reset by peer, broken pipe) or that the load balancer sent an RST packet.
   • No Matching Logs: If a target sent a 502/504 but has no corresponding request log, this points to a network-level failure (e.g., a full connection queue, a firewall drop) or the backend process being completely unresponsive.
4. Identify the Dominant Failure Mode: Aggregate the findings from step 3 across all failing targets. Determine the single most common failure mode (e.g., 80% of errors were due to timeouts on pool member backend-03).

OUTPUT SCHEMA

You must respond with a valid JSON object that strictly follows this schema:

jsonCopy
{
  "incident_summary": "A one-sentence executive summary of the root cause.",
  "root_cause_hypothesis": {
    "primary_failure_mode": "UNHEALTHY_UPSTREAM | UPSTREAM_TIMEOUT | CONNECTION_RESET | NETWORK_FAILURE | UNKNOWN",
    "description": "A detailed explanation of the failure mechanism, citing specific log lines as evidence.",
    "failing_backend_pool_members": ["list of specific IPs or instance IDs that caused the errors"],
    "correlation_evidence": [
      {
        "lb_log_entry": "A summarized or key-value excerpt from the load balancer log.",
        "backend_log_entry": "The correlated log entry from the backend service, or 'NO_MATCHING_LOG' if none was found.",
        "failure_type": "TIMEOUT | RESET | REFUSED | NO_RESPONSE"
      }
    ]
  },
  "confidence": "HIGH | MEDIUM | LOW",
  "recommended_immediate_action": "A clear, actionable step to mitigate the issue (e.g., 'Remove backend-03 from the pool and restart its service.').",
  "requires_human_review": true or false
}

CONSTRAINTS

• Do not invent log entries. If the evidence is insufficient to determine a root cause, set primary_failure_mode to UNKNOWN and confidence to LOW, and explain what data is missing.
• Base all conclusions strictly on the provided log data and context.
• If the primary_failure_mode is UPSTREAM_TIMEOUT, you must state the configured load balancer timeout from the context and the actual backend processing time from the logs.
• The recommended_immediate_action must be a safe, reversible operational action, not a permanent code fix.

To adapt this template for your environment, replace the bracketed placeholders with your actual data. The [LOAD_BALANCER_ACCESS_LOGS] and [BACKEND_SERVICE_LOGS] fields should be populated with pre-filtered JSON arrays of log objects, ensuring you've already narrowed the data to the relevant time window to manage context length. The [BACKEND_POOL_CONFIG] placeholder is critical; providing the load balancer's timeout values and a list of pool members turns a generic diagnosis into a precise one. For high-severity production incidents, always set requires_human_review to true in your harness logic, and route the model's output to an on-call engineer for approval before any automated remediation is triggered.

This prompt is designed to be integrated into an automated incident response pipeline or a diagnostic chatbot for on-call SREs. The core challenge is that the prompt requires multi-source log correlation: load balancer access logs and backend service logs must be fetched, aligned by timestamp and request ID, and injected into the [LOAD_BALANCER_LOGS] and [BACKEND_SERVICE_LOGS] placeholders. Do not paste raw, unprocessed log streams. The harness must pre-process these inputs to extract a relevant time window (e.g., 5 minutes before and after the error spike) and normalize timestamps to UTC for accurate correlation.

For a production-grade implementation, wrap the model call in a validation and retry layer. The expected output is a structured JSON object with fields like root_cause, failing_backend_pool, evidence, and confidence. After the model responds, validate the JSON schema strictly. If the failing_backend_pool field references a pool name not present in the input logs, or if the evidence array is empty, reject the output and retry with a more constrained prompt that asks the model to abstain rather than guess. Log every attempt, including the raw model response and the validation error, to an audit trail for post-incident review. For high-severity incidents, route the validated output to a human SRE for approval before automatically updating the status page or paging the backend service owner.

Model choice matters here. Use a model with a large context window (128k+ tokens) to accommodate verbose log entries, and prefer models with strong JSON mode or structured output guarantees. If you are using a model without native JSON mode, append a strict [OUTPUT_SCHEMA] with a JSON Schema definition and instruct the model to respond with only that JSON object. For teams running this prompt in a RAG architecture, do not embed the logs in a vector database; instead, use a time-series log store and fetch the raw logs directly. The prompt's value comes from precise cross-source correlation, not semantic similarity search. Finally, implement a circuit breaker: if the model fails to produce valid JSON after three retries, escalate the raw logs to the on-call engineer with a note that automated diagnosis was inconclusive.