Race Condition Root Cause Analysis Prompt

Use this prompt when you have a confirmed or strongly suspected race condition that caused a production incident, and you need to reconstruct the specific interleaving of concurrent operations that led to the failure. This playbook is designed for backend engineers, SREs, and on-call responders who have access to concrete concurrency evidence from the incident window: concurrent operation logs with timestamps, transaction traces showing interleaved execution, lock acquisition and release records, and data state snapshots before, during, and after the race window. The prompt turns chaotic concurrency evidence into a precise race window description, a root cause hypothesis grounded in the actual execution order, and code-level fix guidance that addresses the specific interleaving pattern rather than generic thread-safety advice.

The prompt requires structured input to produce reliable output. You must provide: a description of the concurrent operations involved (e.g., two goroutines, parallel database transactions, async event handlers), the observed failure symptom (e.g., duplicate record, lost update, stale read, inconsistent state), and the raw evidence from the incident window. This evidence should include timestamped operation logs showing entry and exit points, lock acquisition and release records with holder identity, transaction begin and commit timestamps, and data state snapshots that reveal the corrupted or inconsistent state. Without this concrete concurrency data, the model will generate plausible but ungrounded hypotheses that waste investigation time. The output requires human review before any production code change because race condition fixes can introduce new concurrency bugs, deadlocks, or performance regressions if applied without careful analysis of all affected code paths.

Do not use this prompt when you lack concrete concurrency data from the incident window. If you only have error messages, stack traces, or user reports without the underlying operation interleaving evidence, start with the Incident Timeline Extraction Prompt or Log Anomaly Pattern Extraction Prompt to gather the necessary data first. Do not use this prompt when the failure is clearly single-threaded, when the incident is caused by a simple null pointer or configuration error, or when you are doing pre-merge code review without a production incident to ground the analysis. For pre-merge concurrency review, use the Concurrency and Race Condition Review Prompts instead. This prompt is specifically designed for post-incident root cause analysis where the race condition has already manifested in production and you need to understand exactly how it happened to prevent recurrence.

The following prompt template is designed to be pasted directly into your AI harness. It accepts structured evidence from your incident investigation—including concurrent operation logs, lock acquisition patterns, and data state transitions—and outputs a specific interleaving hypothesis with code-level fix guidance. Replace every square-bracket placeholder with real data from your incident before execution. Do not leave any placeholder unresolved; if a piece of evidence is unavailable, explicitly note its absence in the input rather than omitting the field.

codeCopy
SYSTEM: You are a senior backend engineer specializing in distributed systems and concurrency debugging. Your task is to analyze incident evidence and produce a structured root cause analysis for a race condition. You must identify the specific interleaving of concurrent operations that caused the failure, explain why the existing synchronization was insufficient, and propose code-level remediation. Do not speculate beyond the provided evidence. If the evidence is insufficient to isolate a single root cause, enumerate the plausible hypotheses ranked by likelihood with the additional evidence needed to confirm each.

USER:
[INCIDENT_SUMMARY]

[CONCURRENT_OPERATION_LOGS]

[LOCK_ACQUISITION_PATTERNS]

[DATA_STATE_TRANSITIONS]

[TRANSACTION_BOUNDARIES]

[SYSTEM_ARCHITECTURE_CONTEXT]

[EXPECTED_VS_ACTUAL_OUTCOME]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

[RISK_LEVEL]

Placeholder Definitions:

• [INCIDENT_SUMMARY]: A concise description of the observed failure, including the affected service, the symptoms (e.g., incorrect data, duplicate writes, stale reads), and the time window of the incident. Example: "Payment service recorded duplicate charges for 3% of transactions between 14:22 and 14:31 UTC. Each affected transaction shows two successful charge records with identical idempotency keys."

• [CONCURRENT_OPERATION_LOGS]: Timestamped log entries for all operations that executed concurrently during the incident window. Include thread or process IDs, operation types (read, write, update, delete), and the specific data entities accessed. Format as a chronologically ordered list or table.

• [LOCK_ACQUISITION_PATTERNS]: Records of lock acquisition attempts, including lock type (pessimistic, optimistic, distributed), lock scope (row, table, key, resource), acquisition timestamps, hold durations, and any timeout or contention events. Include both successful and failed acquisition attempts.

• [DATA_STATE_TRANSITIONS]: The sequence of state changes for the affected data entities, including the value before and after each operation, the operation that caused the transition, and the timestamp. If using event sourcing or change data capture, include the relevant event records.

• [TRANSACTION_BOUNDARIES]: The start and commit/rollback timestamps for each database transaction involved, along with the isolation level used (e.g., READ COMMITTED, REPEATABLE READ, SERIALIZABLE). Note any transactions that spanned multiple service calls or external API invocations.

• [SYSTEM_ARCHITECTURE_CONTEXT]: A brief description of the relevant system components, including the number of service instances, the database technology and version, any caching layers, message queues, or external services involved in the concurrent operations.

• [EXPECTED_VS_ACTUAL_OUTCOME]: A clear statement of what the system should have produced versus what it actually produced. Example: "Expected: exactly one charge record per idempotency key. Actual: two charge records with the same idempotency key, both marked as successful."

• [OUTPUT_SCHEMA]: The desired output structure. Recommended schema:

  jsonCopy
  {
    "root_cause_hypothesis": {
      "interleaving_description": "string",
      "race_window": "string",
      "vulnerable_code_path": "string",
      "synchronization_gap": "string",
      "confidence": "high|medium|low"
    },
    "contributing_factors": ["string"],
    "code_level_fix": {
      "approach": "string",
      "specific_change": "string",
      "risk_of_regression": "string"
    },
    "alternative_hypotheses": [],
    "evidence_gaps": ["string"]
  }

• [CONSTRAINTS]: Any specific requirements for the analysis. Example: "Do not propose database-level serializable isolation as a fix unless no application-level synchronization is feasible. Prefer solutions that minimize contention and preserve throughput. Flag any fix that requires a data migration."

• [RISK_LEVEL]: The severity classification of the incident. Use "critical" for data corruption or financial impact, "high" for user-facing incorrect behavior without data loss, "medium" for performance degradation, or "low" for cosmetic issues. This field influences the depth of analysis and the urgency of the remediation recommendation.

Adaptation Guidance: This template assumes you have structured log data available. If your incident evidence is primarily in unstructured form (chat transcripts, manual observations), run the Incident Timeline Extraction Prompt first to produce normalized event logs, then feed that output into this template. For high-severity incidents where the race condition caused data corruption, always include a human review step before applying any code-level fix. The model's proposed interleaving is a hypothesis—validate it against database audit logs or by reproducing the race window in a staging environment before merging remediation code.

This prompt is designed to be a component within a larger incident analysis pipeline, not a standalone chat interaction. It should be triggered automatically when an incident is declared with a cause:race_condition tag or when a postmortem draft identifies concurrency as a suspected factor. The harness must gather the required inputs—concurrent operation logs, transaction ordering data, lock acquisition records, and data state transitions—from your observability stack (e.g., Datadog, Grafana, Jaeger) and format them into the [INPUT_CONTEXT] block before calling the model. The prompt's output is a structured JSON object containing the specific interleaving hypothesis and code-level fix guidance, which your system should parse and inject directly into the incident document or postmortem draft.

Validation and Retries: The harness must validate the model's output against the [OUTPUT_SCHEMA] defined in the prompt template. A strict JSON schema validator should check for the presence of required fields: race_window, interleaving_sequence, affected_resources, and fix_guidance. If validation fails, implement a single retry with the error message appended to the prompt as [PREVIOUS_OUTPUT_ERROR]. If the retry also fails, the harness should log the failure, flag the incident for manual root cause analysis by an on-call engineer, and attach the raw model output for debugging. Do not silently accept malformed output in a high-risk incident context.

Model Choice and Tool Use: For this analytical task, prefer a model with strong reasoning capabilities and a large context window (e.g., Claude 3.5 Sonnet, GPT-4o) to handle verbose log data. The prompt does not require external tools or RAG during execution because all evidence should be pre-fetched and included in the prompt context. However, the harness should provide a [TOOLS] block with a read_incident_timeline function if the model needs to request additional data, though this is optional for most implementations. Human Review Gate: The harness must route the final, validated output to a human reviewer (the incident commander or postmortem owner) for approval before the analysis is published in the official postmortem. The review UI should display the model's interleaving hypothesis alongside the raw evidence it cited, allowing the engineer to confirm or reject the analysis. This is critical because an incorrect root cause can misdirect remediation efforts and erode trust in the incident review process.