Inferensys

Prompt

Deadlock Detection and Resolution Prompt for Agent Pipelines

A practical prompt playbook for AI reliability engineers debugging stalled multi-agent workflows. Analyzes a wait-for graph of agent dependencies and produces a victim-selection and rollback plan with minimal disruption.
Developer demonstrating multi-agent tool use, agent tool selection interface on laptop, casual tech demo moment.
PROMPT PLAYBOOK

When to Use This Prompt

Defines the exact conditions, required inputs, and operational boundaries for using the deadlock detection and resolution prompt in production agent pipelines.

This prompt is designed for AI reliability engineers and platform teams who operate multi-agent pipelines where agents can deadlock waiting on each other's outputs or resource locks. Use it when your agent orchestrator detects a cycle in the dependency graph and needs a structured recovery plan. The prompt expects a pre-computed wait-for graph as input and returns a specific victim agent to abort or roll back, along with a justification and a step-by-step resolution sequence.

This is not a prompt for live-lock detection, performance degradation, or single-agent timeouts. It assumes the deadlock has already been detected by your infrastructure and the model's job is to decide the least disruptive way to break it. Do not use this prompt when you need real-time cycle detection—that belongs in your orchestrator code, not in the model. Do not use it when agents are merely slow or retrying; deadlock means a provable cycle in the dependency graph where no agent can make progress without external intervention. The prompt also assumes you have already ruled out transient network issues, queue backpressure, and resource exhaustion as root causes.

Before invoking this prompt, ensure your infrastructure has computed a complete wait-for graph with agent IDs, requested resources, current holders, and timestamps. The prompt will fail silently or produce dangerous output if given an incomplete graph. If your system cannot construct this graph reliably, invest in observability instrumentation first. For high-risk pipelines where an incorrect victim selection could cause data loss or financial impact, always route the model's output through a human approval step before executing the abort or rollback. The prompt includes a [RISK_LEVEL] parameter to adjust the conservatism of the resolution strategy.

PRACTICAL GUARDRAILS

Use Case Fit

Where the Deadlock Detection and Resolution Prompt delivers value and where it introduces risk. Use these cards to decide if this prompt fits your pipeline before wiring it into a production harness.

01

Good Fit: Multi-Agent Pipelines with Explicit Dependencies

Use when: your system has a formal or observable wait-for graph (agents waiting on tool outputs, locks, or sub-agent completion). The prompt excels at cycle detection and victim selection when dependencies are explicit. Guardrail: ensure the dependency graph is passed as structured input, not inferred from natural language logs.

02

Bad Fit: Single-Agent Timeouts or Resource Starvation

Avoid when: the stall is caused by a single agent waiting on a slow API, rate limit, or model latency. This prompt looks for cycles in dependency graphs, not linear blocking. Guardrail: route single-agent timeout scenarios to a retry-with-backoff or escalation prompt instead.

03

Required Input: Structured Wait-For Graph

What to watch: the prompt requires a machine-readable graph of agents, their held resources, and their requested resources. Passing unstructured logs or chat transcripts produces unreliable cycle detection. Guardrail: build a graph-builder step before this prompt that extracts nodes and edges from your agent orchestration state.

04

Operational Risk: Victim Selection Disruption

What to watch: the prompt selects a victim agent to roll back. If the victim holds critical side effects (e.g., a sent email, a committed database write), rollback may be impossible or damaging. Guardrail: annotate each agent with a rollback_safety flag (safe, unsafe, irreversible) and pass it as a constraint to the prompt.

05

Operational Risk: False Positive Cycle Detection

What to watch: transient states where Agent A requests a resource from Agent B while B is still processing can look like a cycle but resolve on their own. Premature victim selection disrupts healthy pipelines. Guardrail: add a min_stall_duration_seconds threshold and only invoke this prompt after the threshold is exceeded.

06

Operational Risk: Infinite Rollback Loops

What to watch: if the prompt selects a victim, the system rolls it back, and the same deadlock reforms on retry, you get a rollback loop. Guardrail: track rollback history per agent and escalate to human review if the same agent is selected as victim more than twice in a configurable window.

PROMPT PLAYBOOK

Copy-Ready Prompt Template

Paste this prompt into your orchestrator's recovery handler to analyze a stalled agent pipeline and produce a victim-selection and rollback plan.

This prompt template is designed to be injected into your agent orchestrator's deadlock recovery path. When a multi-agent pipeline stalls—detected by a cycle in the wait-for graph or a timeout on resource acquisition—this prompt receives the current dependency snapshot and returns a structured resolution plan. The plan identifies which agent should be rolled back (the victim), what state must be discarded, and how to re-queue the work to avoid immediate re-deadlock. The template uses square-bracket placeholders that your harness must populate from live runtime data before sending the request to the model.

text
You are a deadlock resolution specialist for a multi-agent pipeline. Your job is to analyze a wait-for graph, select a victim agent for rollback, and produce a safe recovery plan that minimizes wasted work and prevents immediate re-deadlock.

## INPUT

### Wait-For Graph
[WAIT_FOR_GRAPH]
# Format: Each line is "AGENT_ID -> RESOURCE_ID" meaning the agent is blocked waiting for that resource.
# Resources can be locks, queue slots, tool availability, or agent completion signals.

### Agent State Snapshot
[AGENT_STATE_SNAPSHOT]
# Format: JSON object mapping AGENT_ID to { "status": "BLOCKED"|"RUNNING"|"IDLE", "holding": [RESOURCE_ID...], "waiting_for": RESOURCE_ID, "progress": "description of work completed so far", "priority": 1-10, "retry_count": integer }

### Resource Ownership Map
[RESOURCE_OWNERSHIP_MAP]
# Format: JSON object mapping RESOURCE_ID to { "owner": AGENT_ID|null, "type": "LOCK"|"QUEUE_SLOT"|"TOOL"|"COMPLETION_SIGNAL", "timeout_seconds": integer }

### Pipeline Context
[PIPELINE_CONTEXT]
# Description of the overall pipeline goal, agent roles, and any ordering constraints that must be preserved.

## CONSTRAINTS
- You must select exactly one victim agent for rollback unless the graph shows no cycle (then respond with NO_DEADLOCK).
- Prefer victims with the lowest priority, highest retry count, or least progress completed.
- The rollback plan must release all resources held by the victim and re-queue the victim's work after a backoff delay.
- Do not select an agent that is holding a resource required by more than [MAX_DEPENDENTS] other agents unless no other choice exists.
- If any agent has exceeded [MAX_RETRY_LIMIT] retries, escalate instead of rolling back.

## OUTPUT SCHEMA
Return a JSON object with exactly this structure:
{
  "deadlock_detected": true|false,
  "cycle_description": "string describing the cycle found, or null if none",
  "victim_agent_id": "AGENT_ID or null",
  "victim_selection_reason": "string explaining why this agent was chosen",
  "rollback_plan": {
    "release_resources": ["RESOURCE_ID..."],
    "discard_progress": "description of what work must be undone",
    "requeue_instruction": "how to re-submit the agent's task after backoff",
    "backoff_seconds": integer
  },
  "escalation_required": true|false,
  "escalation_reason": "string or null",
  "post_rollback_graph_state": "description of expected resource state after rollback"
}

## EXAMPLES

### Example 1: Simple Two-Agent Deadlock
Input Wait-For Graph:
agent_A -> lock_1
agent_B -> lock_2
Agent State: agent_A holds lock_2 waiting for lock_1, agent_B holds lock_1 waiting for lock_2
Output:
{
  "deadlock_detected": true,
  "cycle_description": "agent_A waits for lock_1 held by agent_B; agent_B waits for lock_2 held by agent_A",
  "victim_agent_id": "agent_B",
  "victim_selection_reason": "agent_B has lower priority (3 vs 7) and less progress completed",
  "rollback_plan": {
    "release_resources": ["lock_1"],
    "discard_progress": "Discard partial data transformation on batch_42",
    "requeue_instruction": "Re-submit agent_B task to queue with 5-second delay",
    "backoff_seconds": 5
  },
  "escalation_required": false,
  "escalation_reason": null,
  "post_rollback_graph_state": "agent_A acquires lock_1 and proceeds; agent_B will retry after backoff"
}

### Example 2: No Deadlock
Input Wait-For Graph:
agent_C -> tool_X
Agent State: agent_C waiting for tool_X, tool_X owner is agent_D which is RUNNING
Output:
{
  "deadlock_detected": false,
  "cycle_description": null,
  "victim_agent_id": null,
  "victim_selection_reason": null,
  "rollback_plan": null,
  "escalation_required": false,
  "escalation_reason": null,
  "post_rollback_graph_state": "No deadlock; agent_C is waiting for a running agent to release tool_X"
}

## RISK LEVEL
[RISK_LEVEL]
# HIGH: Require human approval before executing rollback.
# MEDIUM: Execute rollback automatically but log for review.
# LOW: Execute rollback automatically with standard logging.

If RISK_LEVEL is HIGH, set escalation_required to true and include a human-readable summary for the approval queue.

To adapt this template, your harness must populate the five input placeholders from the agent runtime before each invocation. The [WAIT_FOR_GRAPH] should be built by traversing the orchestrator's resource-wait table; the [AGENT_STATE_SNAPSHOT] should pull from the agent status store; and the [RESOURCE_OWNERSHIP_MAP] should reflect the current lock manager state. The [PIPELINE_CONTEXT] placeholder should include enough semantic information about agent roles and ordering constraints that the model can reason about which rollback causes the least disruption. The [MAX_DEPENDENTS] and [MAX_RETRY_LIMIT] thresholds should be set from your pipeline configuration, not hardcoded. Before executing the returned rollback plan, validate that the victim_agent_id actually exists in the current runtime and that the release_resources list matches what the victim currently holds—never trust the model's output without this structural validation.

After receiving the model's response, your harness must parse the JSON and run at least three validation checks before acting: confirm that deadlock_detected matches your own cycle-detection algorithm's result (the model can hallucinate cycles), verify that the selected victim holds the resources listed in release_resources, and check that escalation_required is consistent with your retry-budget policy. For high-risk pipelines, route the resolution plan to a human approval queue before executing the rollback. Log every deadlock event, the model's victim selection, and the final outcome to build an evaluation dataset for tuning the prompt or switching to a deterministic resolution algorithm if the model proves unreliable over time.

IMPLEMENTATION TABLE

Prompt Variables

Inputs the deadlock detection prompt needs to work reliably. Validate these before calling the model.

PlaceholderPurposeExampleValidation Notes

[WAIT_FOR_GRAPH]

Serialized representation of agent dependencies and held resources

{"nodes":["agent_A","agent_B"],"edges":[{"from":"agent_A","to":"agent_B","resource":"db_lock_1"}]}

Must parse as valid JSON with nodes array and edges array. Each edge requires from, to, and resource fields. Reject if graph contains self-loops or duplicate edges.

[AGENT_STATES]

Current status of each agent in the pipeline

{"agent_A":"WAITING","agent_B":"HOLDING","agent_C":"BLOCKED"}

Must be a valid JSON object with agent IDs matching nodes in WAIT_FOR_GRAPH. Allowed states: IDLE, RUNNING, HOLDING, WAITING, BLOCKED. Reject if any agent is missing or state is unrecognized.

[AGENT_PRIORITIES]

Priority weights for victim selection decisions

{"agent_A":1,"agent_B":3,"agent_C":2}

Must be a valid JSON object with numeric values. Higher number means higher priority. All agents in WAIT_FOR_GRAPH must have a priority entry. Reject if any priority is negative or non-numeric.

[RESOURCE_TIMEOUTS]

Maximum wait time per resource before deadlock is declared

{"db_lock_1":"30s","file_handle_2":"60s"}

Must be a valid JSON object with resource IDs matching those in WAIT_FOR_GRAPH edges. Values must parse as duration strings with unit suffix (s, ms, m). Reject if any resource is missing a timeout.

[MAX_ROLLBACK_DEPTH]

Maximum number of agents that can be selected as victims

3

Must be a positive integer. Reject if value is 0, negative, or exceeds total agent count. Null allowed if no limit is desired.

[PREVIOUS_VICTIMS]

Agents already rolled back in prior resolution attempts to prevent starvation

["agent_D"]

Must be a valid JSON array of agent ID strings. Empty array is valid. Reject if any agent ID does not exist in WAIT_FOR_GRAPH nodes. Used to bias victim selection away from repeatedly penalized agents.

[OUTPUT_SCHEMA]

Expected structure for the resolution plan output

{"victims":[],"rollback_steps":[],"justification":""}

Must be a valid JSON Schema or example structure. Reject if schema is missing required fields: victims, rollback_steps, justification. Used to validate model output shape before downstream processing.

PROMPT PLAYBOOK

Implementation Harness Notes

How to wire the deadlock detection prompt into an agent orchestrator or recovery pipeline with validation, retries, and safe escalation.

The deadlock detection prompt is not a standalone tool; it is a recovery component inside an agent orchestrator's error-handling loop. When a workflow stalls—no agent progress for a configurable timeout window—the orchestrator should collect the current wait-for graph, agent states, held resources, and pending actions, then invoke this prompt. The prompt's output is a structured resolution plan: a selected victim agent, rollback instructions, and a safe resumption order. The orchestrator must then execute that plan programmatically, not ask the model to perform the rollback itself. This separation keeps the model in a decision-support role while the harness enforces safety constraints like resource release ordering and state persistence.

Wire the prompt into a recovery pipeline with these concrete stages. First, stall detection: a watchdog timer or heartbeat monitor identifies that no agent has transitioned state within [STALL_TIMEOUT_SECONDS]. Second, graph construction: the harness builds a JSON wait-for graph from the orchestrator's internal state store, including agent IDs, requested resources, current holders, and timestamps. Third, prompt invocation: send the graph and [CONSTRAINTS] (e.g., 'minimize disruption', 'prefer read-only victims') to the model with a strict [OUTPUT_SCHEMA] requiring victim_agent_id, rollback_steps, resumption_order, and rationale. Fourth, output validation: before acting, validate that the victim exists in the graph, that the rollback steps reference real resources, and that the resumption order does not reintroduce the detected cycle. Reject and retry with added context if validation fails. Fifth, execution: the harness preempts the victim agent, releases its held resources in the specified order, queues rollback compensation actions, and resumes waiting agents per the plan. Log every step for postmortem analysis.

For high-risk production pipelines, add a human approval gate when the model's confidence is below a threshold or when the victim is a critical agent (e.g., payment processor, safety monitor). The harness should serialize the resolution plan, pause execution, and notify an on-call engineer via your incident management system. Do not let the model autonomously kill agents that hold financial or safety-critical state. Additionally, implement a retry budget for the detection prompt itself: if validation fails three times, escalate the entire stall to human operators rather than looping. Model choice matters here—use a model with strong reasoning capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent) because cycle detection in large graphs requires multi-hop reasoning that smaller models frequently get wrong.

Finally, treat this prompt as part of a broader reliability stack. Pair it with the Distributed Lock Acquisition Retry Prompt and the Concurrent Agent Plan Merge Conflict Resolution Prompt from this pillar. When a deadlock is resolved, the orchestrator should increment a deadlock_count metric, record the graph snapshot and resolution plan, and feed both into your observability pipeline. Over time, these traces will reveal whether deadlocks stem from resource contention patterns that should be fixed at the architecture level rather than repeatedly resolved at runtime. The prompt is a safety net, not a substitute for sound concurrency design.

IMPLEMENTATION TABLE

Expected Output Contract

Fields, types, and validation rules for the deadlock resolution response. Use this contract to parse and validate the model output before passing it to the rollback harness.

Field or ElementType or FormatRequiredValidation Rule

deadlock_detected

boolean

Must be true if a cycle exists in the wait-for graph; false otherwise. Parse as boolean.

cycle_description

string

If deadlock_detected is true, must list the agents in the cycle in dependency order. If false, must be null.

victim_agent_id

string

Must match one agent ID from the input [WAIT_FOR_GRAPH]. Must be present only if deadlock_detected is true; otherwise null.

victim_selection_rationale

string

Must cite at least one selection factor from [SELECTION_CRITERIA] (e.g., youngest transaction, fewest held locks). Required when deadlock_detected is true.

rollback_plan

array of objects

Each object must contain agent_id (string), action (enum: ROLLBACK, REQUEUE, NOTIFY), and target_state (string). Array must be non-empty when deadlock_detected is true.

affected_agents

array of strings

Must list all agent IDs impacted by the rollback. Must be a subset of [WAIT_FOR_GRAPH] agent IDs. Required when deadlock_detected is true.

escalation_required

boolean

Must be true if the cycle cannot be resolved without human intervention (e.g., all victims hold irreversible side effects). Parse as boolean.

escalation_reason

string or null

Required only if escalation_required is true. Must describe why automated resolution is blocked. Otherwise must be null.

PRACTICAL GUARDRAILS

Common Failure Modes

Deadlock detection prompts fail silently in production when the wait-for graph is incomplete, the model misidentifies victims, or the resolution plan introduces new conflicts. These cards cover the most common failure patterns and the guardrails that catch them before a stalled pipeline becomes a customer-facing outage.

01

Incomplete Wait-For Graph

What to watch: The prompt receives a partial dependency graph missing indirect blockers, causing the model to declare no deadlock when a cycle exists. This happens when agent heartbeats time out or dependency reporting is lossy. Guardrail: Require a graph-completeness check before deadlock analysis. If any agent is unreachable or its status is stale, the prompt must return NEEDS_MORE_DATA instead of NO_DEADLOCK.

02

Victim Selection Causes Cascade Failure

What to watch: The model selects a victim whose rollback triggers a secondary deadlock or orphans downstream agents holding resources. This turns a localized stall into a wider outage. Guardrail: Include a post-selection impact simulation step in the prompt. Before finalizing the victim, the model must list all agents that depend on the victim's output and flag any that would become blocked.

03

Cycle Detection False Negatives

What to watch: The model misses a cycle because the dependency description is ambiguous or the graph is large enough to exceed the model's reliable reasoning depth. The prompt returns NO_DEADLOCK while agents remain stalled. Guardrail: Pair the prompt with a deterministic cycle-detection pre-check in the harness. Use the model only for explanation and victim selection, not for the raw graph-traversal decision.

04

Rollback Plan Loses Critical State

What to watch: The generated rollback instructions discard intermediate results that other agents need, forcing a full restart instead of a partial recovery. Guardrail: Require the prompt to enumerate all state artifacts held by the victim and classify each as discardable, checkpointed, or must-handoff before producing the rollback plan.

05

Resolution Prompt Itself Times Out

What to watch: The deadlock detection prompt runs inside the same resource-constrained pipeline it's trying to rescue. If the model call hangs or exceeds the retry budget, the deadlock persists with no observer. Guardrail: Run deadlock detection in a separate execution context with its own timeout and dead-letter queue. If the detection prompt fails, escalate to a human operator immediately rather than retrying in-band.

06

Ambiguous Agent Identifiers Cause Misdirected Rollback

What to watch: The prompt receives agent identifiers that collide, are partially qualified, or map to multiple instances. The model rolls back the wrong agent, leaving the actual deadlock intact. Guardrail: Enforce fully qualified, unique agent instance IDs in the dependency graph input. Add a validation step that rejects any graph containing duplicate or unresolved identifiers before the prompt runs.

IMPLEMENTATION TABLE

Evaluation Rubric

Criteria for evaluating the deadlock detection and resolution prompt before production deployment. Each criterion targets a specific failure mode in cycle detection, victim selection, or plan generation.

CriterionPass StandardFailure SignalTest Method

Cycle Detection Accuracy

All cycles in the [WAIT_FOR_GRAPH] are correctly identified with no false positives or false negatives

Output misses a documented cycle or reports a cycle where none exists in the graph

Run against 10 hand-crafted graphs with known cycles; compare detected cycles to ground truth

Victim Selection Minimality

Selected victim set is the smallest possible set that breaks all detected cycles

Output selects more agents than necessary to resolve the deadlock or fails to break all cycles

Verify against pre-computed optimal victim sets for each test graph; count selected victims

Rollback Plan Completeness

Every selected victim agent has a concrete rollback instruction including state restoration and resource release steps

A victim agent is named but no rollback action is specified or the action is ambiguous

Parse output for each victim; confirm presence of non-empty rollback instruction field

Non-Victim Preservation

Agents not selected as victims have zero disruption to their current state or ongoing operations

Output suggests pausing, restarting, or modifying state of an agent not in the victim set

Diff the agent state list before and after the resolution plan; confirm only victims are modified

Resource Release Ordering

Resource release steps are ordered to prevent cascading deadlocks when multiple victims release locks

Release order creates a new wait-for dependency between victims or with surviving agents

Simulate the release sequence against the original graph; confirm no new cycles form

Escalation Trigger Correctness

Prompt correctly identifies when deadlock resolution exceeds its authority and escalates instead of forcing a plan

Output proposes a victim-selection plan for a graph marked as requiring human approval

Inject graphs with [ESCALATION_REQUIRED]=true; confirm output is an escalation request, not a resolution plan

Output Schema Compliance

Output strictly matches the [OUTPUT_SCHEMA] with all required fields present and correctly typed

Missing required field, wrong type, or extra fields that violate the schema contract

Validate output with a JSON Schema validator against the defined [OUTPUT_SCHEMA]

Idempotency Key Handling

When [IDEMPOTENCY_KEY] is provided, the output references it and confirms whether this is a new or duplicate resolution request

Output ignores the provided idempotency key or produces a duplicate resolution for an already-resolved deadlock

Submit same graph twice with identical idempotency key; second response must indicate duplicate and return cached resolution

ADAPTATION OPTIONS

Adapt This Prompt

How to adapt

Start with the base prompt and a simple JSON wait-for graph. Use a frontier model (GPT-4o, Claude 3.5 Sonnet) with default temperature. Skip formal schema validation initially—just eyeball the output for cycle detection and victim selection.

Replace the strict output schema with a looser instruction: Return a JSON object with 'deadlock_detected' (boolean), 'cycles' (array of node lists), and 'victim' (string or null).

Watch for

  • The model inventing cycles that don't exist in the graph
  • Victim selection that ignores the [DISRUPTION_COST] weights
  • Output that drifts from JSON into prose explanation
  • Overly conservative deadlock declarations on sparse graphs
Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.