Distributed Lock Acquisition Retry Prompt

This prompt is designed for Site Reliability Engineers (SREs) and platform engineers who operate distributed systems where lease-based locks (such as those in etcd, ZooKeeper, or Redis Redlock) are critical for mutual exclusion. The primary job-to-be-done is to programmatically generate a safe, context-aware retry strategy when a lock acquisition attempt fails due to contention, network timeouts, or fencing token mismatches. The prompt expects structured failure context—including the lock key, current holder metadata, and the specific error code—to produce a machine-readable retry instruction that a workflow orchestrator can execute.

Use this prompt when the failure mode is transient and contention-based, not when the lock service is permanently down or the caller lacks valid credentials. It is appropriate for high-throughput task queues, leader-election cycles, and idempotent resource provisioning steps where starvation and thundering-herd effects are real risks. The prompt is not a substitute for a deadlock detector or a cluster manager; it assumes the underlying lock infrastructure is healthy and reachable. Do not use this prompt for simple mutexes in single-process applications, for database row-level locks that require SQL-specific resolution, or for human-in-the-loop approval workflows where latency is measured in minutes rather than milliseconds.

Before invoking this prompt, ensure you have captured the raw error response from the lock service, the current monotonic clock reading, and the configured lease TTL. The prompt's value lies in its ability to calculate a jittered exponential backoff window, decide whether to extend a stale lease or abandon it, and escalate to a human operator if a fairness violation is detected. The output is a structured decision object—typically JSON—that your retry harness can parse to sleep, renew, or abort. If your system cannot enforce a maximum retry budget or lacks a dead-letter queue for poison messages, you should implement those circuit breakers in the application layer before relying on this prompt in production.

This template is designed to be copied directly into your AI harness or prompt management system. It instructs the model to act as a distributed-systems reliability engine that produces a concrete retry plan—not generic advice—when a lease-based lock cannot be acquired. The prompt forces the model to reason about backoff schedules, lease renewal windows, starvation risks, and escalation thresholds before returning a structured decision. Every placeholder is enclosed in square brackets and must be replaced with live operational data at runtime: the lock name, the current holder, the failure reason, the configured timeout, and the system's fairness policy.

textCopy
You are a distributed-systems reliability engine. Your job is to produce a safe, actionable retry instruction when a lease-based distributed lock cannot be acquired.

## LOCK CONTEXT
- Lock name: [LOCK_NAME]
- Requested by: [REQUESTER_ID]
- Current holder (if known): [CURRENT_HOLDER_ID]
- Lock TTL (seconds): [LOCK_TTL_SECONDS]
- Elapsed wait time so far (seconds): [ELAPSED_WAIT_SECONDS]
- Maximum total wait budget (seconds): [MAX_WAIT_BUDGET_SECONDS]
- Fairness policy: [FAIRNESS_POLICY]  # e.g., "FIFO", "priority-based", "no-starvation-guarantee"

## FAILURE DETAILS
- Failure reason: [FAILURE_REASON]  # e.g., "lock held by another client", "lease expired during acquisition", "quorum unavailable"
- Error code from lock service: [ERROR_CODE]
- Consecutive acquisition failures for this lock: [CONSECUTIVE_FAILURE_COUNT]
- Are other clients waiting for this lock? [OTHER_WAITERS_EXIST]

## CONSTRAINTS
- [CONSTRAINTS]  # e.g., "Must not exceed 3 retries", "Must preserve FIFO ordering", "Must escalate to on-call after 30s"

## OUTPUT SCHEMA
Return a JSON object with exactly these fields:
{
  "decision": "retry" | "escalate" | "abandon",
  "retry_after_ms": <integer, milliseconds to wait before next attempt, required if decision is retry>,
  "backoff_strategy": "linear" | "exponential" | "decorrelated-jitter" | null,
  "lease_renewal_instruction": <string or null, guidance on renewing the lease if acquired on retry>,
  "starvation_risk": "low" | "medium" | "high",
  "deadlock_detected": true | false,
  "escalation_reason": <string or null, required if decision is escalate>,
  "explanation": <string, one-sentence rationale for the decision>
}

## INSTRUCTIONS
1. If the current holder's lease TTL has likely expired, recommend a short retry delay.
2. If multiple clients are waiting and the fairness policy is FIFO, do not recommend aggressive backoff that would cause queue jumping.
3. If the elapsed wait time exceeds 80% of the max wait budget, escalate unless the lock is expected to release imminently.
4. If the same requester has failed [CONSECUTIVE_FAILURE_COUNT] times and the failure reason is unchanged, consider escalation.
5. If a deadlock pattern is detected (e.g., cyclic dependency between lock holders), flag deadlock_detected as true and escalate.
6. Never recommend a retry delay shorter than the lock service's observed latency floor of 50ms.
7. Never recommend infinite retries. Always respect the max wait budget.

Return only the JSON object. No markdown fences, no commentary.

To adapt this template for your environment, replace each bracketed placeholder with live data from your lock service and orchestration layer. The [LOCK_NAME] and [REQUESTER_ID] should come from your distributed lock client library (e.g., Redlock, etcd, ZooKeeper, or a cloud lock service). The [FAILURE_REASON] and [ERROR_CODE] must be extracted from the lock-acquisition exception or gRPC status code. The [FAIRNESS_POLICY] field is critical: if your system uses a FIFO lock queue, set it to "FIFO" so the model avoids backoff strategies that would let a latecomer jump ahead of waiting clients. If your system has no fairness guarantee, set it to "no-starvation-guarantee" and the model will prioritize liveness over ordering. The [CONSTRAINTS] field is your escape hatch for business-specific rules—use it to inject retry budgets, escalation contacts, or compliance requirements that the model cannot infer from the other fields.

Before deploying this prompt into a production retry loop, validate that your harness enforces the model's output against the schema. If the model returns decision: "retry" but omits retry_after_ms, your harness must reject the output and either re-prompt or fall back to a safe default backoff. If deadlock_detected is true, your harness should immediately halt all lock-acquisition attempts for the involved clients and route to a human operator or a deadlock-resolution workflow. Do not treat this prompt as a replacement for client-side backoff libraries—it is a decision layer that sits above your retry infrastructure and makes context-aware choices that static backoff configs cannot.

The Distributed Lock Acquisition Retry Prompt is designed to be called by a lock-manager service when a lease-based lock acquisition attempt fails. The harness should invoke the model only after a primary fast-path acquisition attempt has been rejected by the lock backend (e.g., Redis, etcd, or a database with advisory locks). The prompt receives the lock key, the current holder's metadata, the failed attempt's error context, and the configured retry policy. The model's job is not to acquire the lock itself but to produce a structured retry instruction that the harness then executes. This separation keeps the model out of the critical path for lock operations while letting it reason about backoff strategy, lease renewal, and escalation when contention patterns are complex.

The harness must validate the model's output against a strict schema before acting on it. Expect a JSON object with fields: decision (one of retry, escalate, abort), backoff_ms (integer, must respect the configured max_backoff_ms), reasoning (a short string for logs), and optional escalation_reason when the decision is escalate. If the model returns retry, the harness should schedule a re-acquisition attempt after the specified backoff, with jitter applied client-side to avoid thundering-herd effects. If the model returns escalate, the harness must route the lock key and contention metadata to a human on-call or a higher-tier automated resolver. If the model returns abort, the harness should surface the reasoning to the caller and stop retrying. Any output that fails schema validation should trigger a single repair retry with the validation error included in the prompt context; if that also fails, the harness should fall back to a configurable default backoff schedule and log the failure for later prompt debugging.

Logging and observability are critical. Every model invocation should log the lock key, the attempt count, the model's decision, the backoff applied, and the eventual outcome (acquired, escalated, or abandoned). These logs feed into starvation and fairness monitors: if a lock key appears in escalation logs repeatedly, the harness should flag it for deadlock review. The harness should also enforce a hard retry budget (e.g., max_retries and max_total_wait_ms) that the model cannot override. When the budget is exhausted, the harness escalates regardless of the model's suggestion. For model selection, a fast, low-cost model (e.g., GPT-4o-mini, Claude Haiku, or a fine-tuned small model) is usually sufficient since the reasoning task is bounded. Reserve larger models for cases where the contention pattern involves multiple interacting locks and the prompt is extended with a wait-for graph. Do not use this prompt in environments where lock acquisition latency must be sub-millisecond; the model round-trip adds unacceptable overhead for those use cases.