Step Retry Logic

Step retry logic is initiated by specific, pre-defined failure conditions rather than all errors. Common triggers include:

• Transient network timeouts (e.g., HTTP 408, 429, 503)
• Resource contention errors (e.g., database deadlocks)
• Temporary service unavailability
• Rate limit exceeded responses

It is explicitly not used for semantic errors (e.g., invalid input, permission denied, logical bugs), as retrying would not resolve the underlying issue.

The behavior of a retry mechanism is governed by a policy that defines its operational limits and strategy. A robust policy includes:

• Maximum Retry Attempts: A hard cap (e.g., 3-5 retries) to prevent infinite loops.
• Retry Delay Strategy: The wait time between attempts. Common patterns are:
  • Fixed Delay: A constant pause (e.g., 1 second).
  • Exponential Backoff: Delay doubles with each attempt (e.g., 1s, 2s, 4s, 8s), reducing load on recovering systems.
  • Jitter: Random variation added to delays to prevent thundering herds.
• Retryable Error Classification: A list or pattern-matching rule for which exceptions trigger a retry.

A core engineering requirement for safe retries is idempotency—the property that an operation can be applied multiple times without changing the result beyond the initial application. This is critical because:

• The initial failed attempt may have partially succeeded.
• The system state may have changed between retries.

Techniques to ensure idempotency include:

• Using idempotency keys (unique client-generated tokens) with APIs.
• Designing compensating transactions or Saga patterns for rollback.
• Implementing checkpointing to save progress before a retryable step.

Sophisticated step retry logic does not blindly repeat the same request. It can modify the execution context based on the failure mode:

• Parameter Variation: Slightly altering query parameters, time ranges, or API endpoints.
• Fallback Resource Switching: Retrying with a different database replica, API gateway, or service instance.
• Degraded Mode Activation: Reducing the complexity of a request (e.g., asking for less data, using a faster but less accurate model).
• Credential Rotation: Automatically switching to a backup API key or authentication token if a quota is exhausted.

Retry logic must be fully instrumented to avoid masking systemic issues. Essential observability includes:

• Retry Counters: Metrics tracking the volume and rate of retries per operation.
• Failure Causality: Structured logs that link the initial failure to subsequent retries, preserving the root error.
• Latency Attribution: Distinguishing time spent in initial attempts versus retry delays in performance dashboards.
• Alerting on Retry Storms: Setting alarms for when retry rates exceed a threshold, indicating a potential downstream service outage.

Step retry logic is often paired with the Circuit Breaker pattern to prevent catastrophic cascading failures. The circuit breaker monitors failure rates:

• Closed State: Normal operation, requests pass through; retries occur per policy.
• Open State: After a failure threshold is breached, the circuit "opens" and fails fast without retrying, allowing the downstream service time to recover.
• Half-Open State: After a timeout, a limited number of trial requests are allowed; success closes the circuit, failure re-opens it.

This combination prevents retry logic from overwhelming an already failing service.

This is the most common implementation for handling transient network failures or rate limiting. The system waits progressively longer between retry attempts.

• Initial Delay: 1 second.
• Backoff Multiplier: 2x (e.g., 1s, 2s, 4s, 8s).
• Max Retries: 3-5 attempts.
• Jitter: Adds random milliseconds to delays to prevent thundering herd problems when many clients retry simultaneously.

Example: An agent calling a weather API receives a 429 Too Many Requests response. It waits 1 second, retries, and if it fails again, waits 2 seconds before the next attempt.

In systems using Optimistic Concurrency Control (OCC) or facing database deadlocks, retry logic manages transient contention.

• Failure Detection: Catches specific SQL error codes (e.g., SQLSTATE 40001 for serialization failure, 40P01 for deadlock).
• Immediate Retry: For deadlocks, a very short, randomized delay (e.g., 50-250ms) is often sufficient, as the competing transaction may have already completed.
• Circuit Breaker: After N failures, the operation may be routed to a different database replica or fail fast to avoid cascading load.

Example: An agent updating a user profile encounters a deadlock. The database driver raises an exception, the agent's retry handler catches it, waits 100ms, and re-executes the transaction.

When an LLM-based agent fails to call an external tool or API, retry logic can involve parameter adjustment and model cascading.

• Parameter Retry: The initial failure may be due to a malformed JSON request. The agent can re-prompt the LLM with a stricter schema or example before retrying the tool call.
• Fallback Path: If the primary LLM (e.g., GPT-4) consistently fails a structured generation step, the system can retry the step using a more reliable, smaller model (e.g., Claude Haiku) specialized for formatting.
• Timeout Handling: If a tool call exceeds a service level objective (SLO) (e.g., 5 seconds), it's canceled and retried, potentially with a simplified query.

Handling momentary I/O errors on network-attached storage or cloud object stores requires simple, fast retries.

• Linear Delay: Fixed, short intervals between attempts (e.g., 100ms).
• Error Classification: Retries only on specific errors like EAGAIN, EWOULDBLOCK, or ECONNRESET. Permissions errors (EACCES) are not retried.
• Stateless Retry: The operation (e.g., open(), read()) is idempotent and can be safely repeated.

Example: An agent writing a checkpoint file to an NFS share gets an EIO (Input/output error). It retries twice with a 200ms delay before escalating to a fallback execution path, like writing to a local tmp directory.

In a Saga pattern, if a local transaction fails, the step retry logic is coupled with compensating transactions for rollback.

• Forward Retry: The system retries the failed business transaction (e.g., DebitAccount) up to a limit.
• Backward Recovery: If forward retries are exhausted, the Saga executor must reliably execute the compensating action (e.g., CreditAccount). This compensating action itself must have robust retry logic.
• Idempotency Keys: All retries use a unique idempotency key to prevent double-charging or duplicate side effects.

Example: In an order placement Saga, the 'Process Payment' step fails due to a bank gateway timeout. It is retried twice. If it ultimately fails, the 'Compensate Payment' (refund) action is invoked, which also includes retries to ensure consistency.

Consumers processing messages from a queue (e.g., AWS SQS, RabbitMQ) use retry logic with a dead-letter queue (DLQ) as a final fallback.

• Visibility Timeout: On failure, the message is not deleted but becomes visible again after a timeout, triggering a retry.
• Max Receives: The queue is configured with a maxReceiveCount (e.g., 3). After exceeding this, the message is automatically moved to a DLQ.
• Poison Message Handling: Messages that consistently fail (e.g., due to corrupted data) are quarantined to the DLQ for manual inspection, preventing the consumer from getting stuck in a retry loop.

Example: An agent processing a 'Send Email' message fails because the SMTP server is down. The message returns to the queue, is retried twice more, and is then moved to a DLQ for alerting and manual reprocessing.