Retry Logic

The retry limit defines the maximum number of times an operation will be reattempted before it is considered a permanent failure. This is a critical guardrail to prevent infinite loops and resource exhaustion.

• Purpose: Balances persistence against the likelihood of success.
• Configuration: Typically set as an integer (e.g., max_attempts: 5).
• Best Practice: Combine with a circuit breaker to stop retries if a downstream service is completely unavailable.

A backoff strategy determines the waiting period between retry attempts. It prevents overwhelming a failing service and increases the chance a transient issue (e.g., network blip, temporary load) resolves.

• Fixed Backoff: Waits a constant duration (e.g., 1 second) between each attempt.
• Exponential Backoff: Doubles the wait time after each failure (e.g., 1s, 2s, 4s, 8s). This is the standard for distributed systems.
• Jitter: Adds random variation to delay times to prevent thundering herd problems where many clients retry simultaneously.

Not all errors should trigger a retry. Retryable error classification is the logic that distinguishes transient, recoverable faults from permanent, logical failures.

• Transient Errors: Timeouts, network connectivity loss, temporary 5xx HTTP status codes (e.g., 503 Service Unavailable), database deadlocks.
• Permanent Errors: 4xx client errors (e.g., 400 Bad Request, 404 Not Found), authentication failures, validation errors.
• Implementation: Policies use allowlists/blocklists of error codes or types to make this determination.

Idempotency is the property that performing an operation multiple times has the same effect as performing it once. For retries to be safe, the retried operation must be idempotent.

• Risk: Without idempotency, a retry could cause duplicate charges, duplicate database entries, or other side effects.
• Techniques: Use unique idempotency keys in API requests, design services with natural idempotency (e.g., SET status = 'complete'), or implement compensating transactions within a Saga pattern for rollback.

The timeout per attempt specifies how long the system will wait for a response on each individual try before considering it a failure and triggering the next retry cycle.

• Function: Prevents a single hung request from blocking the entire retry policy indefinitely.
• Relationship to Backoff: Distinct from the backoff delay. The timeout governs the active request; the backoff governs the inactive waiting period between requests.
• Configuration: Often set separately from the overall workflow timeout.

The fallback action defines what happens after all retry attempts are exhausted and the operation has definitively failed. This is essential for graceful degradation.

• Common Actions:
  • Return a default or cached value.
  • Throw a specific exception to the parent workflow.
  • Trigger a compensating transaction to roll back previous steps in a Saga.
  • Escalate via an alert to human operators.
• Goal: Ensure the system fails predictably and provides a known, controlled outcome.

Idempotent execution is a critical property for reliable retries, where performing the same operation multiple times produces the same, unchanged result as performing it once. This ensures that retrying a failed task does not cause duplicate side effects (e.g., charging a customer twice).

• Key Mechanism: Operations are designed so their effect is the same for one or N identical calls.
• Implementation: Using unique idempotency keys in API requests or ensuring database updates are 'set' operations rather than 'increment'.
• Example: A payment processing task that checks for an existing transaction ID before creating a new charge.

The circuit breaker pattern is a fault-tolerance design pattern that prevents a system from repeatedly attempting an operation that is likely to fail. It acts as a proxy for operations, monitoring for failures and 'opening' the circuit to stop calls temporarily, allowing the underlying service time to recover.

• Three States: Closed (normal operation), Open (fast-fail, no calls made), Half-Open (testing if service is healthy).
• Use Case: Protects against cascading failures when a downstream API or service is experiencing a sustained outage.
• Relation to Retry Logic: A circuit breaker often supersedes aggressive retry policies; when the circuit is open, retries are suspended.

Exponential backoff is a retry strategy where the delay between consecutive retry attempts increases exponentially (e.g., 1s, 2s, 4s, 8s). This is a core policy within retry logic designed to alleviate pressure on a failing system.

• Purpose: Reduces load on a struggling service and helps avoid retry storms that can cause further degradation.
• Common Implementation: Often combined with jitter (randomization) to prevent synchronized retries from multiple clients.
• Example: A workflow task calling a cloud API might wait 2^n seconds (where n is the retry count) before attempting again.

The Saga pattern is a design pattern for managing long-running, distributed transactions. Instead of a traditional ACID transaction, it breaks the process into a sequence of local transactions, each with a corresponding compensating transaction to rollback changes if a later step fails.

• Choreography vs. Orchestration: Can be coordinated via events (choreography) or a central orchestrator.
• Relation to Retry Logic: Individual saga steps often employ retry logic for transient failures. If a step ultimately fails, its compensating transaction is executed, which must also be idempotent and retryable.
• Example: An e-commerce order process involving payment, inventory reservation, and shipping. If shipping fails, compensating transactions refund the payment and restock inventory.

Checkpointing is the process of periodically saving the complete state of a long-running workflow to durable storage. State persistence is the underlying mechanism that durably stores and retrieves runtime state (variables, execution pointer).

• Purpose: Enables fault tolerance by allowing a workflow engine to resume execution from the last saved checkpoint after a failure, rather than from the beginning.
• Critical for Retries: When a task fails and is retried, the workflow engine must be able to restore the exact context (inputs, variables) from which to re-execute it.
• Technology: Often implemented using distributed databases or event-sourcing journals.

A Dead Letter Queue (DLQ) is a holding queue for messages or tasks that have failed all configured retry attempts. It is a last-resort mechanism to ensure failed work is not silently lost and can be inspected for manual intervention or automated reprocessing.

• Workflow Integration: A workflow orchestrator may move a persistently failing task's context to a DLQ after exhausting its retry policy.
• Analysis: DLQs are crucial for observability, allowing engineers to diagnose systemic failures, data errors, or bugs that cause permanent faults.
• Example: A task that processes user uploads fails repeatedly due to a malformed file format. After 5 retries, it's sent to a DLQ where an operator can examine the problematic file.