Retry Logic

Not all errors should trigger a retry. Retryable error classification is the logic that distinguishes transient, recoverable faults from permanent, logical failures.

• Transient Errors: Timeouts, network connectivity loss, temporary 5xx HTTP status codes (e.g., 503 Service Unavailable), database deadlocks.
• Permanent Errors: 4xx client errors (e.g., 400 Bad Request, 404 Not Found), authentication failures, validation errors.
• Implementation: Policies use allowlists/blocklists of error codes or types to make this determination.

Idempotency is the property that performing an operation multiple times has the same effect as performing it once. For retries to be safe, the retried operation must be idempotent.

• Risk: Without idempotency, a retry could cause duplicate charges, duplicate database entries, or other side effects.
• Techniques: Use unique idempotency keys in API requests, design services with natural idempotency (e.g., SET status = 'complete'), or implement compensating transactions within a Saga pattern for rollback.

The timeout per attempt specifies how long the system will wait for a response on each individual try before considering it a failure and triggering the next retry cycle.

• Function: Prevents a single hung request from blocking the entire retry policy indefinitely.
• Relationship to Backoff: Distinct from the backoff delay. The timeout governs the active request; the backoff governs the inactive waiting period between requests.
• Configuration: Often set separately from the overall workflow timeout.

The fallback action defines what happens after all retry attempts are exhausted and the operation has definitively failed. This is essential for graceful degradation.

• Common Actions:
  • Return a default or cached value.
  • Throw a specific exception to the parent workflow.
  • Trigger a compensating transaction to roll back previous steps in a Saga.
  • Escalate via an alert to human operators.
• Goal: Ensure the system fails predictably and provides a known, controlled outcome.

Exponential backoff is a retry strategy where the delay between consecutive retry attempts increases exponentially (e.g., 1s, 2s, 4s, 8s). This is a core policy within retry logic designed to alleviate pressure on a failing system.

• Purpose: Reduces load on a struggling service and helps avoid retry storms that can cause further degradation.
• Common Implementation: Often combined with jitter (randomization) to prevent synchronized retries from multiple clients.
• Example: A workflow task calling a cloud API might wait 2^n seconds (where n is the retry count) before attempting again.

The Saga pattern is a design pattern for managing long-running, distributed transactions. Instead of a traditional ACID transaction, it breaks the process into a sequence of local transactions, each with a corresponding compensating transaction to rollback changes if a later step fails.

• Choreography vs. Orchestration: Can be coordinated via events (choreography) or a central orchestrator.
• Relation to Retry Logic: Individual saga steps often employ retry logic for transient failures. If a step ultimately fails, its compensating transaction is executed, which must also be idempotent and retryable.
• Example: An e-commerce order process involving payment, inventory reservation, and shipping. If shipping fails, compensating transactions refund the payment and restock inventory.

Checkpointing is the process of periodically saving the complete state of a long-running workflow to durable storage. State persistence is the underlying mechanism that durably stores and retrieves runtime state (variables, execution pointer).

• Purpose: Enables fault tolerance by allowing a workflow engine to resume execution from the last saved checkpoint after a failure, rather than from the beginning.
• Critical for Retries: When a task fails and is retried, the workflow engine must be able to restore the exact context (inputs, variables) from which to re-execute it.
• Technology: Often implemented using distributed databases or event-sourcing journals.

A Dead Letter Queue (DLQ) is a holding queue for messages or tasks that have failed all configured retry attempts. It is a last-resort mechanism to ensure failed work is not silently lost and can be inspected for manual intervention or automated reprocessing.

• Workflow Integration: A workflow orchestrator may move a persistently failing task's context to a DLQ after exhausting its retry policy.
• Analysis: DLQs are crucial for observability, allowing engineers to diagnose systemic failures, data errors, or bugs that cause permanent faults.
• Example: A task that processes user uploads fails repeatedly due to a malformed file format. After 5 retries, it's sent to a DLQ where an operator can examine the problematic file.