Retry Logic

The maximum retry count is a hard limit on the number of times an operation will be re-attempted before being considered a permanent failure. This prevents infinite loops and resource exhaustion.

• Purpose: To bound the total time and compute spent on a failing operation.
• Implementation: Typically configured as an integer (e.g., max_retries: 3).
• Consideration: Must be balanced with the operation's timeout and overall system SLOs. A high value on a slow operation can violate latency guarantees.

The retry delay defines the wait time between consecutive attempts. A backoff strategy algorithmically increases this delay to reduce load on a recovering system.

• Fixed Delay: A constant pause (e.g., 1 second) between all retries. Simple but can cause synchronized retry storms.
• Exponential Backoff: Delay doubles (or multiplies by a factor) with each attempt (e.g., 1s, 2s, 4s, 8s). The standard for handling overloaded services.
• Jitter: Randomization added to delay intervals to desynchronize retries from multiple clients, preventing thundering herd problems.

A retry condition is a predicate that classifies whether a specific failure is transient and warrants a retry. Not all errors should be retried.

• Transient Errors: Network timeouts (e.g., TCP/IP connection refused), HTTP 429 Too Many Requests, 503 Service Unavailable, or database deadlock exceptions.
• Non-Retryable Errors: Client errors like HTTP 400 Bad Request (invalid input) or 404 Not Found. Retrying these is futile and wasteful.
• Implementation: Policies often use HTTP status code ranges or exception type whitelists/blacklists to make this determination.

The per-attempt timeout is the maximum duration allowed for a single try of the operation before it is canceled. This is distinct from the total timeout for all retries combined.

• Purpose: To prevent a single hanging request from blocking the retry loop indefinitely.
• Relationship to Retry Delay: The timeout applies to the execution phase; the retry delay is the idle period between timed-out or failed attempts.
• Best Practice: Set this value lower than the client's overall latency budget to allow for multiple retry cycles within the total acceptable time.

Idempotency is the property that an operation can be applied multiple times without changing the result beyond the initial application. Retry logic requires idempotency for safety.

• Critical for: POST or non-idempotent PATCH API calls, database INSERT operations, or payment processing.
• Techniques: Using client-generated idempotency keys (unique UUIDs) passed to the server, or designing APIs to be inherently idempotent (e.g., using PUT for updates).
• Without idempotency, retries can cause duplicate charges, double orders, or corrupted data.

A robust retry policy does not operate in isolation; it integrates with higher-level resilience patterns.

• Fallback Strategy: Defines an alternative action (e.g., return cached data, default value, or call a secondary service) after retries are exhausted.
• Circuit Breaker: Monitors failure rates. After a threshold is breached, it opens and fails-fast all subsequent requests for a period, bypassing the retry policy entirely to allow the downstream service to recover. Retries resume only after the circuit closes.
• Orchestration: The retry policy executes within the closed state of a circuit breaker.

A retry delay algorithm where the wait time between attempts increases exponentially (e.g., 1s, 2s, 4s, 8s). This prevents overwhelming a recovering service with synchronized retry storms.

• Key Mechanism: Delay = Base Delay * (Backoff Factor) ^ (Attempt Number).
• Purpose: Gives a failing backend service adequate time to recover from transient overload or failures.
• Example: A cloud API client might use a base delay of 100ms with a factor of 2, resulting in delays of 100ms, 200ms, 400ms, etc.

A resilience design pattern that proactively blocks requests to a failing service after a failure threshold is met. It operates in three states:

• Closed: Requests flow normally (system is healthy).
• Open: Requests fail immediately without attempting the call (system is unhealthy).
• Half-Open: A limited number of test requests are allowed to probe if the service has recovered.

This prevents cascading failures and allows the downstream system time to heal.

The randomization of retry delays to prevent the "thundering herd" problem. When many clients retry simultaneously using the same deterministic backoff (e.g., after a service restart), they can collectively overwhelm the service again.

• Implementation: Add a random value to the calculated delay. For example: Delay = CalculatedBackoff + random(0, jitter_max).
• Effect: Staggers retry attempts across a client population, smoothing out load and increasing the overall chance of recovery.

The property of an operation where executing it multiple times yields the same result as executing it once. This is a critical enabler for safe retries.

• Idempotent HTTP Methods: GET, PUT, DELETE. POST is typically not idempotent.
• Implementation: Servers can use client-provided idempotency keys to deduplicate requests.
• Why it Matters: Without idempotency, a retried POST request might create duplicate orders or charge a credit card twice.

A temporary failure that is likely to succeed if retried after a short delay. Retry logic is specifically designed to handle these errors.

Common Causes:

• Network timeouts or packet loss.
• Temporary service unavailability (e.g., during a restart).
• Database connection pool exhaustion.
• Rate limiting (429) or server overload (503) responses.

Identification: Typically signaled by HTTP status codes like 408, 429, 500, 502, 503, 504, or specific exception types in SDKs.

A holding queue for messages or requests that have failed all retry attempts. It serves as a safety net for manual inspection and analysis.

• Purpose: Prevents poison pills from blocking processing pipelines. Engineers can analyze DLQ contents to diagnose persistent bugs or malformed data.
• Common Use: In message systems (AWS SQS, Apache Kafka) and serverless workflows (AWS Lambda).
• Workflow: Request fails → retries exhausted → moved to DLQ → alert triggered → manual remediation.