Retry Policies

The retry limit defines the maximum number of times a failed operation will be re-attempted before being considered a permanent failure. This is a critical guardrail to prevent infinite loops and resource exhaustion.

• Purpose: Balances persistence against the likelihood of success. A limit that is too low may abandon recoverable operations; one that is too high wastes resources on doomed requests.
• Typical Values: Often set between 3 and 5 attempts for HTTP-based APIs, but is highly context-dependent.
• Implementation: The policy must maintain a counter for each unique request and terminate the sequence when the limit is reached, propagating the final error.

A backoff strategy is the algorithm that determines the delay between consecutive retry attempts. Its primary function is to reduce load on a distressed service, allowing it time to recover.

• Exponential Backoff: The most common strategy, where wait times increase exponentially (e.g., 1s, 2s, 4s, 8s). It uses a base delay and a multiplier.
• Linear Backoff: Wait times increase by a fixed increment (e.g., 1s, 2s, 3s, 4s). Less aggressive than exponential.
• Constant Backoff: A fixed delay between all attempts (e.g., 2s, 2s, 2s). Simpler but less effective at reducing load.
• Formula (Exponential): delay = base_delay * (multiplier ^ (attempt_number - 1))

Jitter is the intentional addition of randomness to backoff delays. It is essential to prevent the thundering herd problem, where many synchronized clients retry simultaneously, causing repeated waves of load.

• Purpose: Decorrelates retry timings across distributed clients.
• Implementation Methods:
  • Full Jitter: sleep(random_between(0, base_delay * 2^attempt))
  • Equal Jitter: sleep( (base_delay * 2^attempt) / 2 + random_between(0, (base_delay * 2^attempt) / 2) )
  • Decorrelated Jitter: sleep(random_between(base_delay, previous_delay * 3))
• Result: Smoothes out traffic spikes and increases the aggregate success rate for a recovering service.

This component defines the logic to distinguish transient errors (which should be retried) from permanent errors (which should not). Incorrect classification wastes attempts or fails to recover.

• Transient Error Examples:
  • HTTP 429 (Too Many Requests)
  • HTTP 5xx (Server Errors, except 501 Not Implemented)
  • Network timeouts, connection resets
  • Specific service-defined throttling codes
• Permanent Error Examples:
  • HTTP 4xx (Client Errors like 400 Bad Request, 403 Forbidden, 404 Not Found)
  • HTTP 501 (Not Implemented)
  • Business logic validation failures
• Implementation: The policy evaluates the exception type or HTTP status code from the failed call against a configured list or pattern.

Each individual retry attempt should have its own operation timeout. This prevents a single slow or hanging request from blocking the entire retry sequence for an unacceptable duration.

• Relationship to Overall Deadline: The per-attempt timeout is distinct from a total workflow deadline. The total possible latency is roughly (retry_limit * per_attempt_timeout) + sum_of_backoffs.
• Adaptive Timeouts: Advanced policies may adjust timeouts based on observed latency, using algorithms like TCP's RTT estimation.
• Importance: Without per-attempt timeouts, a series of slow failures can cause unacceptable total latency, violating service-level objectives.

A retry policy is often integrated with a circuit breaker pattern. The circuit breaker acts as a proxy that trips after a defined failure threshold, temporarily blocking all requests to a failing service.

• Synergy: Retries handle transient faults for individual calls. The circuit breaker protects the system when a service is detected as persistently unhealthy.
• Mechanism: When the circuit is open, requests fail immediately without attempting the network call, allowing the downstream service to recover. The retry policy is bypassed.
• States:
  • Closed: Requests flow normally; failures count toward the trip threshold.
  • Open: All requests fail fast.
  • Half-Open: A limited probe of requests is allowed to test if the service has recovered.
• Result: Prevents cascading failures and waste of resources on doomed retries.

A retry algorithm where the wait time between consecutive retry attempts increases exponentially. This prevents overwhelming a recovering service.

• Formula: Typically delay = base_delay * (2 ^ attempt_number).
• Purpose: Gives a struggling backend service time to recover from transient overload or failure.
• Example: Retrying after 1 second, then 2 seconds, then 4 seconds, then 8 seconds.

The intentional addition of randomness to retry delay intervals. It is used to prevent thundering herd problems, where many synchronized clients retry simultaneously.

• Implementation: Add a random value (e.g., ±10-50%) to the calculated backoff delay.
• Benefit: Smoothes out retry traffic, distributing load and increasing the likelihood of overall success.

A resilience pattern that temporarily blocks calls to a failing service. It moves between Closed, Open, and Half-Open states based on failure thresholds.

• Closed: Requests flow normally.
• Open: Requests fail immediately without calling the service.
• Half-Open: Allows a test request to see if the service has recovered.
• Use Case: Complements retry logic by stopping futile retries against a completely down service.

The strategy of forwarding exceptions or failure states from a failed tool call back to the AI agent or orchestration layer. This enables the system to reason about and recover from the error.

• Mechanism: The framework catches the API error, wraps it in a structured format, and returns it to the agent's control loop.
• Agent Response: The agent can then decide on a fallback strategy, rephrase the request, or ask the user for clarification.

Predefined contingency plans executed when a primary tool call fails. They are essential for maintaining a seamless user experience.

• Common Patterns:
  • Calling an alternative API endpoint or service provider.
  • Providing a cached response from a previous, similar call.
  • Using a default or estimated value.
  • Degrading functionality gracefully and informing the user.

A persistent storage queue for messages or requests that have repeatedly failed all retry attempts. It is a critical observability and debugging tool.

• Purpose: Prevents data loss from permanent failures and isolates problematic requests for later analysis.
• Workflow: After the retry limit is exhausted, the failed request metadata (parameters, error) is sent to the DLQ.
• Operator Action: Engineers can inspect the DLQ to diagnose systemic API issues or malformed requests.