Retry Logic

Exponential backoff is a core algorithm that progressively increases the wait time between retry attempts. This prevents overwhelming a recovering service and is a standard practice for handling congestion.

• Formula: The delay before the n-th retry is typically calculated as delay = base_delay * (backoff_factor ^ (n-1)).
• Jitter: Adding random jitter (a small, random variation) to the delay helps prevent thundering herd problems where many clients retry simultaneously.
• Example: A policy with a base_delay of 1 second and a backoff_factor of 2 will retry after 1s, 2s, 4s, 8s, etc.

A robust policy must distinguish between transient errors (which can be retried) and permanent errors (which should not be). Retrying permanent errors wastes resources and delays surfacing the actual problem to the application.

• Retryable Errors: Typically include HTTP 429 Too Many Requests, 503 Service Unavailable, 502 Bad Gateway, and network timeouts or connection resets.
• Non-Retryable Errors: Include client errors like 400 Bad Request (invalid query), 401 Unauthorized, 403 Forbidden, and 404 Not Found. These indicate a problem with the request itself that will not change on retry without client-side correction.

These are the guardrails that prevent infinite retry loops and bound the total time spent on a single operation.

• Max Retries: Defines the absolute ceiling for retry attempts (e.g., 5). After this, the failure is propagated to the caller.
• Overall Timeout: A deadline for the entire operation, including all retries. This ensures a request does not hang indefinitely.
• Per-Attempt Timeout: A timeout for each individual API call attempt, which is distinct from the backoff delay. This quickly fails attempts that are stuck.

Retry logic must be designed with idempotency in mind. An idempotent operation produces the same result whether executed once or multiple times.

• Safe Methods: GET, HEAD, and PUT requests are generally idempotent and safe to retry.
• Non-Idempotent Methods: POST requests (like inserting a vector) may not be idempotent. Retrying could cause duplicate data.
• Solution: SDKs should use idempotency keys (client-provided unique identifiers) for non-idempotent operations. The vector database uses this key to deduplicate requests, ensuring a retry does not create a duplicate record.

Retries should be transparent and observable to aid in debugging and system monitoring.

• Request IDs: Each initial request and all its retries should share a unique correlation ID, making it easy to trace the lifecycle of an operation in logs.
• Metrics: The SDK should emit metrics such as retry_attempts_total, retry_duration_seconds, and error counts by type. This is crucial for SLO/SLI tracking.
• Logging: Log entries should indicate when a retry is being attempted, including the attempt number and the reason for the previous failure, without being overly verbose.

Retry logic is rarely used in isolation. It is part of a broader resilience strategy and must integrate with other patterns.

• Circuit Breaker: Works in tandem with retries. If failures persist beyond a threshold, the circuit breaker opens and fails requests immediately, bypassing retries. This allows the downstream service to recover. After a cooldown period, it allows a test request through (half-open state) before closing again.
• Fallbacks: When retries are exhausted, a policy can execute a fallback action, such as returning cached data, a default value, or a graceful error message, rather than just throwing an exception.

A retry strategy where the wait time between consecutive retry attempts increases exponentially. This prevents overwhelming a failing service and is often combined with jitter (randomized delays) to avoid thundering herd problems.

• Formula: delay = base_delay * (2 ^ attempt_number)
• Example: A client might wait 1s, then 2s, then 4s, then 8s between retries.
• Use Case: Essential for handling transient failures like network timeouts or temporary database overload.

A resilience pattern that monitors for failures and, when a threshold is exceeded, "opens" the circuit to fail fast. This prevents cascading failures and allows the downstream service (e.g., a vector database) time to recover.

• States: Closed (normal operation), Open (fail-fast), Half-Open (testing recovery).
• Implementation: Libraries like Resilience4j or Polly provide this functionality.
• Relation to Retries: A circuit breaker often triggers after retry logic has been exhausted, stopping further attempts temporarily.

A property of an API operation where performing it multiple times has the same effect as performing it once. This is critical for safe retries, as a retried request won't create duplicate data.

• Idempotent Methods: HTTP GET, PUT, DELETE. POST is not inherently idempotent.
• Implementation: Vector databases often use client-supplied idempotency keys or unique vector IDs to ensure upsert operations are safe to retry.
• Example: Retrying a vector upsert with ID vec_123 should update the existing vector, not create a duplicate.

A persistent storage mechanism for messages or requests that have failed all processing attempts, including retries. This allows for later analysis and manual intervention.

• Workflow: Request → Retry (N times) → Fail → Send to DLQ.
• Content: The DLQ stores the failed request payload, error context, and retry count.
• Use Case: Capturing persistent failures (e.g., malformed vector data, schema violations) that retries cannot resolve.

Mechanisms used to determine the operational status of a service. SDKs can use these to inform retry logic, avoiding requests to an unhealthy endpoint.

• Liveness Probe: Indicates if the service is running.
• Readiness Probe: Indicates if the service is ready to accept traffic (e.g., database connections are pooled).
• Integration: An intelligent SDK might suspend retries or route traffic away from an instance failing health checks.

A performance and resilience technique where a cache of database connections is maintained and reused, rather than creating a new connection for each request.

• Resilience Benefit: Reduces the overhead and potential point of failure associated with frequent connection establishment, which is a common source of transient errors.
• Retry Context: A retry that uses a fresh connection from a pool may succeed where a retry on a stale or broken connection would fail.
• Management: Pools often include settings for max connections, timeouts, and validation queries.