Token-Based Retry Budget Configuration Prompt Template

This prompt is for AI platform architects and reliability engineers who need to enforce a retry budget based on token consumption rather than attempt count or wall-clock time. Use it when your system retries failed model calls, and you need to cap the total tokens spent across all attempts for a single request. This prevents a single difficult task from consuming disproportionate compute resources. The prompt produces a structured budget definition that your application harness can enforce. It is not a retry loop itself; it is the configuration artifact that governs when your harness must stop retrying and escalate.

The ideal user is building a production AI harness where retry logic already exists, but the cost of retries is unbound. You should have a clear understanding of your model's pricing per 1K tokens and the value classification of the task (e.g., critical, standard, best-effort). The prompt requires you to specify a [TASK_TYPE], a [MODEL_PRICING_TIER], and a [MAX_COST_BUDGET] in your application's currency. The output is a machine-readable policy object containing a token_budget, a cost_budget, and an exhaustion_action (e.g., escalate_to_human, return_partial, fail_gracefully). Do not use this prompt for simple, non-critical tasks where a fixed attempt count is sufficient, or for latency-bound real-time systems where a time-based budget is more appropriate.

Before integrating this prompt's output, ensure your harness can accurately count cumulative prompt and completion tokens across all retry attempts for a single logical request, including tokens consumed by tool calls and their results. The primary failure mode is token counting inaccuracies, which can cause premature or delayed budget exhaustion. Implement a validator that checks the output for a positive integer token_budget, a non-negative cost_budget, and a valid exhaustion_action from a predefined enum. Wire this configuration into your retry loop's conditional check: before each retry, estimate the tokens to be consumed and verify that the running total plus the estimate does not exceed the budget. If it does, execute the exhaustion_action immediately. Log the final token count and budget status for cost attribution and monitoring.

This prompt template defines a retry budget keyed to cumulative token consumption across attempts. It is designed for AI platform architects who need to enforce cost and latency ceilings on retry loops before escalating to a fallback or human review. The template accepts a request type, a token budget, and a set of escalation rules, then produces a structured retry policy that can be consumed by an orchestration harness.

codeCopy
You are a retry budget configuration generator. Your output will be consumed by an AI orchestration harness that enforces retry limits based on cumulative token consumption.

Given the following inputs, produce a strict retry budget definition in valid JSON.

[REQUEST_TYPE]: [INPUT]
[TOKEN_BUDGET]: [INPUT]
[ESCALATION_TARGET]: [INPUT]
[FALLBACK_ACTION]: [INPUT]
[CONSTRAINTS]: [INPUT]

[OUTPUT_SCHEMA]:
{
  "retry_budget": {
    "request_type": "string (from [REQUEST_TYPE])",
    "max_cumulative_tokens": "integer (from [TOKEN_BUDGET])",
    "token_counting_strategy": "'cumulative_input_output' | 'input_only' | 'output_only'",
    "per_attempt_constraints": {
      "max_attempts": "integer (optional, derived from budget and estimated token-per-attempt)",
      "backoff_policy": "'none' | 'linear' | 'exponential'"
    }
  },
  "escalation_policy": {
    "trigger": "'budget_exhausted' | 'non_improving_output' | 'identical_error'",
    "target": "string (from [ESCALATION_TARGET])",
    "fallback_action": "string (from [FALLBACK_ACTION])",
    "context_packaging": {
      "include_retry_history": true,
      "include_error_traces": true,
      "include_token_consumption_log": true
    }
  },
  "validator_checks": [
    "token_budget_is_positive_integer",
    "escalation_target_is_valid_uri_or_queue",
    "fallback_action_is_idempotent_if_applicable",
    "token_counting_strategy_is_recognized"
  ]
}

[RULES]:
1. The token budget must be a positive integer.
2. If [TOKEN_BUDGET] is less than the estimated tokens for a single attempt, set max_attempts to 1 and trigger immediate escalation after failure.
3. The escalation target must be a valid URI, queue name, or human reviewer identifier.
4. The fallback action must be safe to execute without side effects if the escalation target is unreachable.
5. Include validator checks that the orchestration harness can run before enforcing the budget.
6. If [CONSTRAINTS] includes a latency SLA, prefer a token budget that keeps cumulative processing under that SLA.
7. Output ONLY the JSON object. No markdown fences, no commentary.

To adapt this template, replace each square-bracket placeholder with concrete values from your system configuration. For example, set [REQUEST_TYPE] to "customer_support_ticket_summary", [TOKEN_BUDGET] to 8000, [ESCALATION_TARGET] to "human_review_queue_us_east", and [FALLBACK_ACTION] to "return_partial_summary_with_uncertainty_flag". The [CONSTRAINTS] field can carry additional rules such as "must complete within 2000ms p99" or "do not retry non-idempotent POST requests". After generating the policy, validate the JSON against the schema and run the listed validator checks in your harness before accepting the configuration. For high-risk request types, always require a human to approve the escalation target and fallback action before deployment.

Wiring this prompt into a production harness requires a stateful retry loop that tracks cumulative token consumption across attempts. The application must maintain a running counter of prompt tokens, completion tokens, and total tokens for each request, updating the counter after every model call. Before each retry, the harness injects the current token totals into the prompt's [CURRENT_TOKEN_USAGE] placeholder and checks whether the next attempt would exceed the configured [TOKEN_BUDGET] threshold. This pre-flight check prevents unnecessary model calls when the budget is already exhausted, saving both cost and latency.

The harness should implement three core components: a token accumulator that sums usage from the model's usage response object, a budget validator that compares accumulated tokens against thresholds before each retry, and an escalation dispatcher that routes exhausted-budget cases to the configured fallback. For token-counting accuracy, always use the model's reported token counts rather than client-side estimation—different tokenizers produce different counts, and the model's own accounting is the source of truth for billing. Log every attempt's token consumption, the running total, and the budget-check decision to an observability system. When the budget is exhausted, the harness should package the retry history, error summaries, and partial outputs into the escalation payload defined by the prompt's [ESCALATION_TARGET] and [ESCALATION_FORMAT] parameters. For high-risk domains, require human approval before the escalation payload is sent to downstream systems.

Model choice matters for this workflow. Use a fast, inexpensive model for the budget-check and escalation-packaging steps—the prompt is primarily a decision and formatting task, not a generation task. If the retry budget prompt itself fails or produces malformed escalation JSON, the harness must have a hardcoded fallback that escalates with raw error context rather than entering its own retry loop. Avoid the anti-pattern of applying a retry budget to the retry budget prompt; instead, fail closed and alert an operator. Test the harness with simulated token-exhaustion scenarios, malformed model responses, and edge cases where the reported token count is zero or missing from the API response.