Tool Call with Rate Limiting and Cost Awareness Prompt

This prompt is for agent developers whose systems operate under strict API constraints. It teaches the model to make cost-aware and rate-limit-aware tool-calling decisions through few-shot examples rather than abstract rules. Use it when your agent has access to multiple tools with different cost profiles, rate limits, or caching behaviors, and you need the model to batch requests, avoid redundant calls, and stay within a defined budget.

The prompt works by providing concrete demonstrations of good and bad tool-calling sequences under resource constraints. Each example shows the model how to reason about tool costs, identify batching opportunities, and respect rate limits before making a call. This is more effective than describing rules because the model learns the pattern of checking constraints before acting. The prompt template includes placeholders for your specific tools, their cost and rate-limit metadata, and example scenarios that match your application's typical workloads.

This prompt is not a substitute for application-layer rate limiting or circuit breakers. It works alongside those controls to reduce the likelihood of the model proposing call sequences that will fail or exceed cost thresholds. Do not use this prompt when your agent has only one tool, when all tools have identical cost and rate-limit profiles, or when you have already implemented deterministic batching logic in your application code. For high-stakes financial operations or compliance-sensitive workflows, always pair this prompt with human review gates and audit logging of actual tool calls made.

This template uses concrete worked examples to demonstrate the desired behavior rather than relying solely on abstract rules. The examples show the model how to analyze a set of pending tool calls, identify which can be batched into a single request, recognize when a cached result is still valid, and order the remaining calls from cheapest to most expensive. The square-bracket placeholders let you inject your own tool definitions, cost model, rate limit parameters, and the current task context. Copy the template, replace the placeholders with your production values, and test it against the eval criteria described in the testing section of this playbook.

codeCopy
SYSTEM:
You are an agent operating under strict API constraints. Before making any tool calls, you must analyze the requested operations and produce an execution plan that minimizes cost and respects rate limits. Follow these rules:
- Batch independent calls to the same endpoint into a single request when the API supports it.
- Check the [CACHE_STATE] before calling any read operation. If a valid cached result exists, use it instead of making a new call.
- Order remaining calls from lowest cost to highest cost according to [COST_MODEL].
- Never exceed [RATE_LIMIT] calls per [WINDOW].
- If the plan would exceed the budget [MAX_BUDGET], drop the lowest-priority calls and explain why.

USER:
[CURRENT_TASK]

Available tools and costs:
[TOOL_DEFINITIONS]

Current cache state:
[CACHE_STATE]

Rate limit remaining: [RATE_LIMIT_REMAINING] calls in the current [WINDOW] window.
Budget remaining: [BUDGET_REMAINING] of [MAX_BUDGET].

--- EXAMPLES ---

Example 1: Batching opportunity
Task: Get user profiles for IDs [101, 102, 103] and their last order dates.
Tools: getUserProfile (cost: 1, supports batch: yes, max batch: 10), getLastOrderDate (cost: 2, supports batch: no)
Cache: empty
Rate limit: 10 calls remaining
Budget: 15

Plan:
1. Batch call getUserProfile with ids=[101,102,103] (cost: 1, 1 call)
2. Call getLastOrderDate for user 101 (cost: 2, 1 call)
3. Call getLastOrderDate for user 102 (cost: 2, 1 call)
4. Call getLastOrderDate for user 103 (cost: 2, 1 call)
Total cost: 7, Total calls: 4

Example 2: Cache hit
Task: Get user profile for ID 101 and their account balance.
Tools: getUserProfile (cost: 1, supports batch: no), getAccountBalance (cost: 3, supports batch: no)
Cache: userProfile:101 = {"name": "Alice", "tier": "premium"} (cached 2 minutes ago, TTL: 5 minutes)
Rate limit: 5 calls remaining
Budget: 10

Plan:
1. Use cached userProfile:101 (cost: 0, 0 calls)
2. Call getAccountBalance for user 101 (cost: 3, 1 call)
Total cost: 3, Total calls: 1

Example 3: Budget exceeded
Task: Generate reports for Q1, Q2, Q3, Q4 and run fraud scan.
Tools: generateReport (cost: 5, supports batch: no), runFraudScan (cost: 8, supports batch: no)
Cache: empty
Rate limit: 10 calls remaining
Budget: 20

Plan:
1. Call generateReport for Q1 (cost: 5, 1 call)
2. Call generateReport for Q2 (cost: 5, 1 call)
3. Call generateReport for Q3 (cost: 5, 1 call)
Total cost: 15, Total calls: 3
Dropped: generateReport for Q4 (cost: 5, would exceed budget), runFraudScan (cost: 8, would exceed budget)
Reason: Remaining budget after Q1-Q3 is 5, insufficient for Q4 (5) or fraud scan (8). Fraud scan is highest cost and dropped first per cost-ordering rule.

--- END EXAMPLES ---

Now produce your execution plan for the current task. Output a JSON object with this schema:
[OUTPUT_SCHEMA]

To adapt this template, start by replacing [TOOL_DEFINITIONS] with your actual function schemas, including the cost field and supports_batch boolean for each tool. Define [COST_MODEL] as a simple mapping or a reference to the cost fields in your tool definitions. The [CACHE_STATE] placeholder should be populated at runtime with the current keys, values, timestamps, and TTLs from your cache layer. Replace [RATE_LIMIT] and [WINDOW] with your actual API tier limits, and wire [RATE_LIMIT_REMAINING] to a real-time counter in your application. The [OUTPUT_SCHEMA] should be a strict JSON schema that includes fields for the ordered plan steps, total cost, total calls, and any dropped items with reasons. Validate every model output against this schema before executing any tool calls. If the plan's calculated cost exceeds [MAX_BUDGET] or the call count exceeds the rate limit, reject the plan and request a revision from the model with the specific constraint violation included in the error message.

The few-shot examples are the engine of this prompt. They teach batching, cache awareness, and budget-constrained ordering more reliably than instructions alone. When you modify the examples, preserve the pattern of showing the task, the available tools with costs, the cache state, the constraints, and then the step-by-step plan with cost and call counts. Include at least one example that demonstrates dropping calls when the budget is exceeded, and one that shows a cache hit avoiding a call entirely. If your production tools have different batching semantics or cost structures, replace the examples with ones that match your actual API. Test the prompt with the eval harness described in the testing section to catch regressions when you change the examples or tool definitions.

The prompt template is the brain, but the harness is the skeleton. To make this work in production, you need a thin application layer that enforces the rate limits and budget the prompt reasons about. The prompt generates a plan; the harness validates it, executes it, and feeds back results. Start by defining a ToolBudget object in your application code that tracks remaining calls per tool, total cost incurred, and a hard ceiling. Before executing any tool call the model proposes, check it against this budget. If the model suggests a call that would exceed a limit, do not execute it. Instead, inject a system message like SYSTEM: Tool call to [TOOL_NAME] blocked. Rate limit exceeded. Remaining budget: [BUDGET_STATE]. and re-prompt the model for an alternative plan. This turns the prompt's cost-awareness from a suggestion into a hard constraint.

The execution loop should follow a strict sequence: Plan → Validate → Execute → Observe → Repeat. After receiving the model's output, parse the proposed tool calls. Validate each one against the current ToolBudget. For calls that pass validation, execute them and capture the full response, including latency and any error codes. Append the results to the message history as structured tool messages. If any call fails with a retryable error (e.g., 429 rate limit, transient server error), do not immediately retry. Instead, surface the error to the model in the next turn and let the prompt's few-shot error-handling examples decide whether to retry with backoff, switch tools, or escalate. This respects the prompt's own decision logic for error recovery. Implement a maximum loop count (e.g., 5 agent turns) to prevent infinite planning spirals, and log every budget check, tool execution, and model re-prompt for observability.

For evaluation, build a test harness that replays recorded tool response sequences against the prompt. Your eval suite should check three specific failure modes this prompt is designed to prevent: budget-exceeding sequences (the model proposes a plan that would exceed cost or rate limits if executed unchecked), missed batching opportunities (the model makes N individual calls when a single batched call was available and within limits), and unnecessary calls (the model calls a tool when the required data already exists in the conversation context). Each test case should provide a mock ToolBudget state and a set of available tool definitions. The eval passes if the harness blocks all budget violations and the model's final plan uses no more than the available budget while achieving the stated goal. Run these evals in CI before any prompt or tool definition change.