Tool Cost vs. Accuracy Trade-Off Decision Prompt

Use this prompt when your agent has access to multiple tools or model endpoints that can perform the same logical task but differ significantly in cost and expected accuracy. The primary job-to-be-done is producing a structured, auditable decision record that justifies whether invoking a more expensive tool is warranted by a measurable improvement in output quality. This is not a prompt for generic cost-cutting; it requires that you have already established a ground-truth comparison framework—such as an eval dataset or historical accuracy logs—that quantifies the accuracy delta between your cheap and expensive tool paths. Without that data, the prompt cannot produce a valid trade-off analysis and will generate speculative justifications that should not be trusted for production decisions.

The ideal user is an AI cost engineer or platform operator who controls tool routing logic and has access to cost telemetry and evaluation metrics. Before invoking this prompt, you must supply: a clear description of the task the agent is attempting, the cost profiles of the available tools (including per-call cost, latency, and rate limit impact), and the measured accuracy difference between those tools on similar tasks from your eval harness. The prompt expects these inputs in structured form—typically as part of the [TOOLS] and [ACCURACY_DATA] placeholders—so it can reason over concrete numbers rather than vague claims. Do not use this prompt for real-time, latency-critical routing decisions where the overhead of an LLM call would negate the cost savings you are trying to achieve. It is designed for offline or async decision logging, pre-execution planning, or post-hoc audit workflows where the analysis itself is part of the governance record.

This prompt is not a replacement for a cost-aware tool selection system prompt that operates at runtime. It is a decision analysis tool that produces a recommendation and rationale, not a routing engine. Pair it with your existing cost logging infrastructure so the trade-off decision can be traced back to the specific execution context that triggered it. Avoid using this prompt when the accuracy difference between tools is unknown, when the task is so trivial that any tool would suffice, or when the expensive tool is required for compliance reasons that override cost considerations. In those cases, the prompt will produce analysis that looks authoritative but rests on unsupported assumptions. If you are operating in a regulated domain where tool choice must be defensible to auditors, ensure the prompt's output is logged alongside the actual tool call outcome and any human review decisions, creating a complete evidence chain from analysis to action.

This prompt template is designed to be pasted directly into your agent harness. It forces the model to evaluate whether a more expensive tool call is justified by the expected improvement in output quality. The prompt requires explicit cost and accuracy estimates, a structured recommendation, and a fallback plan. Replace every square-bracket placeholder with actual values before execution. Do not leave any placeholder unresolved in production.

textCopy
You are a cost-aware decision agent. Your task is to evaluate whether calling a more expensive tool is justified by the expected accuracy gain for the current task.

## INPUT
Task description: [TASK_DESCRIPTION]
Current context and partial results: [CURRENT_CONTEXT]

## TOOL OPTIONS
Primary (expensive) tool:
- Name: [PRIMARY_TOOL_NAME]
- Estimated cost per call: [PRIMARY_TOOL_COST]
- Expected accuracy range: [PRIMARY_TOOL_ACCURACY_RANGE]
- Latency estimate: [PRIMARY_TOOL_LATENCY]

Fallback (cheaper) tool:
- Name: [FALLBACK_TOOL_NAME]
- Estimated cost per call: [FALLBACK_TOOL_COST]
- Expected accuracy range: [FALLBACK_TOOL_ACCURACY_RANGE]
- Latency estimate: [FALLBACK_TOOL_LATENCY]

## CONSTRAINTS
- Remaining budget for this task: [REMAINING_BUDGET]
- Minimum acceptable accuracy threshold: [MINIMUM_ACCURACY_THRESHOLD]
- Deadline or latency budget: [LATENCY_BUDGET]
- Risk level of incorrect output: [RISK_LEVEL] (low/medium/high/critical)

## OUTPUT_SCHEMA
Return a JSON object with these fields:
{
  "decision": "use_primary" | "use_fallback" | "defer_to_human",
  "confidence": 0.0-1.0,
  "cost_accuracy_ratio": "string describing the trade-off",
  "expected_cost": number,
  "expected_accuracy": "string with range and justification",
  "risk_assessment": "string describing what happens if accuracy is insufficient",
  "fallback_plan": "string describing what to do if the chosen tool fails",
  "budget_remaining_after_call": number,
  "rationale": "string explaining the decision in 2-3 sentences"
}

## DECISION RULES
1. If the primary tool's expected accuracy is below [MINIMUM_ACCURACY_THRESHOLD], prefer defer_to_human.
2. If the primary tool cost exceeds [REMAINING_BUDGET], prefer use_fallback unless the risk level is critical.
3. If the accuracy gain from the primary tool is less than 10% absolute improvement over the fallback, prefer use_fallback.
4. If the risk level is critical and the fallback accuracy is below [MINIMUM_ACCURACY_THRESHOLD], prefer defer_to_human.
5. If the primary tool latency exceeds [LATENCY_BUDGET], prefer use_fallback.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## INSTRUCTIONS
Analyze the trade-off. Be explicit about what accuracy gain is expected and whether it justifies the additional cost. If the decision is close, err toward the cheaper option unless the risk level is high or critical. Return ONLY the JSON object.

Adaptation notes: Replace [FEW_SHOT_EXAMPLES] with 2-3 concrete examples showing correct decisions across different risk levels and budget states. If your system does not have ground-truth accuracy data for tools, replace the accuracy ranges with proxy metrics such as historical success rate, user satisfaction scores, or benchmark performance. For high-risk domains, add a [REQUIRES_HUMAN_APPROVAL] boolean field to the output schema and route defer_to_human decisions to a review queue. Validate the JSON output against the schema before allowing the agent to proceed with the tool call. Log every decision with the rationale for post-hoc cost-audit analysis.

This prompt is designed to run as a decision node inside an agent loop or cost-governance middleware, not as a standalone chat. The typical integration point is immediately before an expensive tool call: the agent proposes a tool and arguments, the harness intercepts the proposal, invokes this prompt with the proposed call and available alternatives, and then routes based on the recommendation field. The prompt expects structured inputs ([PROPOSED_TOOL], [ALTERNATIVES], [TASK_CONTEXT], [BUDGET_REMAINING], [ACCURACY_REQUIREMENT]) and returns a JSON decision object. Do not use this prompt for real-time, sub-100ms decisions; it is designed for deliberative cost-accuracy trade-offs where the latency of an LLM call is acceptable relative to the cost of the tool invocation under review.

Validation and routing logic must sit in the application layer, not in the prompt. After the model returns JSON, validate that recommendation is one of proceed_with_proposed, use_alternative, or skip_tool. If use_alternative, confirm that the selected_alternative field references a tool that exists in the current tool registry and that its estimated_cost is less than budget_remaining. If validation fails, retry once with the validation error injected into [CONSTRAINTS]. After a second failure, default to skip_tool and log the incident. For high-stakes domains (finance, healthcare, legal), route any proceed_with_proposed decision where cost_exceeds_budget is true to a human approval queue before execution. The approval payload should include the full trade-off analysis JSON, the original task context, and a one-click approve/deny interface.

Model selection matters for this prompt. The trade-off analysis requires structured reasoning about costs, accuracy, and task priorities. Use a model with strong JSON mode and instruction-following (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller models that may collapse the trade-off into a simplistic "always use the cheaper tool" or "always use the most accurate tool" pattern. Logging should capture the full prompt input, the model's JSON output, the validation result, and the final routing decision. Attach a decision_id and trace_id to correlate this decision with the subsequent tool call and its outcome. This trace is essential for the ground-truth comparison step: after the task completes, compare the actual accuracy achieved against the prompt's expected_accuracy_gain to detect systematic over- or under-estimation. Feed these comparisons back into few-shot examples or fine-tuning data to improve future decisions.

Retry and fallback behavior should be handled outside the prompt. If the expensive tool call fails after a proceed_with_proposed decision, do not re-invoke the trade-off prompt; instead, fall back to the top-ranked alternative from the original analysis, or escalate to a human if no alternatives remain. If the prompt itself times out or returns unparseable JSON, treat it as a skip_tool decision and log a tradeoff_analysis_failed event. Cost tracking integration requires that the estimated_cost values in the prompt input come from a live cost registry, not hardcoded in the prompt. Wire the harness to fetch current per-call costs from your model gateway or tool provider API before assembling the prompt. Stale cost data produces trade-off decisions that look reasonable but waste money in production.