Cost-vs-Accuracy Tradeoff Decision Prompt

This prompt is for infrastructure engineers who need to make an explicit, documented decision about whether to route a request to a high-accuracy (and higher-cost) model or a lower-accuracy (and lower-cost) model. It is not a generic model selector. Use it when the tradeoff between cost and quality is the primary decision dimension, and when you need a rationale that can be logged, audited, and tuned over time. The prompt assumes you have already classified the request and now face a binary or tiered routing choice. It does not handle multi-model consensus, latency-only routing, or hard budget enforcement with no quality consideration.

The ideal user is an AI infrastructure engineer or platform developer building a model routing middleware layer. They have already implemented intent classification upstream and now need a deterministic, explainable decision point that weighs accuracy requirements against cost constraints. The prompt expects structured inputs: a request complexity assessment, a defined accuracy threshold, a cost budget or relative cost weight, and a catalog of available models with their cost and capability profiles. It produces a routing decision with a structured rationale, not a free-text recommendation. This makes the output suitable for logging systems, audit trails, and downstream dispatch logic.

Do not use this prompt when the primary constraint is latency rather than cost, when you need multi-model consensus or voting logic, or when you are enforcing a hard dollar cap with zero tolerance for overage. Those scenarios require different prompt architectures with latency SLA variables, consensus aggregation logic, or hard-stop enforcement rules. Also avoid this prompt for user-facing chatbot routing where tone, personality, or brand safety are the dominant factors—cost-accuracy tradeoffs are infrastructure decisions, not user-experience decisions. If you need to route based on customer tier entitlements or contractual SLAs, use a tier-respecting routing prompt that incorporates account metadata before cost optimization.

Before deploying this prompt, ensure you have instrumented your routing path with cost tracking and accuracy measurement. The prompt's value comes from the feedback loop: logged decisions let you tune tradeoff weights, identify over-escalation patterns, and justify infrastructure spend to stakeholders. Start by running the prompt in shadow mode against a labeled dataset where you know the ground-truth accuracy requirements. Compare its routing decisions against a simple cost-minimizing baseline and a quality-maximizing baseline to calibrate the tradeoff weight parameter. Only promote to production routing after you've established that the prompt's decisions align with your organization's actual risk tolerance and budget constraints.

This prompt template is designed to be pasted into your routing middleware. It forces the model to make an explicit, documented tradeoff between cost and accuracy for each request or request class. The output includes the selected routing target, the rationale, and any override conditions triggered. Replace every square-bracket placeholder with runtime values from your application context before invoking the model.

textCopy
SYSTEM:
You are a cost-aware request router. Your job is to decide which model or processing tier should handle a given request. You must balance accuracy requirements against a strict cost budget. Always prefer the cheapest option that meets the minimum accuracy threshold, unless an override condition applies.

INPUT:
[INPUT]

REQUEST CONTEXT:
- User Tier: [USER_TIER]
- Request Type: [REQUEST_TYPE]
- Estimated Input Complexity: [COMPLEXITY_SCORE]/10

ROUTING OPTIONS (ordered by cost, lowest first):
[ROUTING_OPTIONS]

CONSTRAINTS:
- Hard Cost Cap Per Request: [COST_CAP]
- Minimum Acceptable Accuracy Score: [MIN_ACCURACY]
- Latency Budget (ms): [LATENCY_BUDGET]

OVERRIDE CONDITIONS:
[OVERRIDE_CONDITIONS]

OUTPUT_SCHEMA:
{
  "decision": {
    "selected_option": "string (must match an option ID from ROUTING_OPTIONS)",
    "estimated_cost": "number (must be <= COST_CAP)",
    "estimated_accuracy": "number (0-1, must be >= MIN_ACCURACY)",
    "estimated_latency_ms": "number (must be <= LATENCY_BUDGET)"
  },
  "rationale": {
    "tradeoff_summary": "string (explain why this option was chosen over cheaper and more expensive alternatives)",
    "override_triggered": "boolean",
    "override_detail": "string or null (explain which override condition fired and why)"
  },
  "alternatives_considered": [
    {
      "option_id": "string",
      "rejection_reason": "string (why this option was not selected)"
    }
  ]
}

INSTRUCTIONS:
1. Evaluate the INPUT against the ROUTING OPTIONS.
2. Select the cheapest option that meets MIN_ACCURACY and LATENCY_BUDGET.
3. If an OVERRIDE CONDITION is met, select the option specified by the override, even if it exceeds the cost cap. Document this in rationale.override_triggered.
4. If no option meets all constraints, select the option that comes closest and set rationale.tradeoff_summary to explain the violation.
5. Output ONLY valid JSON matching OUTPUT_SCHEMA. No markdown, no commentary.

Adaptation guidance: The ROUTING_OPTIONS placeholder should be replaced with a structured list of your available models or processing tiers, each with an ID, cost profile, accuracy benchmark, and latency characteristic. The OVERRIDE_CONDITIONS field is critical for high-stakes requests—define explicit rules such as "if USER_TIER is 'enterprise' and REQUEST_TYPE is 'legal', route to premium model regardless of cost." The COMPLEXITY_SCORE should be populated by a lightweight pre-classifier or heuristic before this prompt runs. Always validate the output JSON against the schema before acting on the routing decision. For production systems, log every decision including the rationale and alternatives considered for auditability and cost attribution.

This prompt is designed to sit inside a model routing layer—typically middleware that intercepts a request before it reaches a generative model. The harness should call this prompt when the system needs an explicit, auditable tradeoff decision rather than a static routing rule. The prompt expects structured inputs: a request summary, accuracy requirements, cost constraints, tradeoff weights, and any override conditions. The output is a routing decision with a documented rationale, which downstream systems use to select a model, tool, or workflow path.

Input assembly is the first integration point. Before invoking the prompt, the harness must gather and normalize the required variables. The [REQUEST_SUMMARY] should be a concise description of the user's task, extracted from the original input or generated by a lightweight classifier. [ACCURACY_REQUIREMENT] should be a structured field—such as HIGH, MEDIUM, or LOW—derived from request context (e.g., a medical query maps to HIGH, a casual chat maps to LOW). [COST_CONSTRAINT] should be a numeric value in dollars or tokens, pulled from a per-request budget or a user-tier configuration. [TRADEOFF_WEIGHTS] should be a JSON object like {"accuracy": 0.7, "cost": 0.3} that reflects business priorities. The harness must validate all inputs before calling the prompt: missing or malformed fields should trigger a fallback routing rule, not a prompt invocation with empty placeholders.

Output validation and routing dispatch are critical. The prompt returns a JSON object with decision, rationale, selected_model_tier, estimated_cost, and override_applied. The harness must parse this output and validate it against a schema before acting on it. If the estimated_cost exceeds the [COST_CONSTRAINT], the harness should reject the decision and fall back to the cheapest available model. If an override_applied flag is true, the harness should log the override reason and route accordingly. The selected_model_tier field should map to an actual model endpoint in your infrastructure (e.g., premium → gpt-4o, standard → gpt-4o-mini, budget → claude-3-haiku). Retry logic: if the prompt returns malformed JSON or a decision that fails validation, retry once with a simplified prompt that removes tradeoff weights and forces the cheapest safe option. If the retry also fails, route to the budget model and log the failure for review.

Observability and cost tracking must be built into the harness. Log every tradeoff decision with the request ID, input variables, prompt output, final routed model, actual cost incurred, and whether an override was triggered. This creates an audit trail for tuning tradeoff weights and detecting routing drift. Set up a dashboard that tracks the percentage of requests routed to each tier, the average cost per request, and the override rate. If overrides exceed 10% of traffic, the static weights likely need adjustment. Human review is not required for every decision, but the harness should flag decisions where the prompt's confidence appears low (e.g., rationale contains hedging language like "unclear" or "borderline") or where the cost estimate is within 10% of the budget cap. These flagged decisions can be sampled for offline review to improve the prompt or the input assembly logic.

Model choice for the router itself matters. This prompt should run on a fast, cheap model (e.g., gpt-4o-mini or claude-3-haiku) because it is a meta-decision that adds latency and cost before the actual work begins. Avoid using a premium model for routing unless the cost of a misroute is demonstrably higher than the router's own cost. Test the router prompt with your chosen model under load to ensure p95 latency stays under 500ms. If latency spikes, consider caching routing decisions for identical or highly similar [REQUEST_SUMMARY] values with a short TTL (e.g., 60 seconds). What to avoid: do not use this prompt for every single request in a high-throughput system without caching; do not skip output validation—an unvalidated routing decision can silently send sensitive requests to the wrong model; and do not hardcode model names in the prompt—use tier labels that map to a configuration file, so model swaps don't require prompt changes.