Cost-Budget-Aware Router System Prompt Template

This prompt is a system-level instruction set for infrastructure engineers building multi-model routing middleware. Its job is not to answer the user. Its job is to analyze an incoming request, evaluate available models against a hard cost budget, and output a deterministic routing decision that maximizes expected output quality without exceeding the budget. Use this when you operate multiple models with different pricing tiers and you need every routing decision to be cost-auditable—for example, when your platform serves different customer tiers, when you have a daily or per-request spend cap, or when you need to prove to finance that no single request blew through an unreasonable amount of compute spend.

The ideal user is an AI infrastructure engineer or platform engineer who already has a model catalog with per-token pricing, a defined cost cap variable, and a fallback chain. You should have a clear understanding of your models' relative quality on your specific task distribution—this prompt does not discover quality rankings; it applies them. The required context includes: a cost budget (in dollars or tokens), a model catalog with pricing and capability descriptions, the user's request, and any quality weights or preferences. Without this context, the router cannot make a cost-informed decision and will either default to the cheapest model (wasting the budget) or the most expensive (breaching the cap).

Do not use this prompt when cost is irrelevant, when only one model is available, or when the routing decision requires real-time latency data not available in the prompt context. It is also the wrong tool when you need to route based on content policy violations, user entitlement tiers, or multi-modal input type detection—those are separate classification problems that should happen before cost-aware routing. If your routing logic requires sub-millisecond decisions, this prompt-based approach adds unacceptable latency; instead, pre-compute routing tables or use a cached classifier. Finally, if your cost cap is so low that no model can produce a useful response, this prompt will correctly refuse to route—but you need a separate escalation path for that case, not a prompt tweak.

Before deploying this prompt, build a small eval set of requests with known cost constraints and verify that the router never exceeds the budget, selects the highest-quality model that fits, and produces auditable reasoning. Common failure modes include: the router ignoring the budget when the user request implies urgency, the router over-prioritizing quality weights and selecting an over-budget model, and the router failing to consider the fallback chain when the preferred model is unavailable. Test each of these explicitly. Once validated, wire the router's output into your actual model dispatch code—the prompt produces a decision, but your application enforces it and logs the audit trail.

The following template is designed to be set as the system message in a routing middleware service. Its job is to analyze an incoming user request, consult a provided model catalog with per-token pricing, and select the most capable model that can handle the task without exceeding a defined cost budget. The prompt forces a structured decision, requires explicit justification, and mandates a fallback chain if the primary choice is unavailable or too expensive. This ensures that cost constraints are enforced at the routing layer, not just monitored after the fact.

textCopy
You are a cost-budget-aware model router. Your only job is to select the best model for a given user request while strictly adhering to a hard cost budget.

## INPUT
[USER_REQUEST]

## MODEL CATALOG
[MODEL_CATALOG_JSON]

## CONSTRAINTS
- Hard Cost Budget: [MAX_COST_PER_REQUEST]
- Latency Target (p95): [MAX_LATENCY_MS]
- Required Capabilities: [REQUIRED_CAPABILITIES]
- Fallback Chain: [FALLBACK_CHAIN]

## INSTRUCTIONS
1. Analyze the complexity and requirements of the [USER_REQUEST].
2. Review the [MODEL_CATALOG_JSON] for capability, pricing, and latency profiles.
3. Select the most capable model that fits within the [MAX_COST_PER_REQUEST] budget and meets the [MAX_LATENCY_MS] target.
4. If no model in the primary tier fits, step down the [FALLBACK_CHAIN] in order until a viable model is found.
5. If no model in the entire chain is viable, select the cheapest available model and set a critical flag.

## OUTPUT_SCHEMA
Return a single, valid JSON object with no other text:
{
  "selected_model": "string",
  "estimated_cost": "number",
  "estimated_tokens": "number",
  "justification": "string",
  "fallback_used": "boolean",
  "critical_budget_breach": "boolean"
}

## RISK_LEVEL
[RISK_LEVEL]

To adapt this template, replace the square-bracket placeholders with concrete values from your infrastructure. [MODEL_CATALOG_JSON] should be a JSON array of model objects, each containing an id, cost_per_1k_input_tokens, cost_per_1k_output_tokens, capabilities array, and p95_latency_ms. [FALLBACK_CHAIN] is an ordered list of model IDs to try if the primary selection fails cost or latency checks. For high-stakes applications, set [RISK_LEVEL] to high to instruct the router to prefer safety and accuracy over cost savings, and always log the justification field for audit trails. The next step is to wire this prompt into your request pipeline and validate its output against your budget enforcement logic.

This prompt is designed to be called synchronously inside a model gateway or routing middleware before any downstream model invocation. The application layer must supply the [MODEL_CATALOG], [HARD_BUDGET_CAP], [USER_INPUT], and any optional [CONTEXT] or [CONSTRAINTS] at request time. The router prompt itself should target a fast, cheap model (e.g., GPT-4o-mini, Claude Haiku, or a fine-tuned small classifier) because the routing decision must cost orders of magnitude less than the downstream model call it is protecting. If the router costs more than 2-3% of the budget it is enforcing, the economics break down.

The harness must enforce the budget decision, not just suggest it. After parsing the router's JSON output, validate the selected_model field against the [MODEL_CATALOG] to prevent hallucinated model names. Then check estimated_cost against the [HARD_BUDGET_CAP]. If the estimated cost exceeds the cap, the harness must override the selection with the cheapest viable model or return a budget-exceeded error to the caller. Implement a circuit breaker: if the router returns invalid JSON or a model not in the catalog after [MAX_RETRIES] (default 2), fall back to the cheapest model in the catalog and log the failure. Log every routing decision with request_id, selected_model, estimated_cost, budget_remaining, and override_applied for cost attribution and debugging.

For high-throughput systems, cache routing decisions when the same [USER_INPUT] and [MODEL_CATALOG] combination repeats. Use a TTL-based cache keyed on a hash of the input and catalog version. Invalidate the cache when the catalog pricing changes. If the router is part of a multi-step agent pipeline, deduct the estimated cost from a session-level budget tracker before each step and halt the pipeline when the cumulative spend reaches the cap. Never allow the router's own token cost to be excluded from the budget calculation—include it in the total spend tracking. For regulated or high-stakes domains, add a human approval gate when the router selects a model below a quality threshold or when the budget remaining drops below 10% of the cap.