Cost-Optimized Model Selection Prompt

This prompt is for infrastructure teams managing token budgets across model tiers. It produces a cost-aware routing decision that maps task complexity to model capability within defined budget constraints. Use it when you operate multiple model endpoints at different price points and need a deterministic, auditable way to select the cheapest model that can handle the task reliably. The ideal user is an AI infrastructure engineer or platform developer who has already instrumented their application with a model router and needs a classification step that makes a structured decision before any downstream model invocation occurs. You must have a defined model tier list with known capability profiles and per-token costs; this prompt does not discover or recommend models.

This is not a general-purpose model recommendation prompt. Do not use it when you lack concrete cost data, when your model tiers have overlapping or undefined capabilities, or when the task requires subjective quality judgments that your tier definitions cannot capture. The prompt assumes you have already profiled your models against representative workloads and can express their strengths as discrete capability classes (e.g., 'handles complex reasoning,' 'handles simple extraction,' 'handles creative generation'). It also assumes you have a budget policy—whether a hard per-request cost ceiling, a daily token budget, or a cost-priority weighting—that can be expressed as a constraint. Without these inputs, the prompt will produce unreliable routing decisions that may appear reasonable but fail under production load.

The prompt works as a classification step inside a model router. It receives a task description and budget parameters, then returns a structured routing decision before any downstream model invocation occurs. This means you should wire it into your request pipeline after authentication and input validation but before the model call. The output should be treated as a machine-readable routing instruction, not a user-facing explanation. Validate the output against your actual model registry before executing the route; if the prompt selects a model that doesn't exist or exceeds budget, your router should fall back to a safe default and log the discrepancy for review. Common failure modes include over-classifying simple tasks to expensive models out of caution, under-classifying complex tasks to cheap models that produce unusable output, and failing to account for input token costs when calculating total request cost.

This prompt template is designed to be dropped directly into your model router or classification middleware. It forces the routing model to consider three dimensions simultaneously: task complexity, required capability tier, and token budget. The output is a structured decision that your application can parse and execute without additional interpretation. Replace each square-bracket placeholder with your actual values before sending to the routing model.

textCopy
You are a cost-aware model router. Your job is to select the most cost-effective model that can reliably complete the task within the stated constraints.

## TASK TO ROUTE
[USER_INPUT]

## AVAILABLE MODELS
[MODEL_CATALOG]

## TOKEN BUDGET
Maximum budget: [MAX_TOKENS] tokens
Cost per 1K tokens for each model: [MODEL_PRICING]

## TASK COMPLEXITY INDICATORS
- Requires reasoning depth: [REASONING_DEPTH low/medium/high]
- Requires external tool use: [TOOLS_REQUIRED true/false]
- Requires structured output: [STRUCTURED_OUTPUT true/false]
- Domain specificity: [DOMAIN_SPECIFICITY general/domain/niche]
- Latency tolerance: [LATENCY_TOLERANCE real-time/near-real-time/batch]

## ROUTING RULES
1. Select the cheapest model that meets all task requirements.
2. If no model fits within budget, select the closest fit and flag the budget overrun.
3. If task complexity is ambiguous, default to the safer (more capable) model and flag for review.
4. Never route PII or regulated data to models outside approved zones.

## OUTPUT FORMAT
Return a JSON object with these exact fields:
{
  "selected_model": "model_id",
  "confidence": 0.0-1.0,
  "estimated_tokens": integer,
  "estimated_cost": "$X.XXXX",
  "budget_compliant": true/false,
  "budget_overrun_amount": "$X.XXXX or null",
  "fallback_model": "model_id or null",
  "routing_reasoning": "Brief explanation of the decision",
  "flags": ["budget_overrun", "complexity_ambiguous", "requires_review"]
}

To adapt this template, start by populating [MODEL_CATALOG] with your actual model inventory. Include model IDs, capability descriptions, context window sizes, and per-token pricing. The [MODEL_PRICING] field should mirror your current rate card. If you use dynamic pricing or spot instances, update this field at runtime from your cost API. The complexity indicators should be pre-computed by an upstream classification step or set to sensible defaults based on your workload profile. If you lack a pre-classifier, remove the complexity indicators and let the routing model infer complexity from [USER_INPUT] alone, but expect higher variance in routing decisions.

Before deploying, run this prompt through your eval harness with at least 50 labeled examples covering: obvious cheap-model cases, obvious expensive-model cases, edge cases near budget boundaries, and inputs that should trigger the requires_review flag. Measure precision and recall on budget-compliant routing decisions. If the router consistently over-selects expensive models, tighten the routing rules or add few-shot examples showing correct cheap-model selections. If it under-selects and produces failures, loosen the safety default in rule 3 or increase the budget. Wire the output JSON into your application's model dispatch layer, and log every routing decision with the actual cost incurred so you can detect drift between estimated and actual token consumption.

The Cost-Optimized Model Selection Prompt is designed to sit inside a model routing middleware—a thin service that receives a task payload, invokes the prompt against a fast, cheap classifier model, and uses the structured output to dispatch the actual work to the appropriate model tier. This harness is not a standalone chatbot. It is a decision function that must return a deterministic, validatable routing decision within a strict latency budget. The primary integration point is a select_model() function that wraps the LLM call, parses the JSON response, validates the decision against current budget state and rate limits, and returns a model identifier plus reasoning trace to the upstream orchestrator.

Validation and state checks are the most critical part of this harness. Before accepting the LLM's routing decision, validate that: (1) the selected model exists in the current inventory and is not in a degraded state; (2) the estimated token cost of the selected model does not exceed the remaining budget for the tenant, session, or time window; (3) the complexity score is internally consistent with the model tier (a complexity: high task routed to a lightweight model should trigger a review log entry); and (4) the output schema is parseable with all required fields present. Implement a validate_routing_decision() function that checks these invariants and returns either an approved decision or a fallback override. If validation fails, fall back to a conservative default model (e.g., the cheapest capable model) and increment a budget_override_count metric. Log every override with the original LLM decision, the validation failure reason, and the fallback applied.

Retry logic should be minimal and bounded. The classifier model call should have a single retry on timeout or malformed JSON, with a short backoff (100-300ms). Do not retry on budget-exceeded decisions—those are policy violations, not transient errors. If the classifier model returns an unparseable response after one retry, route to the default model and emit a routing_parse_failure alert. Observability requires logging: the input task summary (not full content, to control log costs), the raw LLM routing decision, the validated decision, the current budget state, the selected model, and the latency of the routing call itself. These logs feed into cost dashboards that track budget utilization, unnecessary expensive-model usage, and routing decision accuracy over time.

Model choice for the router itself matters. Use the cheapest fast model that reliably produces valid JSON for this task—typically a lightweight model like GPT-4o-mini, Claude Haiku, or Gemini Flash. The router model should cost less than 1% of the average downstream task cost to justify its existence. If the router model's cost approaches 5% of downstream costs, the routing logic is too expensive and should be replaced with a rule-based classifier or cached decisions for repetitive task patterns. Budget state must be passed into the prompt's [BUDGET_CONTEXT] placeholder as a structured object containing: remaining budget, budget period, current spend rate, and any hard caps. Do not rely on the LLM to fetch or calculate budget state—provide it explicitly from your application's budget tracker.

Testing this harness requires a suite of task payloads with known complexity levels and budget scenarios. Test cases should include: tasks clearly appropriate for cheap models, tasks clearly requiring expensive models, borderline tasks where either choice is defensible, and budget-exhausted scenarios where the only valid decision is the cheapest model or a rejection. Measure routing accuracy against human-labeled complexity assessments, but also measure cost efficiency: what percentage of tasks are routed to the cheapest model that can handle them? A router that always selects the most expensive model is failing its core purpose, even if each individual decision seems defensible. Run these tests in CI before deploying prompt changes, and gate deployments on cost-efficiency regression thresholds.

What to avoid: Do not use this prompt for real-time streaming decisions where the routing latency budget is under 200ms—the LLM call overhead will violate that constraint. Do not pass raw user content into the routing prompt if it contains PII; extract a task summary or complexity signal first. Do not let the router model select itself for downstream work. And do not treat the complexity score as a quality metric for anything other than routing—it is a cost-control signal, not an evaluation of output correctness. The next step after implementing this harness is to deploy a shadow mode where routing decisions are logged but not enforced, letting you measure cost savings potential before cutting over production traffic.