Model Selection Routing Prompt for Cost-Optimized Batches

This prompt is designed for AI infrastructure engineers operating multi-model serving systems who need to classify batch items by complexity, required capability, and latency tolerance to route them to the most cost-effective model. The core job-to-be-done is avoiding the default pattern of sending every inference request to the largest, most expensive model in your fleet. Instead, this prompt acts as a pre-processing classifier that inspects each item and produces a structured routing decision—target model, confidence score, and cost tier—before the item reaches your inference layer. The ideal user is someone managing a heterogeneous model pool (e.g., a mix of lightweight open-weight models, mid-tier fine-tuned models, and frontier models) who needs deterministic, auditable routing logic that balances quality, speed, and token budget across thousands or millions of items.

Deploy this prompt when you have batch processing workloads with relaxed latency requirements—offline ETL pipelines, nightly document processing jobs, async evaluation runs, or near-real-time queues where 500ms–2s of classification overhead is acceptable. You need a defined model pool with known capability profiles and cost characteristics, plus a clear taxonomy of what 'complexity' means for your domain (e.g., reasoning depth, domain specificity, output structure requirements, context length). Do not use this prompt for sub-100ms real-time streaming where the classification step itself would violate latency SLOs; in those cases, use a lightweight heuristic classifier or pre-computed routing rules. Do not use it when you have only one model available or when all items genuinely require frontier-model quality—the prompt adds overhead without providing routing value. It is also inappropriate for workloads where routing decisions require understanding the full content of large documents or images that would need to be processed just to classify, negating the cost savings.

Before implementing this prompt, ensure you have instrumented your model pool with per-model cost tracking (tokens processed, compute time, dollar cost) and quality metrics (accuracy, factuality, task success rate) so you can validate that routing decisions actually reduce cost without degrading outcomes below acceptable thresholds. Start with a log-only deployment that records routing decisions alongside actual processing results for at least a week before enabling automated routing. This lets you measure routing accuracy, detect systematic over-routing or under-routing, and calibrate confidence thresholds before the prompt controls production traffic.

This prompt template is designed to be used as a system message in your model routing middleware. It instructs the routing model to classify each item in a batch by complexity, required capability, and latency tolerance, then assign it to the most cost-effective model from your catalog. The prompt enforces cost-awareness by requiring explicit reasoning about why a cheaper model is insufficient before escalating to a more expensive one. Before copying, ensure your application layer handles the input assembly: the batch items should be provided in the user message as a structured list, and the placeholders below must be replaced with your specific model catalog, cost constraints, and output schema.

textCopy
You are a cost-aware model router for a multi-model serving system. Your task is to analyze each item in a batch and assign it to the most cost-effective model that can handle the request correctly.

## AVAILABLE MODELS
[MODEL_CATALOG]

## COST CONSTRAINTS
- Budget per batch: [BUDGET_PER_BATCH]
- Max latency per item: [MAX_LATENCY_MS]ms
- Target cost tier preference: [COST_TIER_PREFERENCE]

## ROUTING RULES
1. Default to the cheapest model that meets the item's requirements.
2. Only escalate to a more expensive model when the item explicitly requires capabilities the cheaper model lacks.
3. For each routing decision, provide a one-sentence justification explaining why the selected model is appropriate.
4. If no model in the catalog can handle an item, flag it as [UNROUTABLE_FLAG] with the reason.
5. Batch items that are simple, low-risk, or high-latency-tolerance should never be routed to premium models.

## OUTPUT FORMAT
Return a JSON object with a "routing_decisions" array. Each element must follow this schema:
{
  "item_id": "string",
  "selected_model": "string",
  "confidence": "high|medium|low",
  "justification": "string",
  "estimated_tokens": number,
  "fallback_model": "string|null"
}

## CONSTRAINTS
- Do not route items to models that lack required modalities or language support.
- If an item is ambiguous, assign it to a mid-tier model and set confidence to "low".
- Never exceed [MAX_LATENCY_MS]ms estimated latency for any routing decision.
- Flag items containing [SENSITIVE_DATA_PATTERNS] for human review before routing.

After pasting this template, replace the square-bracket placeholders with your production values. [MODEL_CATALOG] should be a structured list of available models with their capabilities, cost per token, and latency profiles. [BUDGET_PER_BATCH] and [MAX_LATENCY_MS] should reflect your actual operational constraints. [COST_TIER_PREFERENCE] can be set to cheapest-first, balanced, or performance-first depending on your workload. [UNROUTABLE_FLAG] should be a string like UNROUTABLE that your harness can detect and escalate. [SENSITIVE_DATA_PATTERNS] should describe patterns that require human review, such as PII, credentials, or regulated data. The user message should contain the batch items as a JSON array with item_id, content, and any metadata your routing logic needs. Your application harness should parse the response, validate each routing decision against your model catalog, and log any UNROUTABLE flags or low-confidence assignments for review.

Before deploying, test this prompt with a golden dataset containing items of known complexity. Verify that simple items are consistently routed to cheaper models and that premium models are only selected when justified. Monitor for cost overruns by tracking the ratio of premium-to-budget model assignments in production. If you observe premium model usage exceeding expectations, review the justifications to identify whether the prompt is over-escalating or whether your cheap models genuinely lack required capabilities. The next section covers how to wire this prompt into an application harness with validation, retries, and cost monitoring.

The routing prompt is a single decision point inside a larger batch processing harness. Its job is to classify each item and return a model target and cost tier. The harness is responsible for everything around that decision: splitting batches, enriching items with context, enforcing rate limits, dispatching to the selected model, tracking actual cost against the budget, and recovering when the router or downstream model fails. Treat the prompt as a pure function that takes a pre-processed item and returns a structured routing decision. The harness owns all side effects, state, and retry logic.

A production harness typically follows this sequence: (1) Batch ingestion reads items from a queue, stream, or object store. (2) Pre-processing attaches metadata each item needs for routing—item size in tokens, client tier, latency budget, and any priority flags. (3) Routing dispatch sends each pre-processed item to the routing prompt with a strict output schema ({"model_target": "<model_id>", "cost_tier": "low|medium|high", "reasoning": "<brief>"}). (4) Validation checks that the returned model_target is in the allowed model registry and that cost_tier is a recognized value. Invalid outputs go to a dead-letter queue with the raw response and item ID. (5) Cost-aware dispatch routes the item to the selected model, recording the estimated cost tier against the item. (6) Post-execution cost tracking compares actual token usage against the tier budget and flags items that exceeded their assigned tier by more than a configurable threshold (e.g., 20%). These overruns feed back into router evaluation and threshold tuning.

Failure recovery requires clear separation between router failures and downstream model failures. If the routing prompt itself fails—timeout, malformed JSON, or model unavailability—retry with exponential backoff up to a maximum of three attempts, then fall back to a default routing rule (e.g., route to the medium-cost model with a cost_tier: medium override and log the fallback). If the downstream model fails after correct routing, the harness should retry on the same model once, then escalate to the next cost tier up if the item's priority or SLA requires completion. All routing decisions, fallback activations, and cost overruns must be logged with the item ID, timestamp, and model version for later analysis. Avoid routing prompts that carry conversational state across items—each routing call should be stateless and idempotent to keep retry behavior predictable and batch reprocessing safe.