Cost-Budget-Aware Dispatch Prompt Template

This prompt is designed for infrastructure engineers and platform architects who manage AI request routing across multiple processing tiers with different cost profiles. The core job-to-be-done is making a dispatch decision that minimizes cost while respecting a defined budget and a minimum quality threshold. You should use this prompt when you have a classified request ready for processing, a set of available pipelines each with a known cost (per request, per token, or per compute unit), and a budget context that could be per-request, per-session, or per-tenant. The prompt assumes that an upstream classification step has already identified the request type and that downstream processing pipelines are operational and ready to receive the dispatch decision.

This prompt is not a universal router. Do not use it when cost is irrelevant to the routing decision, when only a single processing path exists, or when latency is the sole or overriding criterion. It is also unsuitable for real-time systems where the cost of the routing decision itself must be near-zero, as the prompt requires a non-trivial model call. The prompt operates on the assumption that cost optimization is the primary objective and that quality thresholds are defined as minimum acceptable standards, not aspirational targets. If your system requires complex multi-objective optimization balancing cost, latency, and quality simultaneously, you will need a more sophisticated routing layer that may combine this prompt with other decision logic.

Before implementing this prompt, ensure you have a clear mapping of your processing options, their costs, and their quality characteristics. The prompt template includes placeholders for [PROCESSING_OPTIONS], [BUDGET_CONTEXT], and [MINIMUM_QUALITY_THRESHOLD] that must be populated with concrete, machine-readable values. The output is a structured dispatch decision that includes the selected pipeline, the expected cost, remaining budget, and a justification. This output is designed to be consumed by downstream orchestration code, not displayed directly to end users. For high-stakes financial decisions or regulated environments, always include a human review step before executing the dispatch decision, and log every routing choice with its full context for audit purposes.

This prompt template is designed to be pasted directly into your system instructions for a cost-budget-aware dispatch agent. It forces the model to reason about the financial cost of downstream processing before making a routing decision. The template uses square-bracket placeholders that you must replace with your specific application values, such as your model pricing tiers, budget thresholds, and queue identifiers. The core logic balances the estimated complexity of the request against a pre-defined cost budget, ensuring that simple queries are not wastefully routed to expensive, high-capability models.

textCopy
You are a Cost-Budget-Aware Dispatch Router. Your only job is to analyze an incoming user request and route it to the most cost-effective processing queue that can handle it, while respecting a strict cost budget.

# AVAILABLE QUEUES AND COST TIERS
[QUEUE_DEFINITIONS]
# Example:
# - queue_id: "fast-cpu-v1", cost_per_request_usd: 0.001, capability: "simple_classification_and_retrieval"
# - queue_id: "haiku-v1", cost_per_request_usd: 0.01, capability: "complex_reasoning_and_summarization"
# - queue_id: "opus-v1", cost_per_request_usd: 0.10, capability: "multi_step_agentic_workflow"

# BUDGET CONSTRAINTS
[BUDGET_RULES]
# Example:
# - user_tier: "free", max_cost_per_request_usd: 0.005
# - user_tier: "pro", max_cost_per_request_usd: 0.05
# - user_tier: "enterprise", max_cost_per_request_usd: 0.50

# ROUTING LOGIC
1. Analyze the user's request to determine the minimum capability required to fulfill it successfully.
2. Identify the user's tier from the provided [USER_CONTEXT].
3. Select the cheapest queue from [QUEUE_DEFINITIONS] that meets the capability requirement and whose `cost_per_request_usd` is less than or equal to the user's `max_cost_per_request_usd`.
4. If no queue meets both the capability and budget requirements, route to the [FALLBACK_QUEUE_ID] and set a flag `budget_exceeded: true`.

# INPUTS
[USER_CONTEXT]
[USER_REQUEST]

# OUTPUT SCHEMA
You must output a single, valid JSON object with no other text.
{
  "decision": {
    "queue_id": "string",
    "reasoning": "string",
    "estimated_cost_usd": "number",
    "budget_exceeded": "boolean"
  }
}

To adapt this template, start by defining your [QUEUE_DEFINITIONS]. This is the most critical block; it maps your actual infrastructure (serverless endpoints, model APIs, or internal microservices) to a cost and a plain-language capability description. The capability description is what the LLM uses for reasoning, so use clear, comparative terms like 'simple retrieval' versus 'multi-hop agentic reasoning'. Next, define your [BUDGET_RULES] by mapping user tiers or request types to a maximum cost. The [USER_CONTEXT] placeholder should be populated at runtime with a JSON object containing the user's tier and any other relevant attributes. Finally, ensure your application layer validates the output JSON against the schema before acting on the queue_id. A common failure mode is the model hallucinating a queue name; your application should only accept queue_id values that exactly match your defined set.

This prompt is designed to be a decision node in a programmatic dispatch system, not a standalone chatbot. The application layer should call the model with this prompt, parse the structured output, and then use the routing_decision field to enqueue the task in the correct processing pipeline. The model's output is advisory; the actual enforcement of cost budgets and tier assignments must happen in the application code that reads this decision. Treat the model as a classifier that produces a cost-optimized suggestion, while the surrounding harness owns the budget ledger, tier definitions, and hard limits.

To integrate this prompt, first define your cost tiers and their associated model/pipeline identifiers in a configuration store (e.g., TIER_CONFIG = { 'low_cost': {'max_budget': 0.01, 'model': 'gpt-3.5-turbo', 'queue': 'batch-low-priority'}, ... }). When a request arrives, the application should inject the current budget state and tier definitions into the prompt's [BUDGET_CONTEXT] and [TIER_DEFINITIONS] placeholders. After receiving the model's JSON response, validate the assigned_tier against your known tier list and confirm that the estimated_cost does not exceed the remaining_budget. If validation fails, log the mismatch as a cost_tier_misassignment metric and fall back to a safe default tier (typically the lowest-cost option) or escalate to a human review queue. Implement a retry policy with a maximum of 2 attempts for malformed JSON or schema validation errors, using a repair prompt that includes the validation error message.

For production observability, instrument every dispatch decision with structured logs that capture: request_id, assigned_tier, estimated_cost, remaining_budget, model_used_for_routing, and validation_passed. Set up alerts for budget overrun events where estimated_cost > remaining_budget and for tier misassignment rates exceeding 5% of traffic. The cost budget itself should be managed by a rate limiter or token bucket in the application layer that decrements actual inference costs post-execution, not by the prompt's estimates alone. Never trust the model's estimated_cost for financial reconciliation; use it only for routing decisions and compare it against actual measured costs to detect drift in the model's cost estimation accuracy over time.