Cost vs Latency Routing Tradeoff Prompt

This prompt is designed for FinOps engineers, infrastructure SREs, and AI platform operators who need to audit whether a model router's production decisions are aligned with the configured cost or latency optimization objective. The core job-to-be-done is ingesting a batch of production trace segments—each containing the request context, the model selected, its pricing tier, and the observed latency—and scoring each routing decision against the declared policy. The output is a structured misrouting report that identifies decisions where a cheaper model could have met the latency budget or where a faster model was bypassed in favor of an unnecessarily expensive one, complete with an estimated financial impact.

You should use this prompt when you have access to structured trace data that includes per-request model identifiers, token counts, pricing metadata, and end-to-end latency measurements. The prompt assumes the routing objective (minimize cost, minimize latency, or a balanced constraint) is known and can be supplied as part of the input context. It is most effective when run over a statistically significant window of production traces—typically hundreds or thousands of requests—rather than isolated incidents, because the goal is to surface systematic routing policy drift or configuration errors, not one-off anomalies. The prompt works best with a pre-defined output schema that your application can parse and feed into a dashboard or alerting pipeline.

Do not use this prompt for real-time routing decisions; it is a post-hoc audit tool, not a live decision engine. It is also not a replacement for a router health check or a fallback root-cause analysis—those are separate playbooks in this content group. Avoid using this prompt when trace data is incomplete or missing pricing and latency fields, as the scoring logic will produce unreliable results. If you are operating in a high-stakes environment where misrouting has direct revenue or compliance impact, always pair the automated report with a human review step before taking corrective action on the router configuration. After reading this section, proceed to the prompt template to adapt it to your specific router policy and trace schema.

This prompt template is designed to ingest trace segments that include model pricing and latency data, then score each routing decision against the configured optimization objective (cost, latency, or balanced). The output is a structured misrouting report with estimated cost impact, making it directly usable in FinOps dashboards, automated alerting pipelines, or periodic routing health checks. Adapt the placeholders to match your trace schema, pricing model, and latency thresholds before running against production data.

textCopy
You are a routing auditor analyzing production trace segments to determine whether model routing decisions optimized for the declared objective.

## INPUT DATA
[TRACE_SEGMENTS]

## CONFIGURED OBJECTIVE
[OBJECTIVE: cost | latency | balanced]

## MODEL CATALOG
[MODEL_CATALOG: JSON array of available models with per-token pricing and expected latency ranges]

## ROUTING RULES
[ROUTING_RULES: the declared routing policy, including capability requirements, cost ceilings, and latency budgets]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "trace_id": "string",
  "request_summary": "one-line description of the request",
  "routed_model": "string",
  "routing_objective": "cost | latency | balanced",
  "decision_score": "optimal | acceptable | suboptimal | misrouted",
  "score_reasoning": "explanation referencing specific pricing, latency, and capability data",
  "alternative_model": "the model that would have been optimal under the objective",
  "cost_impact_estimate": {
    "actual_cost": "number",
    "optimal_cost": "number",
    "delta": "number",
    "currency": "USD"
  },
  "latency_impact_estimate": {
    "actual_latency_ms": "number",
    "optimal_latency_ms": "number",
    "delta_ms": "number"
  },
  "capability_gaps": ["list of missing capabilities if the routed model couldn't fulfill the request"],
  "rule_compliance": "compliant | violation | no_applicable_rule",
  "violated_rule": "string or null"
}

## CONSTRAINTS
- Score as 'misrouted' only when a clearly better model was available under the objective and routing rules.
- Score as 'acceptable' when the routed model is not optimal but the delta is within 10% of optimal cost or latency.
- If capability requirements rule out the cheaper or faster model, do not flag as misrouted.
- Flag missing capability gaps explicitly rather than assuming the router made an error.
- If trace data is incomplete, note what's missing in score_reasoning and score as 'acceptable' with a caveat.
- Do not invent pricing or latency data. Use only the MODEL_CATALOG provided.

## EVALUATION STEPS
1. Extract the request characteristics from the trace segment.
2. Identify which models in the catalog satisfy capability requirements.
3. From the eligible models, determine which one is optimal under the declared objective.
4. Compare the routed model to the optimal model.
5. Check whether the routing decision complied with declared ROUTING_RULES.
6. Compute cost and latency deltas.
7. Assign a decision score with reasoning.

## EXAMPLES
[EXAMPLES: 2-3 annotated trace segments showing optimal, acceptable, and misrouted decisions]

## RISK LEVEL
[RISK_LEVEL: low | medium | high]
- Low: advisory analysis, no automated action taken.
- Medium: results feed into a dashboard for human review.
- High: results may trigger automated fallback reconfiguration. Require human approval before acting on high-risk outputs.

To adapt this template, replace [TRACE_SEGMENTS] with your actual trace data in a structured format that includes the request payload, the model selected, observed latency, token counts, and any error or fallback metadata. Populate [MODEL_CATALOG] from your live pricing API or a configuration file that stays in sync with your model gateway. Set [OBJECTIVE] based on the workload's declared optimization target—do not infer it from the trace itself, as that creates circular logic. The [EXAMPLES] placeholder is critical for consistent scoring: include at least two annotated traces showing what 'optimal' and 'misrouted' look like for your specific model pool and pricing structure. For high-risk workflows where routing changes could disrupt production traffic, set [RISK_LEVEL] to 'high' and route all outputs through a human approval step before any automated reconfiguration.

Before deploying this prompt in a production pipeline, validate the output against a golden set of 20-30 manually scored traces to calibrate decision_score thresholds. Pay special attention to edge cases where capability requirements override cost optimization—these are the most common source of false-positive misrouting flags. If your trace data lacks per-token pricing or latency measurements, add a pre-processing step to enrich traces with this data before passing them to the prompt. Do not rely on the model to estimate pricing from memory, as this produces unreliable cost impact numbers.

This prompt is designed to operate as a batch analysis step within a trace observability pipeline, not as a real-time router. You feed it a structured batch of trace segments—each containing the request characteristics, the model selected, observed latency, computed cost, and the configured routing objective—and it returns a scored misrouting report. The implementation harness must therefore handle trace extraction, prompt assembly, response validation, and integration with your existing monitoring or FinOps dashboard. The prompt expects a JSON array of trace objects as [TRACE_BATCH] and a routing objective string (e.g., 'minimize_cost' or 'minimize_latency') as [ROUTING_OBJECTIVE]. The output is a structured JSON report with per-decision scores and an aggregate summary, which you should validate before surfacing to stakeholders or triggering automated alerts.

Start by extracting trace segments from your observability store (e.g., LangSmith, Arize, or a custom trace database) using a time-windowed query. Each trace object must include: trace_id, request_tokens, response_tokens, model_id, model_cost_per_1k_tokens, observed_latency_ms, and available_models (an array of alternative models with their own cost and expected latency profiles). If your traces lack available_models, you must enrich them by querying your model registry or router configuration at the time of the request. Assemble the [TRACE_BATCH] as a JSON array of up to 50 traces to stay within context window limits and avoid token overruns. Set [ROUTING_OBJECTIVE] to the configured policy for that time window. If your router allows per-request objective overrides, include an objective field in each trace object and omit the global [ROUTING_OBJECTIVE] placeholder. Validate the prompt output against a JSON schema that enforces: misrouting_report (array of objects with trace_id, selected_model, optimal_model, objective, cost_delta, latency_delta, misrouting_score), and aggregate_summary (object with total_traces_analyzed, misrouted_count, total_cost_impact, average_latency_penalty_ms). Reject responses that fail schema validation and retry with a stricter [CONSTRAINTS] block that includes the exact schema.

For production use, implement a retry wrapper with exponential backoff (3 attempts max) for malformed JSON or schema validation failures. Log every prompt invocation with the trace batch ID, model used for analysis, token consumption, and validation status. If the aggregate total_cost_impact exceeds a configured threshold (e.g., $500/day), route the report to a human reviewer or trigger a PagerDuty alert. Avoid running this prompt on every individual trace in real time—it is cost-effective only as a periodic batch audit (hourly or daily). For high-frequency routing systems, pair this prompt with a streaming anomaly detection rule that triggers batch analysis only when fallback rates or cost-per-request deviate from baseline. The next step is to integrate the validated aggregate_summary fields into your FinOps dashboard as time-series metrics, enabling trend analysis of routing efficiency over weeks and across model updates.