Latency-Profile Matching Prompt for Model Selection

This prompt is designed for platform engineers and AI infrastructure teams who need to make automated, latency-aware model selection decisions at runtime. Use it when your routing middleware must match incoming requests to the fastest model that still meets quality requirements, based on p50/p95/p99 latency targets and known model performance profiles. The core job-to-be-done is deterministic, auditable model dispatch where every millisecond counts and the cost of a wrong selection is either a blown latency SLO or unnecessary spend on oversized models.

This is not a prompt for one-off model comparisons or manual benchmarking. It belongs inside a model router, API gateway, or request dispatcher. The prompt assumes you already have latency profile data for your candidate models and that you can inject it as structured context. Required inputs include a latency target specification (p50, p95, p99, or a composite), a model catalog with measured latency characteristics under representative load, and any degradation thresholds that define when a faster model becomes mandatory. If you lack measured latency data, start with benchmarking before deploying this prompt. Do not use this prompt when the routing decision depends primarily on capability matching, cost budgets, or content safety—those require separate classification prompts with different evaluation criteria.

In production, wire this prompt into your request pre-processing layer so that every incoming request receives a model assignment before invocation. The output should be a structured decision containing the selected model identifier, the latency tier it satisfies, and a confidence flag. Log every routing decision alongside the actual observed latency so you can detect profile drift. Avoid using this prompt for batch inference pipelines where latency is not the primary constraint, and never use it as a substitute for real-time observability of model serving performance. If your latency profiles are stale, the routing decisions will be wrong regardless of prompt quality.

The prompt below acts as a classification and routing step before model invocation. It takes a request's latency target (p50, p95, p99), a catalog of available models with their measured latency characteristics, and the current system load context. The output is a structured model selection decision with justification, designed to be parsed by a model router or gateway. Use this prompt when your system must make deterministic, auditable trade-offs between speed and capability under variable load conditions.

textCopy
You are a latency-aware model router. Your task is to select the best model to handle a user request given strict latency targets and current system conditions.

## INPUT
- Request Latency Target: [LATENCY_TARGET_MS]
- Target Percentile: [TARGET_PERCENTILE] (e.g., p50, p95, p99)
- Request Complexity Estimate: [COMPLEXITY_LEVEL] (low, medium, high)
- Current System Load: [SYSTEM_LOAD] (normal, elevated, critical)
- Degradation Threshold: [DEGRADATION_THRESHOLD_PERCENT]

## AVAILABLE MODELS
Each model has a measured latency profile in milliseconds under normal load.
[MODEL_CATALOG_JSON]

## CONSTRAINTS
1. The selected model's latency at the [TARGET_PERCENTILE] must be less than or equal to [LATENCY_TARGET_MS].
2. If [SYSTEM_LOAD] is "elevated", apply a [DEGRADATION_THRESHOLD_PERCENT]% buffer to all model latency values before comparison.
3. If [SYSTEM_LOAD] is "critical", you may only select from models tagged as "degraded-ok".
4. If no model satisfies the constraints, select the fastest available model and flag the decision as a "breach".
5. Prefer the most capable model that meets the latency target. Capability ranking is provided in the catalog.

## OUTPUT_SCHEMA
Return a JSON object with the following fields:
{
  "selected_model": "string",
  "expected_latency_ms": number,
  "decision_type": "compliant" | "breach",
  "justification": "string",
  "fallback_chain": ["string"]
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

To adapt this template, replace the square-bracket placeholders with values from your runtime context. The [MODEL_CATALOG_JSON] should be a JSON array of objects, each containing model_id, capability_rank, p50_ms, p95_ms, p99_ms, and a tags array. The [FEW_SHOT_EXAMPLES] placeholder should be populated with 2-3 input-output pairs showing correct routing decisions, including at least one breach scenario and one elevated-load scenario. For high-risk domains where a wrong model choice could cause financial or safety issues, set [RISK_LEVEL] to "high" and ensure the output is logged for human audit before the routing decision is executed. Validate the output against the schema before dispatching; if parsing fails, fall back to the fastest model in the catalog and log the failure.

This prompt is designed to sit inside a model selection middleware layer—typically a lightweight gateway service that inspects each incoming request, extracts latency requirements, and decides which model to invoke. The prompt itself is stateless: it receives a structured latency profile and a set of candidate model characteristics, then returns a ranked selection. The surrounding harness is responsible for supplying accurate, up-to-date model latency data, enforcing the decision, and handling cases where no model meets the target.

Integration pattern. Deploy this prompt as a pre-invocation step in your request pipeline. The harness should: (1) extract or infer the required p50/p95/p99 latency targets from the request context (e.g., an X-Target-Latency-Ms header, a user tier lookup, or a workload classifier); (2) populate [LATENCY_PROFILES] with current benchmark data for each candidate model—this data must come from a regularly refreshed source, not hardcoded assumptions; (3) call the LLM with the prompt template and parse the JSON response; (4) validate that the returned model identifier exists in your available model pool and that the selection rationale references actual latency numbers, not hallucinations. If validation fails, fall back to a safe default model and log the mismatch.

Validation and retry logic. The output schema should be strictly validated before the router acts on it. Check that selected_model is a non-empty string matching a known model ID, that confidence is a float between 0.0 and 1.0, and that rationale is present and non-empty. If the model returns malformed JSON, retry once with a simplified prompt that includes the raw response and a repair instruction. If the confidence score falls below a configurable threshold (e.g., 0.7), route to a conservative fallback model and flag the decision for review. For high-throughput systems, cache latency profile data with a short TTL (30–60 seconds) to avoid redundant LLM calls for identical profile sets.

Observability and drift detection. Log every routing decision with: the input latency targets, the candidate profiles used, the selected model, confidence score, and the actual observed latency after invocation. This creates a feedback loop. Compare predicted latency from the profile data against measured latency over time. If a model's actual p95 latency drifts beyond its profiled value by more than 20% for a sustained window, trigger a profile refresh and consider temporarily removing that model from the candidate pool. This harness turns the prompt from a one-shot decision into a self-correcting routing control loop.

What to avoid. Do not use this prompt for requests where latency targets are unknown or unbounded—it will produce arbitrary selections. Do not feed it stale latency profiles; the model cannot detect drift on its own. Do not skip the fallback path: if the LLM call itself times out or errors, the router must have a hardcoded default model to use immediately. Finally, do not treat the confidence score as a statistical probability; it is a model-generated estimate and should be calibrated against actual success rates in your environment before you rely on it for automated decisions.