Latency-SLA Classification Prompt for Queue Assignment

This prompt is for platform engineers and infrastructure teams who need to classify incoming requests into latency tiers—realtime, nearline, or batch—before dispatching them to processing queues. Use it when your system has multiple queues with different SLA targets and you need a deterministic, auditable classification step that considers both request characteristics and current system state. The prompt produces a latency tier label with a confidence score, a rationale, and a recommended timeout. It is designed to be wired into a routing middleware layer where misclassification directly causes SLA breaches or wasted compute.

Before integrating this prompt, you must have defined queue definitions with explicit latency targets and ensure the classification decision will be executed by downstream infrastructure. The prompt expects variables for your SLA targets, queue definitions, and current system load metrics. It works best when the classification logic must be transparent and auditable—for example, when you need to log why a request was routed to the batch queue instead of realtime. The confidence score allows you to set thresholds for human review or fallback routing when the model is uncertain.

Do not use this prompt for general intent classification or for systems with a single processing queue. It is not a replacement for a rules engine when your routing logic is purely deterministic based on a few known fields. If your latency tiers are static and request characteristics map cleanly to queues without ambiguity, a simple rule-based router will be faster, cheaper, and more reliable. This prompt adds value when classification requires reasoning across multiple signals—request content, system load, SLA definitions, and edge cases that are hard to codify in static rules. Before deploying, run evals measuring SLA breach rate and misclassification cost against your current routing approach to confirm the prompt improves outcomes rather than adding latency and cost to the routing step itself.

This prompt template is designed to be wired into a routing middleware layer that assigns every incoming request to a processing queue based on its latency sensitivity. The model receives the request payload, the defined SLA targets for each tier, the available queues and their characteristics, and any hard timeout constraints. It must return a structured classification decision—never free text—so that downstream queue dispatchers can act deterministically. Replace every square-bracket placeholder with your system's actual configuration before deployment. The template assumes you have already defined your latency tiers, queue names, and SLA thresholds; if those are still in flux, start with a smaller evaluation dataset before scaling.

textCopy
You are a latency-SLA classifier for a multi-queue processing system. Your job is to read an incoming request and assign it to exactly one latency tier: realtime, nearline, or batch. You must also provide a confidence score and a brief justification grounded in the request characteristics and the SLA definitions provided below.

[QUEUE_DEFINITIONS]

[LATENCY_SLA_TARGETS]

[TIMEOUT_CONSTRAINTS]

[INPUT_REQUEST]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

[EXAMPLES]

After copying the template, replace each placeholder with concrete values. [QUEUE_DEFINITIONS] should list the available queues, their expected latency profiles, and any capacity notes. [LATENCY_SLA_TARGETS] must specify the p95 or p99 latency targets in milliseconds for each tier, along with breach consequences. [TIMEOUT_CONSTRAINTS] defines hard deadlines after which a request must be abandoned or escalated. [INPUT_REQUEST] is the raw payload your system receives—keep it unmodified to avoid information loss. [OUTPUT_SCHEMA] should be a strict JSON schema with fields for tier, confidence, and justification. [CONSTRAINTS] can encode rules like "never classify a payment request as batch" or "if confidence is below 0.7, flag for human review." [EXAMPLES] should include at least one correct classification per tier, plus an ambiguous edge case. Do not ship this prompt without running it against a golden dataset that measures SLA breach rate and misclassification cost.

This prompt is designed to sit at the ingress point of a request processing pipeline, immediately before queue dispatch logic. The application layer should extract the raw user request or task payload, inject it into the [INPUT] placeholder, and populate [QUEUE_DEFINITIONS] with the current SLA tiers, latency targets, and timeout values from your runtime configuration. The [SLA_TARGETS] variable should be drawn from a service catalog or config map, not hardcoded, so that SLA changes propagate without prompt edits. The model call should be wrapped in a function that enforces a strict timeout—typically 2-5 seconds for a classification task—because a slow classifier defeats the purpose of latency-aware routing.

Validation and retry logic is critical before the classification result touches any queue. Parse the model's JSON output and validate that latency_tier matches one of the allowed enum values from your queue definitions, that confidence is a float between 0.0 and 1.0, and that rationale is a non-empty string. If validation fails, retry once with the same prompt plus the validation error message appended as additional context. If the retry also fails, log the failure and route the request to a configurable fallback queue—typically nearline or a dedicated unclassified dead-letter queue for human review. Never silently default to realtime on failure; that creates a hidden SLA breach vector under load.

Observability and logging should capture the full classification event: input hash, assigned tier, confidence score, model used, latency of the classification call itself, and whether validation or retries were triggered. Emit these as structured logs or metrics so you can track SLA breach rate, misclassification cost, and classifier latency percentiles over time. Set up an eval harness that replays a golden dataset of labeled requests through the prompt weekly, measuring tier accuracy and confidence calibration. When you detect drift—such as the model increasingly over-classifying requests as realtime—trigger a prompt review or model update before production SLA breaches accumulate.

Model choice matters for both cost and latency. This prompt works well with fast, instruction-tuned models like Claude 3.5 Haiku, GPT-4o-mini, or Gemini 1.5 Flash. Avoid using large reasoning models for this task; the classification logic is straightforward and the added latency of a larger model undermines the routing goal. If you observe the classifier itself becoming a latency bottleneck under high throughput, consider batching classification requests where the SLA window permits, or caching classification results for identical or near-identical inputs using a content hash. Do not add RAG or tool calls to this prompt—the classification decision should be self-contained and deterministic given the input and queue definitions.

Human review integration should be gated on low-confidence outputs. When confidence falls below a configurable threshold (start with 0.7 and tune based on your eval data), route the request to a review queue instead of directly to a processing queue. The review interface should display the original input, the model's tentative classification, and the rationale, allowing a human operator to confirm or override within your SLA window. Track override rates by tier to identify systematic misclassification patterns and feed those examples back into your few-shot examples or fine-tuning dataset.