Novel Input Classifier Prompt for Review Routing

This prompt is designed for product reliability teams who need to build an automated review queue that catches inputs outside the normal operating envelope of a production AI system. The core job-to-be-done is distribution-aware triage: before an input reaches a downstream model, agent, or RAG pipeline, you need to classify it as in-distribution (safe to process automatically), a boundary case (process with caution or elevated logging), or out-of-distribution (route to a human review queue with diagnostic context). The ideal user is an MLOps engineer, AI safety engineer, or backend developer integrating this classifier into a request preprocessing layer where silent failures on novel inputs are more expensive than a human review loop.

You should use this prompt when you have a defined baseline of expected inputs and a taxonomy of known edge cases, but you anticipate distribution shift, adversarial inputs, or genuinely novel scenarios in production. The prompt requires several concrete inputs to function: a representative sample of in-distribution examples, a set of known boundary cases, and a defined output schema that your review queue system can parse. It works best when paired with a nearest-neighbor retrieval step that provides the classifier with similar known examples for comparison. Do not use this prompt as a standalone safety filter for blocking toxic content or enforcing hard policy rules—it is a routing classifier, not a guardrail. It also should not replace a dedicated prompt injection detector or a hallucination scorer; those are separate specialized tools that may feed into the same review queue.

Before deploying this prompt, you must prepare eval criteria that measure recall on known edge cases (you should catch the boundary cases you already know about) and routing accuracy (in-distribution inputs should not be escalated, and out-of-distribution inputs should not be passed through). Common failure modes include over-escalation when the in-distribution examples are too narrow, and under-escalation when the boundary between in-distribution and boundary-case is poorly defined. Start by running this classifier on a golden dataset of labeled examples, measure the false positive and false negative rates for escalation, and tune the prompt's instructions and few-shot examples until the routing decisions match your risk tolerance. Only then should you wire it into a live review queue with human reviewers who can provide feedback to relabel misclassified cases.

This prompt template is the core classification engine for your review routing pipeline. It takes an input, compares it against a defined distribution baseline and known edge cases, and produces a structured classification with a confidence score and routing decision. The template uses square-bracket placeholders so you can swap in your own taxonomy, examples, and output schema without rewriting the logic. Copy it directly into your prompt management system, replace the placeholders with your domain-specific values, and wire it into your application's pre-processing or guard stage.

textCopy
You are an input classifier for a production AI system. Your job is to determine whether an input falls within the system's known operating distribution, sits at a boundary case, or is out-of-distribution and requires human review.

## Classification Taxonomy
- IN_DISTRIBUTION: The input matches known patterns, topics, and complexity levels the system handles reliably.
- BOUNDARY_CASE: The input partially matches known patterns but contains unusual elements, edge conditions, or ambiguity that reduces confidence.
- OUT_OF_DISTRIBUTION: The input is novel, unexpected, or falls outside the system's defined capability boundary.

## Distribution Baseline
[IN_DISTRIBUTION_DESCRIPTION]

## Known Edge Cases
[EDGE_CASE_EXAMPLES]

## Input to Classify
[INPUT]

## Output Schema
Return a JSON object with exactly these fields:
{
  "classification": "IN_DISTRIBUTION" | "BOUNDARY_CASE" | "OUT_OF_DISTRIBUTION",
  "confidence": 0.0-1.0,
  "rationale": "Brief explanation of the classification decision",
  "nearest_neighbor_context": "Description of the closest known pattern or edge case, or null if none applies",
  "recommended_action": "PROCEED_AUTONOMOUSLY" | "ROUTE_TO_REVIEW" | "BLOCK_AND_ESCALATE",
  "diagnostic_signals": ["List of specific signals that informed the classification"]
}

## Constraints
[CONSTRAINTS]

## Examples
[EXAMPLES]

Adaptation guidance: Replace [IN_DISTRIBUTION_DESCRIPTION] with a concrete description of your system's normal operating range—include topic boundaries, input formats, complexity limits, and any domain-specific guardrails. Populate [EDGE_CASE_EXAMPLES] with 3-5 real boundary cases your system has encountered, including both false positives and false negatives from prior classification attempts. The [CONSTRAINTS] field should capture hard rules such as 'never classify medical emergency language as IN_DISTRIBUTION' or 'inputs exceeding 2000 tokens are automatically BOUNDARY_CASE.' For [EXAMPLES], provide 2-3 few-shot demonstrations showing the full JSON output for each classification category. If your system uses tools or retrieval, add a [TOOLS] placeholder describing available lookup functions. For high-risk domains, set [RISK_LEVEL] to trigger mandatory human review on all BOUNDARY_CASE and OUT_OF_DISTRIBUTION classifications regardless of confidence score.

Before deploying this prompt, validate it against a golden dataset of 50-100 labeled examples spanning all three classification categories. Measure recall on known edge cases and routing accuracy—the proportion of OUT_OF_DISTRIBUTION inputs correctly routed to human review. Common failure modes include over-classifying slightly unusual but safe inputs as BOUNDARY_CASE (false escalations that flood your review queue) and under-classifying genuinely novel inputs as IN_DISTRIBUTION (missed escalations that let unsafe inputs through). Start with a confidence threshold of 0.7 for autonomous routing and adjust based on your tolerance for missed escalations versus review queue volume.

The Novel Input Classifier prompt is designed to sit at the ingress point of an AI pipeline, before any autonomous action or generation occurs. In a production harness, the raw user input is intercepted and passed to this classifier as the [INPUT] variable. The [CONTEXT] variable should be populated with a representative sample of in-distribution examples drawn from your production logs, typically the last 1,000–5,000 inputs, to serve as the baseline for novelty detection. The [OUTPUT_SCHEMA] must be enforced as a strict JSON contract containing classification (one of in_distribution, boundary_case, out_of_distribution), confidence_score (0.0–1.0), nearest_neighbor_examples (an array of the 3 closest matches from the provided context), and diagnostic_notes (a brief explanation of why the input was classified as it was). This structured output is what your application code will parse to make a routing decision.

After the model returns a classification, the application harness must validate the response before acting on it. First, confirm that the classification field matches one of the three allowed enum values; if not, retry the prompt once with a stricter schema reminder. Second, check that confidence_score is a float between 0 and 1. Third, if the classification is boundary_case or out_of_distribution, the harness should automatically route the input to a human review queue, packaging the original input, the classifier's full JSON response, and any available session metadata into a review ticket. For in_distribution classifications with a confidence score below 0.85, consider routing to a secondary review queue with a lower priority label. Log every classification decision, including the model version, prompt template version, input hash, and routing outcome, to enable offline evaluation of classifier accuracy and drift detection over time.

Model choice matters here. Use a model with strong instruction-following and JSON mode support, such as GPT-4o, Claude 3.5 Sonnet, or a fine-tuned open-weight model if you have labeled edge-case data. Avoid smaller or older models that may collapse boundary_case and out_of_distribution into a single bucket or produce malformed JSON under pressure. If your application has strict latency requirements, deploy this classifier as an asynchronous pre-check that populates a review queue rather than blocking the user-facing response. Never allow an input classified as out_of_distribution to proceed to autonomous execution without human approval. For high-stakes domains such as healthcare, finance, or safety-critical operations, require explicit human sign-off on all boundary_case inputs as well, and log the reviewer's identity and timestamp alongside the classification record for audit purposes.