This prompt is a pre-classification stage for domain routing pipelines where static few-shot examples degrade under real-world input variability. Use it when you have a library of labeled examples, a defined domain taxonomy, and a downstream classifier whose accuracy depends on example quality. The job-to-be-done is selecting the most relevant examples from your pool for each incoming request before the classification step, so the classifier sees examples that match the input's length, terminology, and ambiguity profile rather than a fixed set that works for some inputs and confuses others.
Prompt
Domain Classification Few-Shot Example Selection Prompt

When to Use This Prompt
Understand when dynamic few-shot example selection is the right pre-classification step and when a simpler approach will work better.
The ideal user is an AI platform engineer or ML engineer building a multi-tenant or multi-product routing system. You need a labeled example pool with at least 20–30 examples per domain to make selection meaningful, and you need a downstream classifier prompt that accepts a [SELECTED_EXAMPLES] placeholder. Do not use this prompt when you have fewer than 10 examples per domain, when your classifier already performs above your accuracy threshold with a static prompt, or when latency constraints make an extra model call unacceptable. This prompt is also inappropriate as a replacement for the classifier itself—it selects examples, it does not perform classification.
Before implementing, measure your current classifier's accuracy variance across input lengths and domains. If you see a 15%+ accuracy drop on long or ambiguous inputs, dynamic selection is likely worth the added latency. Wire the selection prompt as a separate call before classification, cache selections per input characteristics if throughput matters, and log which examples were selected alongside classification results so you can measure whether selection actually improves downstream accuracy. Avoid using this prompt when your domain taxonomy changes frequently, since the example pool and selection criteria will need constant maintenance.
Use Case Fit
Where this prompt works and where it does not. Use these cards to decide if dynamic few-shot example selection is the right pattern for your domain classifier, or if a simpler approach will ship faster and break less often.
Good Fit: High-Cardinality Taxonomies with Sparse Examples
Use when: your domain taxonomy has 20+ categories and you cannot statically include representative examples for every class in a single prompt. Guardrail: measure per-category recall improvement with dynamic selection versus a static baseline before committing to the added latency.
Bad Fit: Stable, Low-Cardinality Classification
Avoid when: you have fewer than 10 well-separated domains with consistent input patterns. Guardrail: a static few-shot prompt with hand-picked examples is cheaper, faster, and easier to debug. Only add dynamic selection when static performance demonstrably fails on specific categories.
Required Inputs: Example Store and Retrieval Mechanism
Risk: dynamic selection is only as good as the example pool it draws from. Guardrail: you need a curated, labeled example store with at least 5–10 diverse examples per domain, plus a retrieval step (embedding similarity, keyword, or hybrid) that runs before the classification prompt is assembled.
Operational Risk: Selection Latency Adds to Classification Latency
Risk: adding a retrieval step before classification doubles the critical path latency. Guardrail: benchmark end-to-end latency (retrieval + classification) under load. Set a timeout and fall back to a static example set or a default routing decision if retrieval exceeds the budget.
Operational Risk: Example Drift Over Time
Risk: the example store becomes stale as domains shift, new categories emerge, or input distributions change. Guardrail: log which examples were selected per classification and periodically audit whether they still represent the target domain. Schedule example store refreshes alongside taxonomy updates.
Operational Risk: Selection Bias Amplifies Classification Errors
Risk: if the retrieval step consistently selects examples from the wrong domain, the classifier inherits and amplifies that error. Guardrail: implement a confidence threshold on the classifier output. When confidence is low, log the selected examples and the input for offline review rather than silently misrouting.
Copy-Ready Prompt Template
A ready-to-use prompt that selects the most relevant few-shot examples from a candidate pool for a given domain classification input.
This prompt template is the core of your dynamic few-shot example selector. It is designed to be inserted into a step that runs before your main domain classifier. The prompt takes a pool of candidate examples and a new input to classify, then returns a ranked list of the most relevant examples with a relevance score. The goal is to maximize the downstream classifier's accuracy by providing it with examples that are semantically and structurally similar to the input at hand, rather than using a static set.
textYou are an expert example selector for a domain classification system. Your task is to analyze the input text and select the most relevant few-shot examples from the provided candidate pool. The selected examples will be used to guide a downstream classifier, so they must maximize the classifier's ability to correctly identify the domain. # INPUT TO CLASSIFY [INPUT] # CANDIDATE EXAMPLE POOL [CANDIDATE_POOL] # SELECTION CRITERIA 1. **Semantic Similarity:** Choose examples whose text is semantically closest to the input. Prioritize examples that share key terminology, intent, and subject matter. 2. **Structural Similarity:** Favor examples that mirror the input's format (e.g., question length, presence of code blocks, list structure). 3. **Domain Coverage:** If the input is ambiguous, select examples that represent the most likely candidate domains to help the downstream classifier disambiguate. 4. **Diversity:** Avoid selecting multiple examples that are near-duplicates of each other. Prioritize a diverse set that covers different facets of the relevant domain(s). # OUTPUT FORMAT Return a valid JSON object with a single key "selected_examples". The value must be a list of objects, each with the keys "example_id", "relevance_score", and "rationale". The list must be sorted by "relevance_score" in descending order and contain exactly [K] examples. ```json { "selected_examples": [ { "example_id": "<id from candidate pool>", "relevance_score": 0.95, "rationale": "One-sentence explanation of why this example was chosen." }, ... ] }
CONSTRAINTS
- Do not select more than [K] examples.
- Only use example_ids that exist in the candidate pool.
- If the candidate pool has fewer than [K] examples, return all of them.
- The relevance_score must be a float between 0.0 and 1.0.
To adapt this template, you must populate the three square-bracket placeholders. [INPUT] is the raw text you need to classify. [CANDIDATE_POOL] should be a structured list of all possible examples, each with a unique example_id and the text of the example. [K] is the integer number of examples you want to select, typically between 3 and 8. The output is a clean JSON payload that your application can parse and use to dynamically construct the prompt for your downstream domain classifier. Ensure your application logic validates that the returned example_id values are present in your source of truth before injecting the corresponding full examples into the next step.
Prompt Variables
Inputs the prompt needs to select optimal few-shot examples for domain classification. Validate these before each call to prevent silent misclassification from stale or mismatched examples.
| Placeholder | Purpose | Example | Validation Notes |
|---|---|---|---|
[INPUT_TEXT] | The raw user query or document to classify | How do I reset my API key for the payments service? | Required. Must be non-empty string. Check length under model context limit minus example budget. Reject if null or whitespace-only. |
[DOMAIN_TAXONOMY] | The full list of valid domain labels with descriptions | ["billing", "technical_support", "account_management", "compliance"] | Required. Must be a valid JSON array of strings. Each label must have a non-empty description. Validate no duplicate labels. Reject if taxonomy changed since example set was curated. |
[EXAMPLE_POOL] | The complete set of labeled few-shot examples available for selection | [{"text": "My invoice is wrong", "label": "billing", "features": ["invoice", "charge"]}] | Required. Must be valid JSON array with text, label, and features fields. Label must exist in DOMAIN_TAXONOMY. Validate pool size is sufficient for selection count. Reject if any example has missing fields. |
[SELECTION_COUNT] | Number of few-shot examples to select per domain or total | 3 | Required. Integer between 1 and 10. Must not exceed EXAMPLE_POOL size per domain. Validate against model context window budget. Default to 3 if unspecified but log warning. |
[SELECTION_STRATEGY] | The method for choosing examples: similarity, diversity, contrast, or balanced | balanced | Required. Must be one of: similarity, diversity, contrast, balanced. Validate against downstream classifier performance expectations. Reject unrecognized strategies. |
[SIMILARITY_THRESHOLD] | Minimum similarity score for an example to be considered relevant | 0.65 | Optional. Float between 0.0 and 1.0. Used only when SELECTION_STRATEGY is similarity or balanced. If null, default to 0.5. Validate threshold is calibrated against embedding model used. |
[OUTPUT_FORMAT] | Expected structure for the selected examples in the response | {"selected_examples": [{"text": "...", "label": "...", "rationale": "..."}]} | Required. Must be a valid JSON schema or type description. Validate that downstream classifier prompt expects this exact shape. Reject if schema is incompatible with classifier harness. |
Implementation Harness Notes
How to wire the Domain Classification Few-Shot Example Selection Prompt into a production classification pipeline with validation, retries, and evaluation.
This prompt is not a standalone classifier; it is a pre-processing step that dynamically selects the most relevant few-shot examples for a downstream domain classification call. The implementation harness must treat this as a synchronous dependency: the example selector runs first, its output is validated, and the selected examples are injected into the classifier prompt template. This architecture works best when you maintain a curated example bank—a JSON or database store of labeled domain examples with metadata such as domain, sub-domain, difficulty, and edge-case flags. The selector prompt queries this bank (via retrieval or direct inclusion) and returns a ranked list of example IDs or the full example text to inject into the classifier.
Wire the selector as a function select_examples(input_text, example_bank, top_k=5) that: (1) formats the prompt template with the input text and a serialized subset of the example bank (or retrieves the top-N candidates via vector search before prompting); (2) calls the model with response_format set to a strict JSON schema expecting {"selected_examples": [{"id": "string", "reason": "string"}]}; (3) validates that returned IDs exist in the bank and that the count does not exceed top_k; (4) on validation failure, retry once with an explicit error message injected into the prompt; (5) log the selected IDs, input hash, and latency for offline evaluation. For high-throughput systems, cache selections by input embedding cluster to avoid redundant LLM calls on similar inputs. If the downstream classifier uses a different model family, test that the selected examples transfer well—some models benefit from shorter examples, others from contrastive pairs. Always run an offline eval that compares downstream classification accuracy with random example selection versus selector-chosen examples to confirm the selector adds value.
The primary failure mode is the selector overfitting to surface-level lexical similarity rather than semantic domain fit, especially when the example bank is small or skewed. Mitigate this by including negative examples in the bank (examples that look similar but belong to a different domain) and by periodically auditing selection patterns for bias toward certain example authors or formats. If the downstream classifier has a confidence threshold, route low-confidence classifications to human review and log whether the selector's chosen examples were appropriate—this creates a feedback loop for improving the example bank. Do not let the selector silently mask a bad example bank; if selection quality degrades, fall back to a static curated set and alert the pipeline owner.
Expected Output Contract
The model must return a JSON object containing the selected example IDs, a confidence score, and a rationale. Validate every field before using the output to construct the few-shot prompt for the downstream domain classifier.
| Field or Element | Type or Format | Required | Validation Rule |
|---|---|---|---|
selected_example_ids | array of strings | Each ID must exist in the [EXAMPLE_POOL]. Array length must be between [MIN_EXAMPLES] and [MAX_EXAMPLES]. No duplicate IDs allowed. | |
confidence_score | number (float) | Must be between 0.0 and 1.0 inclusive. Parse as float and check bounds. If below [CONFIDENCE_THRESHOLD], route to human review or fallback. | |
rationale | string | Must be non-empty after trimming whitespace. Length must not exceed [MAX_RATIONALE_CHARS]. Should reference specific characteristics of [INPUT_TEXT]. | |
domain_hint | string or null | If provided, must match a canonical domain label from [DOMAIN_TAXONOMY]. Null allowed when no clear domain signal exists. | |
selection_strategy | string (enum) | Must be one of: 'similarity', 'diversity', 'contrastive', 'boundary'. Reject any value not in this enum. | |
abstention_flag | boolean | Must be true if the model cannot select examples with confidence above [CONFIDENCE_THRESHOLD]. When true, selected_example_ids must be an empty array. | |
processing_metadata | object | If present, must contain 'model_version' (string) and 'inference_time_ms' (integer). Reject if schema does not match. Null allowed. |
Common Failure Modes
Few-shot example selection is sensitive to distribution drift, label noise, and selection logic errors. These failure modes degrade downstream classification accuracy before you notice them in production.
Example Selection Bias Toward Majority Classes
What to watch: The selector over-indexes on high-frequency domain examples, starving minority classes of representative few-shot demonstrations. This causes the downstream classifier to miss rare but critical domains. Guardrail: Stratify the example pool by domain label and enforce minimum representation per class. Log per-class example counts in selection traces.
Semantic Similarity Mismatch
What to watch: Embedding-based selection retrieves superficially similar examples that share vocabulary but not the correct domain label. The classifier learns from misleading demonstrations. Guardrail: Filter candidate examples by label agreement before similarity ranking. Use a dual-threshold: minimum similarity score plus label match requirement.
Stale Example Pool Drift
What to watch: The example pool reflects old domain distributions, terminology, or product names. Selected examples teach the classifier outdated patterns that no longer match production inputs. Guardrail: Version the example pool with a freshness timestamp. Run periodic distribution comparison between the pool and recent production inputs. Trigger review when drift exceeds threshold.
Selection Prompt Leaking Label Information
What to watch: The selection prompt accidentally includes the correct label or strong hints in its instructions, causing the selector to cheat rather than learn to generalize from input features alone. Guardrail: Audit selection prompts for label leakage. Test with held-out examples where labels are stripped before selection. Measure whether selection quality degrades without label access.
Overfitting to Selection Heuristics
What to watch: The selector learns brittle patterns from the selection prompt itself rather than the input characteristics. It consistently picks the same examples regardless of input variation. Guardrail: Measure example diversity across a batch of inputs. If the same examples appear in more than 50% of selections, add randomization or temperature to the selection step.
Silent Degradation from Downstream Model Changes
What to watch: The downstream classifier model is updated or swapped, but the example selection logic remains unchanged. Previously effective examples become misleading for the new model's behavior. Guardrail: Couple example pool versions to downstream model versions. Re-evaluate selection quality whenever the classifier model changes. Maintain a regression test suite that measures end-to-end classification accuracy with the selected examples.
Evaluation Rubric
How to test output quality before shipping. Run these checks on a golden dataset of at least 100 inputs with known correct domain labels.
| Criterion | Pass Standard | Failure Signal | Test Method |
|---|---|---|---|
Domain Label Accuracy | Selected examples share the same domain label as the golden input in >= 95% of test cases | Example domain mismatches golden domain in > 5% of cases; downstream classifier accuracy degrades | Run golden dataset through example selector, compare selected-example domains to golden labels, compute exact-match rate |
Example Relevance Score | Mean cosine similarity between input embedding and selected example embeddings exceeds 0.80 threshold | Mean similarity below 0.65; selected examples are semantically distant from the target input | Embed input and selected examples with same model, compute pairwise cosine similarity, aggregate across dataset |
Downstream Classification Impact | Few-shot classifier using selected examples achieves >= 2% accuracy gain over random example selection baseline | Selected examples produce lower or equal accuracy vs. random baseline; selector adds cost without benefit | Run classifier with selected examples vs. random examples on held-out test set, compare F1 scores with statistical significance test |
Selection Diversity | Selected examples span at least 2 distinct sub-categories or facets within the target domain for each input | All selected examples are near-duplicates or share identical sub-category; no coverage of domain variation | Cluster selected examples by sub-category label, count unique clusters per input, compute mean across dataset |
Confidence Calibration | Selector confidence score correlates with downstream classification accuracy at Spearman ρ >= 0.70 | High-confidence selections produce low-accuracy classifications; confidence is uninformative or misleading | Bin selections by confidence decile, compute mean downstream accuracy per bin, calculate rank correlation |
Abstention Rate on OOD Inputs | Selector returns empty set or explicit rejection for >= 90% of out-of-distribution inputs | Selector confidently returns examples for OOD inputs; forces incorrect domain classification downstream | Feed inputs from held-out domains, verify selector returns no examples or rejection signal, compute abstention rate |
Latency Budget Compliance | 95th percentile selection latency <= 200ms for single-input example retrieval | P95 latency exceeds 500ms; selector becomes bottleneck in real-time routing pipeline | Measure end-to-end selection time across 1000 requests under load, compute P50/P95/P99 latency |
Edge Case: Ambiguous Inputs | Selector returns examples from all plausible domains or abstains when input spans domain boundaries | Selector picks single domain confidently for ambiguous multi-domain input; downstream routing is silently wrong | Curate 20 ambiguous inputs with known multi-domain labels, verify selector output includes all plausible domains or abstains |
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.

Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.

Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
Adapt This Prompt
How to adapt
Start with the base prompt and a small hand-curated example bank of 20-30 labeled domain pairs. Use a simple string-match fallback if the model returns an invalid index. Skip the full eval harness initially—just log classification accuracy on a spreadsheet.
Modify the prompt to accept a flat list instead of a structured taxonomy:
code[EXAMPLES] = [ {"input": "...", "domain": "billing"}, {"input": "...", "domain": "technical"} ]
Watch for
- Model selecting examples that look similar but map to the wrong domain
- Empty example banks causing random selection
- Token limits if you stuff too many examples into the prompt

About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us