Inferensys

Prompt

Domain Classification Few-Shot Example Selection Prompt

A practical prompt playbook for dynamically selecting the most relevant few-shot examples to maximize domain classification accuracy in production AI routing workflows.
Elegant overhead shot of a polished wooden communal table in a sun-drenched WeWork lounge, laptops and tablets displaying AI workflow dashboards, plants and pendant lights in background.
PROMPT PLAYBOOK

When to Use This Prompt

Understand when dynamic few-shot example selection is the right pre-classification step and when a simpler approach will work better.

This prompt is a pre-classification stage for domain routing pipelines where static few-shot examples degrade under real-world input variability. Use it when you have a library of labeled examples, a defined domain taxonomy, and a downstream classifier whose accuracy depends on example quality. The job-to-be-done is selecting the most relevant examples from your pool for each incoming request before the classification step, so the classifier sees examples that match the input's length, terminology, and ambiguity profile rather than a fixed set that works for some inputs and confuses others.

The ideal user is an AI platform engineer or ML engineer building a multi-tenant or multi-product routing system. You need a labeled example pool with at least 20–30 examples per domain to make selection meaningful, and you need a downstream classifier prompt that accepts a [SELECTED_EXAMPLES] placeholder. Do not use this prompt when you have fewer than 10 examples per domain, when your classifier already performs above your accuracy threshold with a static prompt, or when latency constraints make an extra model call unacceptable. This prompt is also inappropriate as a replacement for the classifier itself—it selects examples, it does not perform classification.

Before implementing, measure your current classifier's accuracy variance across input lengths and domains. If you see a 15%+ accuracy drop on long or ambiguous inputs, dynamic selection is likely worth the added latency. Wire the selection prompt as a separate call before classification, cache selections per input characteristics if throughput matters, and log which examples were selected alongside classification results so you can measure whether selection actually improves downstream accuracy. Avoid using this prompt when your domain taxonomy changes frequently, since the example pool and selection criteria will need constant maintenance.

PRACTICAL GUARDRAILS

Use Case Fit

Where this prompt works and where it does not. Use these cards to decide if dynamic few-shot example selection is the right pattern for your domain classifier, or if a simpler approach will ship faster and break less often.

01

Good Fit: High-Cardinality Taxonomies with Sparse Examples

Use when: your domain taxonomy has 20+ categories and you cannot statically include representative examples for every class in a single prompt. Guardrail: measure per-category recall improvement with dynamic selection versus a static baseline before committing to the added latency.

02

Bad Fit: Stable, Low-Cardinality Classification

Avoid when: you have fewer than 10 well-separated domains with consistent input patterns. Guardrail: a static few-shot prompt with hand-picked examples is cheaper, faster, and easier to debug. Only add dynamic selection when static performance demonstrably fails on specific categories.

03

Required Inputs: Example Store and Retrieval Mechanism

Risk: dynamic selection is only as good as the example pool it draws from. Guardrail: you need a curated, labeled example store with at least 5–10 diverse examples per domain, plus a retrieval step (embedding similarity, keyword, or hybrid) that runs before the classification prompt is assembled.

04

Operational Risk: Selection Latency Adds to Classification Latency

Risk: adding a retrieval step before classification doubles the critical path latency. Guardrail: benchmark end-to-end latency (retrieval + classification) under load. Set a timeout and fall back to a static example set or a default routing decision if retrieval exceeds the budget.

05

Operational Risk: Example Drift Over Time

Risk: the example store becomes stale as domains shift, new categories emerge, or input distributions change. Guardrail: log which examples were selected per classification and periodically audit whether they still represent the target domain. Schedule example store refreshes alongside taxonomy updates.

06

Operational Risk: Selection Bias Amplifies Classification Errors

Risk: if the retrieval step consistently selects examples from the wrong domain, the classifier inherits and amplifies that error. Guardrail: implement a confidence threshold on the classifier output. When confidence is low, log the selected examples and the input for offline review rather than silently misrouting.

PROMPT PLAYBOOK

Copy-Ready Prompt Template

A ready-to-use prompt that selects the most relevant few-shot examples from a candidate pool for a given domain classification input.

This prompt template is the core of your dynamic few-shot example selector. It is designed to be inserted into a step that runs before your main domain classifier. The prompt takes a pool of candidate examples and a new input to classify, then returns a ranked list of the most relevant examples with a relevance score. The goal is to maximize the downstream classifier's accuracy by providing it with examples that are semantically and structurally similar to the input at hand, rather than using a static set.

text
You are an expert example selector for a domain classification system. Your task is to analyze the input text and select the most relevant few-shot examples from the provided candidate pool. The selected examples will be used to guide a downstream classifier, so they must maximize the classifier's ability to correctly identify the domain.

# INPUT TO CLASSIFY
[INPUT]

# CANDIDATE EXAMPLE POOL
[CANDIDATE_POOL]

# SELECTION CRITERIA
1. **Semantic Similarity:** Choose examples whose text is semantically closest to the input. Prioritize examples that share key terminology, intent, and subject matter.
2. **Structural Similarity:** Favor examples that mirror the input's format (e.g., question length, presence of code blocks, list structure).
3. **Domain Coverage:** If the input is ambiguous, select examples that represent the most likely candidate domains to help the downstream classifier disambiguate.
4. **Diversity:** Avoid selecting multiple examples that are near-duplicates of each other. Prioritize a diverse set that covers different facets of the relevant domain(s).

# OUTPUT FORMAT
Return a valid JSON object with a single key "selected_examples". The value must be a list of objects, each with the keys "example_id", "relevance_score", and "rationale". The list must be sorted by "relevance_score" in descending order and contain exactly [K] examples.

```json
{
  "selected_examples": [
    {
      "example_id": "<id from candidate pool>",
      "relevance_score": 0.95,
      "rationale": "One-sentence explanation of why this example was chosen."
    },
    ...
  ]
}

CONSTRAINTS

  • Do not select more than [K] examples.
  • Only use example_ids that exist in the candidate pool.
  • If the candidate pool has fewer than [K] examples, return all of them.
  • The relevance_score must be a float between 0.0 and 1.0.

To adapt this template, you must populate the three square-bracket placeholders. [INPUT] is the raw text you need to classify. [CANDIDATE_POOL] should be a structured list of all possible examples, each with a unique example_id and the text of the example. [K] is the integer number of examples you want to select, typically between 3 and 8. The output is a clean JSON payload that your application can parse and use to dynamically construct the prompt for your downstream domain classifier. Ensure your application logic validates that the returned example_id values are present in your source of truth before injecting the corresponding full examples into the next step.

IMPLEMENTATION TABLE

Prompt Variables

Inputs the prompt needs to select optimal few-shot examples for domain classification. Validate these before each call to prevent silent misclassification from stale or mismatched examples.

PlaceholderPurposeExampleValidation Notes

[INPUT_TEXT]

The raw user query or document to classify

How do I reset my API key for the payments service?

Required. Must be non-empty string. Check length under model context limit minus example budget. Reject if null or whitespace-only.

[DOMAIN_TAXONOMY]

The full list of valid domain labels with descriptions

["billing", "technical_support", "account_management", "compliance"]

Required. Must be a valid JSON array of strings. Each label must have a non-empty description. Validate no duplicate labels. Reject if taxonomy changed since example set was curated.

[EXAMPLE_POOL]

The complete set of labeled few-shot examples available for selection

[{"text": "My invoice is wrong", "label": "billing", "features": ["invoice", "charge"]}]

Required. Must be valid JSON array with text, label, and features fields. Label must exist in DOMAIN_TAXONOMY. Validate pool size is sufficient for selection count. Reject if any example has missing fields.

[SELECTION_COUNT]

Number of few-shot examples to select per domain or total

3

Required. Integer between 1 and 10. Must not exceed EXAMPLE_POOL size per domain. Validate against model context window budget. Default to 3 if unspecified but log warning.

[SELECTION_STRATEGY]

The method for choosing examples: similarity, diversity, contrast, or balanced

balanced

Required. Must be one of: similarity, diversity, contrast, balanced. Validate against downstream classifier performance expectations. Reject unrecognized strategies.

[SIMILARITY_THRESHOLD]

Minimum similarity score for an example to be considered relevant

0.65

Optional. Float between 0.0 and 1.0. Used only when SELECTION_STRATEGY is similarity or balanced. If null, default to 0.5. Validate threshold is calibrated against embedding model used.

[OUTPUT_FORMAT]

Expected structure for the selected examples in the response

{"selected_examples": [{"text": "...", "label": "...", "rationale": "..."}]}

Required. Must be a valid JSON schema or type description. Validate that downstream classifier prompt expects this exact shape. Reject if schema is incompatible with classifier harness.

PROMPT PLAYBOOK

Implementation Harness Notes

How to wire the Domain Classification Few-Shot Example Selection Prompt into a production classification pipeline with validation, retries, and evaluation.

This prompt is not a standalone classifier; it is a pre-processing step that dynamically selects the most relevant few-shot examples for a downstream domain classification call. The implementation harness must treat this as a synchronous dependency: the example selector runs first, its output is validated, and the selected examples are injected into the classifier prompt template. This architecture works best when you maintain a curated example bank—a JSON or database store of labeled domain examples with metadata such as domain, sub-domain, difficulty, and edge-case flags. The selector prompt queries this bank (via retrieval or direct inclusion) and returns a ranked list of example IDs or the full example text to inject into the classifier.

Wire the selector as a function select_examples(input_text, example_bank, top_k=5) that: (1) formats the prompt template with the input text and a serialized subset of the example bank (or retrieves the top-N candidates via vector search before prompting); (2) calls the model with response_format set to a strict JSON schema expecting {"selected_examples": [{"id": "string", "reason": "string"}]}; (3) validates that returned IDs exist in the bank and that the count does not exceed top_k; (4) on validation failure, retry once with an explicit error message injected into the prompt; (5) log the selected IDs, input hash, and latency for offline evaluation. For high-throughput systems, cache selections by input embedding cluster to avoid redundant LLM calls on similar inputs. If the downstream classifier uses a different model family, test that the selected examples transfer well—some models benefit from shorter examples, others from contrastive pairs. Always run an offline eval that compares downstream classification accuracy with random example selection versus selector-chosen examples to confirm the selector adds value.

The primary failure mode is the selector overfitting to surface-level lexical similarity rather than semantic domain fit, especially when the example bank is small or skewed. Mitigate this by including negative examples in the bank (examples that look similar but belong to a different domain) and by periodically auditing selection patterns for bias toward certain example authors or formats. If the downstream classifier has a confidence threshold, route low-confidence classifications to human review and log whether the selector's chosen examples were appropriate—this creates a feedback loop for improving the example bank. Do not let the selector silently mask a bad example bank; if selection quality degrades, fall back to a static curated set and alert the pipeline owner.

IMPLEMENTATION TABLE

Expected Output Contract

The model must return a JSON object containing the selected example IDs, a confidence score, and a rationale. Validate every field before using the output to construct the few-shot prompt for the downstream domain classifier.

Field or ElementType or FormatRequiredValidation Rule

selected_example_ids

array of strings

Each ID must exist in the [EXAMPLE_POOL]. Array length must be between [MIN_EXAMPLES] and [MAX_EXAMPLES]. No duplicate IDs allowed.

confidence_score

number (float)

Must be between 0.0 and 1.0 inclusive. Parse as float and check bounds. If below [CONFIDENCE_THRESHOLD], route to human review or fallback.

rationale

string

Must be non-empty after trimming whitespace. Length must not exceed [MAX_RATIONALE_CHARS]. Should reference specific characteristics of [INPUT_TEXT].

domain_hint

string or null

If provided, must match a canonical domain label from [DOMAIN_TAXONOMY]. Null allowed when no clear domain signal exists.

selection_strategy

string (enum)

Must be one of: 'similarity', 'diversity', 'contrastive', 'boundary'. Reject any value not in this enum.

abstention_flag

boolean

Must be true if the model cannot select examples with confidence above [CONFIDENCE_THRESHOLD]. When true, selected_example_ids must be an empty array.

processing_metadata

object

If present, must contain 'model_version' (string) and 'inference_time_ms' (integer). Reject if schema does not match. Null allowed.

PRACTICAL GUARDRAILS

Common Failure Modes

Few-shot example selection is sensitive to distribution drift, label noise, and selection logic errors. These failure modes degrade downstream classification accuracy before you notice them in production.

01

Example Selection Bias Toward Majority Classes

What to watch: The selector over-indexes on high-frequency domain examples, starving minority classes of representative few-shot demonstrations. This causes the downstream classifier to miss rare but critical domains. Guardrail: Stratify the example pool by domain label and enforce minimum representation per class. Log per-class example counts in selection traces.

02

Semantic Similarity Mismatch

What to watch: Embedding-based selection retrieves superficially similar examples that share vocabulary but not the correct domain label. The classifier learns from misleading demonstrations. Guardrail: Filter candidate examples by label agreement before similarity ranking. Use a dual-threshold: minimum similarity score plus label match requirement.

03

Stale Example Pool Drift

What to watch: The example pool reflects old domain distributions, terminology, or product names. Selected examples teach the classifier outdated patterns that no longer match production inputs. Guardrail: Version the example pool with a freshness timestamp. Run periodic distribution comparison between the pool and recent production inputs. Trigger review when drift exceeds threshold.

04

Selection Prompt Leaking Label Information

What to watch: The selection prompt accidentally includes the correct label or strong hints in its instructions, causing the selector to cheat rather than learn to generalize from input features alone. Guardrail: Audit selection prompts for label leakage. Test with held-out examples where labels are stripped before selection. Measure whether selection quality degrades without label access.

05

Overfitting to Selection Heuristics

What to watch: The selector learns brittle patterns from the selection prompt itself rather than the input characteristics. It consistently picks the same examples regardless of input variation. Guardrail: Measure example diversity across a batch of inputs. If the same examples appear in more than 50% of selections, add randomization or temperature to the selection step.

06

Silent Degradation from Downstream Model Changes

What to watch: The downstream classifier model is updated or swapped, but the example selection logic remains unchanged. Previously effective examples become misleading for the new model's behavior. Guardrail: Couple example pool versions to downstream model versions. Re-evaluate selection quality whenever the classifier model changes. Maintain a regression test suite that measures end-to-end classification accuracy with the selected examples.

IMPLEMENTATION TABLE

Evaluation Rubric

How to test output quality before shipping. Run these checks on a golden dataset of at least 100 inputs with known correct domain labels.

CriterionPass StandardFailure SignalTest Method

Domain Label Accuracy

Selected examples share the same domain label as the golden input in >= 95% of test cases

Example domain mismatches golden domain in > 5% of cases; downstream classifier accuracy degrades

Run golden dataset through example selector, compare selected-example domains to golden labels, compute exact-match rate

Example Relevance Score

Mean cosine similarity between input embedding and selected example embeddings exceeds 0.80 threshold

Mean similarity below 0.65; selected examples are semantically distant from the target input

Embed input and selected examples with same model, compute pairwise cosine similarity, aggregate across dataset

Downstream Classification Impact

Few-shot classifier using selected examples achieves >= 2% accuracy gain over random example selection baseline

Selected examples produce lower or equal accuracy vs. random baseline; selector adds cost without benefit

Run classifier with selected examples vs. random examples on held-out test set, compare F1 scores with statistical significance test

Selection Diversity

Selected examples span at least 2 distinct sub-categories or facets within the target domain for each input

All selected examples are near-duplicates or share identical sub-category; no coverage of domain variation

Cluster selected examples by sub-category label, count unique clusters per input, compute mean across dataset

Confidence Calibration

Selector confidence score correlates with downstream classification accuracy at Spearman ρ >= 0.70

High-confidence selections produce low-accuracy classifications; confidence is uninformative or misleading

Bin selections by confidence decile, compute mean downstream accuracy per bin, calculate rank correlation

Abstention Rate on OOD Inputs

Selector returns empty set or explicit rejection for >= 90% of out-of-distribution inputs

Selector confidently returns examples for OOD inputs; forces incorrect domain classification downstream

Feed inputs from held-out domains, verify selector returns no examples or rejection signal, compute abstention rate

Latency Budget Compliance

95th percentile selection latency <= 200ms for single-input example retrieval

P95 latency exceeds 500ms; selector becomes bottleneck in real-time routing pipeline

Measure end-to-end selection time across 1000 requests under load, compute P50/P95/P99 latency

Edge Case: Ambiguous Inputs

Selector returns examples from all plausible domains or abstains when input spans domain boundaries

Selector picks single domain confidently for ambiguous multi-domain input; downstream routing is silently wrong

Curate 20 ambiguous inputs with known multi-domain labels, verify selector output includes all plausible domains or abstains

ADAPTATION OPTIONS

Adapt This Prompt

How to adapt

Start with the base prompt and a small hand-curated example bank of 20-30 labeled domain pairs. Use a simple string-match fallback if the model returns an invalid index. Skip the full eval harness initially—just log classification accuracy on a spreadsheet.

Modify the prompt to accept a flat list instead of a structured taxonomy:

code
[EXAMPLES] = [
  {"input": "...", "domain": "billing"},
  {"input": "...", "domain": "technical"}
]

Watch for

  • Model selecting examples that look similar but map to the wrong domain
  • Empty example banks causing random selection
  • Token limits if you stuff too many examples into the prompt
Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.