Multi-Label Confidence Scoring Prompt

This prompt is for platform teams building classification systems where a single input can belong to multiple categories simultaneously. Instead of forcing a single-label selection, it assigns independent confidence scores to each applicable label. Use this when your routing middleware needs to dispatch one input to multiple downstream handlers, when you need to surface all relevant categories with their certainty levels, or when label co-occurrence patterns carry signal. The ideal user is an AI infrastructure engineer or technical decision maker who owns a routing or triage system and needs to move beyond single-intent classification. You must have a defined label taxonomy, an input to classify, and optionally a set of examples or context that clarifies label boundaries.

Do not use this prompt for simple single-intent classification where only one label should win. If your system requires a single, mutually exclusive label per input, a standard classification prompt with a softmax-style output is more appropriate and less prone to ambiguity. This prompt is also not a replacement for intent decomposition. If a user input contains multiple distinct requests that must be executed sequentially, use a multi-intent decomposition prompt first, then apply multi-label confidence scoring to each sub-intent. Before wiring this into production, define your label taxonomy carefully. Overlapping or poorly differentiated labels will produce noisy confidence scores and make threshold-based routing unreliable. Validate your taxonomy with real inputs and adjust label definitions until inter-label confusion is minimized.

The next step is to copy the prompt template, adapt the placeholders to your taxonomy, and run it against a golden dataset of inputs with known label assignments. Measure per-label precision and recall at different confidence thresholds, and watch for systematic overconfidence on common labels and underconfidence on rare ones. If your production system routes based on these scores, log every routing decision with its confidence vector so you can audit for silent misroutes later.

This prompt template is designed for platform teams building multi-label classification systems where a single input can belong to multiple categories simultaneously. Unlike single-label classifiers that force a choice, this prompt instructs the model to evaluate every applicable label independently and assign a confidence score to each. The output is structured JSON that can be consumed directly by downstream routing logic, threshold-based dispatch, or human review queues. The template uses square-bracket placeholders that you must replace with your specific taxonomy, scoring rubric, and output constraints before use.

textCopy
You are a multi-label classification system. Your task is to evaluate the provided input against a predefined set of labels and assign an independent confidence score to each label that applies.

## INPUT
[INPUT]

## LABEL SET
Evaluate the input against each of the following labels independently. A single input may match zero, one, or multiple labels.

[LABEL_DEFINITIONS]

## SCORING RUBRIC
For each label that applies, assign a confidence score from 0.0 to 1.0 using this rubric:
- 0.9–1.0: The input unambiguously matches this label with clear, direct evidence.
- 0.7–0.89: The input likely matches this label, but some minor ambiguity or missing detail exists.
- 0.5–0.69: The input partially matches this label, but significant uncertainty remains.
- 0.3–0.49: The input has only weak or tangential association with this label.
- 0.0–0.29: The input does not match this label, or evidence is absent. Do not include labels scored below [MIN_CONFIDENCE_THRESHOLD] in the output.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
Return ONLY valid JSON. No markdown fences, no commentary. Use this exact structure:
{
  "classifications": [
    {
      "label": "string",
      "confidence": number,
      "evidence": "string (brief quote or paraphrase from input supporting this label)",
      "uncertainty_note": "string (explain what information is missing or ambiguous, or null if confidence is high)"
    }
  ],
  "no_match": boolean (true if no labels met the minimum threshold),
  "ambiguous_labels": ["string"] (list of labels where confidence is between 0.5 and 0.69, indicating borderline cases that may need review)
}

## EXAMPLES
[EXAMPLES]

## INSTRUCTIONS
1. Evaluate each label independently. Do not compare labels against each other.
2. Assign a confidence score to every label that meets or exceeds [MIN_CONFIDENCE_THRESHOLD].
3. Include specific evidence from the input for each assigned label.
4. For scores below 0.7, explain what information is missing or ambiguous in the uncertainty_note field.
5. If no label meets the minimum threshold, set no_match to true and return an empty classifications array.
6. List any labels scored between 0.5 and 0.69 in the ambiguous_labels array for potential human review.
7. Do not invent labels not present in the LABEL SET.
8. Return only the JSON object. No other text.

To adapt this template for your system, replace the placeholders with concrete values. [INPUT] should be replaced with your actual user query, document, or message at runtime. [LABEL_DEFINITIONS] must contain your complete taxonomy with clear, mutually understandable descriptions for each label—include edge-case guidance and examples of what does and does not count for each label. [CONSTRAINTS] should specify any domain-specific rules, such as label co-occurrence restrictions, required disclaimers, or regulatory handling notes. [MIN_CONFIDENCE_THRESHOLD] should be set to your operational cutoff (commonly 0.3 or 0.5 depending on your tolerance for false positives). [EXAMPLES] should include at least three few-shot demonstrations covering single-label, multi-label, and no-match scenarios with correct JSON outputs. For high-stakes domains such as healthcare triage, legal intake, or compliance routing, add a [RISK_LEVEL] field and require that low-confidence or ambiguous outputs trigger human review before any automated action is taken.

The Multi-Label Confidence Scoring Prompt is designed to be called as a stateless scoring function within a broader classification service. It accepts a single input and a list of candidate labels, returning independent confidence scores for each applicable label. This design makes it suitable for deployment behind a lightweight API endpoint, a queue worker, or a serverless function where each invocation is self-contained. The prompt does not maintain conversation state, does not call tools, and does not require retrieval-augmented generation (RAG) unless the input itself depends on external context. For most implementations, you will call this prompt synchronously, validate the structured output, apply per-label confidence thresholds, and then dispatch the results to downstream handlers.

Integration pattern: Wrap the prompt in a function with a strict contract: score_labels(input_text: str, candidate_labels: list[str], threshold: float) -> MultiLabelResult. The MultiLabelResult should contain a list of scored labels, each with label, confidence (0.0–1.0), and rationale fields. After receiving the model response, validate the JSON schema before any routing decision occurs. Reject malformed responses and retry once with a repair prompt that includes the original output and a specific format error message. If the retry also fails, log the failure and route the input to a human review queue rather than silently dropping it. For high-throughput systems, consider batching multiple inputs into a single prompt call with clear delimiters, but be aware that batching can cause label leakage between inputs if the model confuses boundaries—always validate per-input label assignments in batched responses.

Threshold routing logic: After validation, apply per-label confidence thresholds to determine which labels are actionable. A common pattern is to use a primary threshold (e.g., 0.7) for automatic routing and a secondary threshold (e.g., 0.4) for low-confidence flagging. Labels scoring above the primary threshold proceed to their assigned workflows. Labels between the secondary and primary thresholds trigger a clarification or confirmation step. Labels below the secondary threshold are discarded but logged for threshold calibration analysis. If no label exceeds the primary threshold, route the input to a fallback handler—either a general-purpose model, a human review queue, or a clarification prompt. Avoid the common mistake of routing on the single highest score when multiple labels are near-threshold; instead, treat all above-threshold labels as co-equal routing signals and let downstream handlers resolve conflicts.

Observability and eval integration: Log every scoring call with the input hash, candidate labels, raw confidence scores, applied thresholds, and final routing decision. This log becomes your audit trail for threshold tuning and failure analysis. Implement eval checks that run offline against a labeled test set: measure per-label precision and recall at your chosen thresholds, track the rate of empty routing decisions (no label above threshold), and monitor label co-occurrence patterns to detect when the model consistently over-assigns or under-assigns specific label pairs. For high-risk domains, add a human-in-the-loop sampling step where a configurable percentage of routing decisions—especially those near threshold boundaries—are reviewed before dispatch. Start with 5-10% sampling and adjust based on observed error rates.