Classification Certainty Rubric Prompt

This prompt is a meta-assessment tool for quality assurance teams and MLOps engineers who need to standardize how classification confidence is evaluated across different models, human reviewers, or system versions. The core job-to-be-done is calibration and audit: you have a classifier that already produced a label and a confidence score, and you need a second, independent assessment of whether that confidence claim is justified according to a defined rubric. This is not a prompt for performing the initial classification. It is a prompt for judging the quality of the classification output itself, producing consistent high, medium, or low certainty labels that can be compared across raters.

Use this prompt when you are onboarding a new classifier and need to establish baseline confidence behavior, when you are auditing an existing production classifier for confidence drift or overconfidence, or when you are running inter-rater reliability studies between human reviewers and automated systems. The prompt requires three inputs to function correctly: the original input text that was classified, the classifier's predicted label and confidence score, and a defined rubric with explicit criteria for each certainty level. Without a concrete rubric, the prompt degenerates into subjective opinion. The rubric must define what evidence, ambiguity patterns, or boundary conditions map to high, medium, and low certainty for your specific domain. A generic rubric will produce inconsistent labels across different classification tasks.

Do not use this prompt when you need the initial classification itself, when you lack a written rubric, or when the classifier's internal confidence score is already well-calibrated and you only need a binary pass/fail threshold check. This prompt adds latency and cost through a second model call; it is valuable when you need auditable, human-readable certainty labels that can be compared across raters for governance, compliance, or reliability engineering purposes. If you are building a real-time routing system, use the Classification Confidence Scoring Prompt Template instead. If you need to decide what action to take on a low-confidence result, use the Low-Confidence Routing Fallback Prompt. This prompt is for assessment, not action.

This prompt template operationalizes a classification certainty rubric by forcing the model to evaluate a classification decision against explicit criteria for evidence sufficiency, ambiguity, and alternative-intent plausibility. Instead of returning a raw numeric score that can drift in meaning across different models or runs, it produces a structured certainty label (High, Medium, Low) with a supporting rationale tied to the rubric. This makes confidence assessments comparable across classifiers, reviewers, and audit cycles—a critical requirement for quality assurance teams standardizing their triage and routing pipelines.

textCopy
You are a classification quality assessor applying a standardized certainty rubric. Your job is to evaluate a classification decision and assign a certainty label, not to reclassify the input.

## INPUT
- Original User Input: [USER_INPUT]
- Classification Label: [CLASSIFICATION_LABEL]
- Classification Rationale: [CLASSIFICATION_RATIONALE]
- Intent Taxonomy Context: [TAXONOMY_CONTEXT]

## CERTAINTY RUBRIC
Apply the following criteria strictly:

**HIGH Certainty**
- The input contains explicit, unambiguous signals that directly match the assigned label.
- No reasonable alternative label could apply.
- All required evidence for the label is present in the input.
- The classification rationale cites specific spans from the input.

**MEDIUM Certainty**
- The input contains signals consistent with the assigned label, but some evidence is implicit or inferred.
- At least one alternative label is plausible but less likely.
- The classification rationale relies partly on reasonable assumptions rather than explicit evidence.
- Missing information could change the classification if it were present.

**LOW Certainty**
- The input contains weak, contradictory, or insufficient signals for the assigned label.
- Multiple alternative labels are equally or more plausible.
- The classification rationale relies heavily on assumptions or external knowledge not present in the input.
- The input is ambiguous, underspecified, or could reasonably be classified differently by another assessor.

## OUTPUT SCHEMA
Return ONLY valid JSON with this exact structure:
{
  "certainty_label": "High" | "Medium" | "Low",
  "rubric_criteria_met": ["List of specific rubric criteria that apply to this case"],
  "evidence_summary": "Brief summary of the evidence supporting the assigned label, citing specific input spans where possible.",
  "alternative_labels_considered": ["List of plausible alternative labels, or empty array if none"],
  "missing_information": ["List of information that, if present, would increase certainty, or empty array if none"],
  "assessor_notes": "Any additional observations about classification quality, ambiguity sources, or edge-case concerns."
}

## CONSTRAINTS
- Do not reclassify the input. Evaluate the given classification only.
- If the classification rationale is missing or empty, default to LOW certainty.
- If the input is empty or unintelligible, set certainty_label to "Low" and explain why in assessor_notes.
- Cite specific evidence from the input. Do not invent facts not present in the input.
- If multiple alternative labels are equally plausible, you MUST assign LOW certainty.

Before integrating this prompt into a production harness, replace each square-bracket placeholder with data from your classification pipeline. [USER_INPUT] should receive the raw, unmodified user text. [CLASSIFICATION_LABEL] and [CLASSIFICATION_RATIONALE] must come from your upstream classifier—if your classifier does not produce a rationale, modify the prompt to note that absence and default to LOW certainty when rationale is missing. [TAXONOMY_CONTEXT] should include the full set of possible labels and their definitions so the assessor can identify plausible alternatives. For high-stakes domains such as healthcare triage or legal intake, add a [RISK_LEVEL] parameter and require human review for any LOW certainty assessment on high-risk inputs. Test this prompt against a golden set of 20–30 labeled examples with known certainty levels before deploying, and monitor inter-rater reliability between this prompt and human assessors during the first two weeks of production use.

The Classification Certainty Rubric Prompt is designed to be used as a standardized evaluation layer, not a one-off chat. Its primary value comes from consistent, repeatable application across multiple classifiers, raters, or model versions. To achieve this, the prompt must be embedded in a harness that controls input formatting, captures structured output, and enables statistical comparison of certainty labels. The core workflow involves sending the same [INPUT_TEXT] and [CLASSIFICATION_OUTPUT] pair to the rubric prompt, potentially multiple times or across different models, and collecting the resulting certainty label and rationale for analysis.

Build the harness to enforce strict input and output contracts. The prompt expects two primary inputs: the original text that was classified and the classification result (label and any accompanying metadata). Your application layer should assemble these into the [INPUT_TEXT] and [CLASSIFICATION_OUTPUT] placeholders. On the output side, validate that the response contains a valid certainty level from the defined set (e.g., 'High', 'Medium', 'Low') and a non-empty rationale string. If the model returns an invalid level or omits the rationale, implement a retry with a stricter reformulation of the prompt, such as 'You MUST select exactly one of: High, Medium, Low.' Log all raw responses, validation failures, and retry attempts for later audit. For inter-rater reliability testing, store each assessment as a record with fields for rater_id (which could be a model version, a human annotator ID, or a temperature seed), input_text_hash, classification_output_hash, certainty_level, and rationale.

After collecting a batch of assessments, the real work begins: measuring agreement. Use Cohen's kappa or Fleiss' kappa (for more than two raters) to quantify inter-rater reliability on the certainty labels. A low kappa score on a specific subset of inputs often reveals that the rubric's criteria are ambiguous for those edge cases. Your harness should therefore support slicing results by metadata, such as the original classifier's confidence score or the input's domain, to pinpoint where the rubric breaks down. Avoid the temptation to treat the rubric prompt's output as ground truth; it is a measurement instrument that itself requires calibration. The next step is to review disagreement cases—where two raters assigned different certainty levels—and refine the rubric's criteria or provide clearer few-shot examples in the prompt until kappa reaches an acceptable threshold for your use case.