Confidence Score Explanation Prompt

This prompt is for post-hoc audit and debugging workflows where you need a human-readable, evidence-based explanation of why a classifier produced a specific confidence score. The ideal user is an engineer, QA analyst, or operations team member investigating a specific classification decision in an observability dashboard, an audit trail generator, or a debugging tool. The required context is a completed classification event: you must already have the original input text, the predicted label, and the numerical confidence score. The prompt's job is to close the gap between a single number and a verifiable rationale by citing supporting evidence from the input, identifying missing information that lowered confidence, and producing an explanation that aligns with actual model behavior.

This is not a prompt for performing classification. Do not use it to classify new inputs or to generate a confidence score. It assumes classification has already happened. It is also not a substitute for a calibration plot or a statistical measure of model uncertainty. Use it when a human needs to understand a specific decision at a granular level—for example, when a high-priority support ticket was routed to the wrong queue with 92% confidence, or when a compliance auditor requires a traceable reason for an automated content flag. The prompt is designed to be wired into a post-processing step in a classification pipeline, not into the real-time routing path itself.

Before using this prompt, ensure you have a reliable way to capture the input, label, and score tuple at the point of classification. If your system cannot surface this data for a given decision, the prompt cannot function. For high-risk domains, the explanation output should be treated as a draft for human review, not as a final audit record. The explanation is only as faithful as the model's ability to introspect its own behavior, which is imperfect. Always pair this prompt with an evaluation step that checks for explanation faithfulness, such as verifying that cited evidence actually exists in the input and that the explanation does not introduce hallucinated details. If you need a real-time guardrail that blocks low-confidence classifications before they reach a user, use the Low-Confidence Routing Fallback Prompt instead.

This prompt template is designed for audit and debugging workflows where you need to understand why a classifier assigned a particular confidence score. It forces the model to ground its explanation in the provided evidence rather than generating plausible-sounding but unfaithful rationalizations. Use this when you need to debug classification failures, build audit trails for compliance reviewers, or verify that your classifier is attending to the right signals instead of spurious correlations.

textCopy
You are an audit assistant explaining classification decisions. Your task is to produce a faithful, evidence-grounded explanation of why a classification received its confidence score.

## INPUT
- Classification Result: [CLASSIFICATION_LABEL]
- Confidence Score: [CONFIDENCE_SCORE]
- Input Text: [INPUT_TEXT]
- Classification Taxonomy: [TAXONOMY_DEFINITIONS]
- Model Behavior Notes (optional): [MODEL_BEHAVIOR_NOTES]

## INSTRUCTIONS
1. Identify the specific spans, terms, or patterns in the input text that support the assigned classification.
2. Identify any conflicting signals, missing information, or ambiguous spans that reduced confidence.
3. Explain how each piece of evidence contributed to or detracted from the final confidence score.
4. If the confidence score seems inconsistent with the evidence, flag this explicitly.
5. Do not invent evidence that does not appear in the input text.
6. Do not speculate about model internals or training data.

## OUTPUT FORMAT
Return a JSON object with this exact schema:
{
  "classification": "string (the assigned label)",
  "confidence_score": number,
  "supporting_evidence": [
    {
      "span": "string (exact text from input)",
      "contribution": "positive | negative",
      "explanation": "string (how this affected confidence)"
    }
  ],
  "missing_information": [
    "string (what information would have increased confidence)"
  ],
  "consistency_flag": "consistent | inconsistent | borderline",
  "consistency_explanation": "string (only if inconsistent or borderline)",
  "summary": "string (one-paragraph plain-English explanation suitable for audit review)"
}

## CONSTRAINTS
- Every supporting_evidence span must be a verbatim substring of the input text.
- If no clear evidence exists, return an empty supporting_evidence array and explain why.
- The summary must be understandable by a non-technical reviewer.
- Flag inconsistency if the evidence appears to contradict the assigned label or confidence level.

Adaptation guidance: Replace [CLASSIFICATION_LABEL] and [CONFIDENCE_SCORE] with the actual outputs from your classifier. [TAXONOMY_DEFINITIONS] should contain the label definitions your classifier uses—this constrains the explanation to your actual taxonomy rather than the model's assumptions. [MODEL_BEHAVIOR_NOTES] is optional but useful for providing calibration context, such as 'this classifier tends to be overconfident on short inputs' or 'confidence scores below 0.7 are unreliable.' The output schema enforces span-level grounding, which is critical for auditability: every piece of evidence must be traceable to the original text.

What to do next: After generating explanations, run eval checks for faithfulness—verify that each cited span actually appears in the input and that the explanation doesn't hallucinate evidence. Compare explanations against a held-out set where you know the ground-truth reasons for classification. If explanations frequently cite irrelevant spans or miss obvious signals, your classifier may be attending to spurious features and needs retraining or prompt adjustment. For high-stakes audit workflows, always route explanations through human review before they become part of a compliance record.

This prompt is designed for offline audit and debugging, not for real-time user-facing explanations. Wire it into a batch processing pipeline that consumes classification logs: for each classification event, feed the original input, the assigned label, the confidence score, and the model's top-k alternatives into the prompt. The output is a structured explanation object that must be validated before it enters your audit trail. Because this prompt is used to justify downstream routing decisions, treat its output as evidence that requires verification, not as a self-certifying artifact.

Implement a validation layer that checks the explanation's faithfulness before storage. At minimum, confirm that every cited evidence span appears verbatim in the original input, that the explanation references the correct label and score, and that the missing_information field contains only information genuinely absent from the input. Use a secondary LLM call with a strict faithfulness eval prompt to compare the explanation against the input and the model's token-level logprobs if available. If the faithfulness check fails, retry the explanation prompt once with the validation errors appended as additional constraints. If the retry also fails, flag the record for human review and store the raw classification data alongside the failed explanation attempt.

Choose a model with strong instruction-following and structured output capabilities for this task. The explanation prompt requires precise extraction of evidence spans and disciplined reasoning about missing information, which degrades rapidly on smaller or less capable models. Log the full prompt, response, validation results, and any human review decisions to an append-only audit store. Never use the explanation output to automatically override the original classification decision without human approval. The next step after implementing this harness is to run the explanation prompt against a golden set of known-correct and known-incorrect classifications to calibrate your faithfulness eval thresholds before production deployment.