Rubric with Confidence Interval Output Prompt Template

This prompt is for evaluation engineers and AI platform teams who need automated scoring with built-in uncertainty quantification. The job-to-be-done is producing a model-graded evaluation rubric that requires judges to output not just a numeric score, but a score range, a confidence level, and a structured explanation of uncertainty sources. This is essential when downstream decisions depend on knowing whether a score of 7/10 means 'definitely good enough to ship' or 'probably good enough but the judge is unsure.' The ideal user is someone running automated evaluation pipelines who has already observed that point-estimate scores can be misleading and needs richer signal before gating a release, selecting a model, or accepting generated content into production.

Use this prompt when your evaluation workflow requires calibration-aware scoring. This means you need to verify over time that confidence intervals actually contain the true score at the expected rate—for example, a 90% confidence interval should contain the ground-truth score roughly 90% of the time across a test set. The prompt template includes calibration check instructions that make this verifiable. It also forces judges to articulate why they are uncertain, which helps diagnose whether uncertainty comes from ambiguous instructions, conflicting evidence, edge cases, or genuine task difficulty. This diagnostic layer is what separates a useful confidence interval from a vague hedge.

Do not use this prompt when you only need a fast binary pass/fail gate with no uncertainty analysis, or when your evaluation pipeline cannot store and analyze interval data over time. The prompt adds token overhead and reasoning latency because it requires the judge to produce a range, confidence level, and uncertainty source list alongside the score. If your system only consumes a single numeric value and discards everything else, start with a simpler rubric prompt first. Also avoid this prompt when your human baseline annotations don't include confidence judgments—you need calibrated human intervals to validate that the model's uncertainty aligns with human uncertainty. Without that alignment check, the confidence numbers are ungrounded and potentially misleading. After adapting this prompt, wire the output into an eval harness that logs score distributions over time and triggers a recalibration review when observed interval coverage drifts outside acceptable bounds.

This prompt template is the core contract for an LLM judge that must output a confidence interval instead of a single point score. It is designed for evaluation pipelines where uncertainty quantification is a first-class requirement—such as when automated scores feed into high-stakes decisions, model release gates, or calibration workflows. The template enforces a structured output schema that includes a lower bound, an upper bound, a confidence level, and a list of uncertainty sources. Use this template when you need to know not just what the score is, but how sure the judge is about that score.

Below is the copy-ready prompt template. Every placeholder in square brackets must be replaced before use. The [RUBRIC] placeholder expects the full evaluation rubric with dimensions, levels, and anchors. The [OUTPUT_TO_EVALUATE] placeholder expects the model response or artifact being scored. The [CONFIDENCE_LEVEL] placeholder should be set to a target confidence level, such as 90% or 95%, which the judge will use to calibrate its interval width. The [CALIBRATION_INSTRUCTIONS] placeholder is where you inject any domain-specific guidance on how to map uncertainty to interval width.

textCopy
You are an expert evaluator trained to apply rubrics and quantify your uncertainty. Your task is to score the provided output against the rubric and produce a confidence interval that reflects your uncertainty about the true score.

## RUBRIC
[RUBRIC]

## OUTPUT TO EVALUATE
[OUTPUT_TO_EVALUATE]

## INSTRUCTIONS
1. Apply the rubric carefully and determine the most likely score for each dimension.
2. For each dimension, identify sources of uncertainty that could cause the true score to differ from your best estimate. Consider ambiguity in the output, rubric boundary cases, missing information, and subjective interpretation.
3. Produce a confidence interval for each dimension score. The interval should be wide enough that you are [CONFIDENCE_LEVEL] confident the true score falls within it.
4. Aggregate dimension intervals into an overall score interval using the aggregation rules specified in the rubric. If no aggregation rules are provided, use a weighted average where weights are proportional to dimension importance.
5. Output your evaluation in the exact JSON schema provided below.

[CALIBRATION_INSTRUCTIONS]

## OUTPUT SCHEMA
{
  "overall_score_interval": {
    "lower_bound": <number>,
    "upper_bound": <number>,
    "confidence_level": <number between 0 and 1>,
    "most_likely_score": <number>
  },
  "dimension_scores": [
    {
      "dimension_name": "<string>",
      "interval": {
        "lower_bound": <number>,
        "upper_bound": <number>,
        "most_likely_score": <number>
      },
      "uncertainty_sources": [
        {
          "source_type": "<ambiguity|boundary|missing_info|subjectivity|other>",
          "description": "<string explaining why this source creates uncertainty>",
          "severity": "<low|medium|high>"
        }
      ]
    }
  ],
  "global_uncertainty_notes": [
    "<string describing cross-dimension uncertainty factors>"
  ]
}

## CONSTRAINTS
- Every dimension in the rubric must appear in `dimension_scores`.
- `confidence_level` must be exactly [CONFIDENCE_LEVEL].
- `lower_bound` must be less than or equal to `most_likely_score`, and `most_likely_score` must be less than or equal to `upper_bound`.
- If you have no uncertainty for a dimension, set `lower_bound` equal to `upper_bound` and provide an empty `uncertainty_sources` array.
- Do not invent dimensions not present in the rubric.

After copying this template, validate that your [RUBRIC] placeholder contains explicit scoring levels with numeric ranges. If your rubric uses qualitative labels like "Excellent" or "Poor," add a mapping to numeric scores inside the rubric itself before passing it to the judge. The [CALIBRATION_INSTRUCTIONS] field is critical for production use: without it, judges tend to produce intervals that are either too narrow (overconfident) or too wide (uninformative). Start with a calibration instruction like "If the output falls exactly on a rubric boundary, widen the interval by at least 0.5 points on each side" and iterate based on observed coverage rates. For high-risk domains, always pair this prompt with a calibration harness that tracks whether true scores fall within the predicted intervals at the expected rate over time.

This prompt is designed to be called as a scoring step inside an automated evaluation pipeline, not as a one-off chat interaction. The typical harness sends a candidate output and its source context to the model, receives a JSON response containing a score range, confidence level, and uncertainty sources, and then validates that response before storing it. Because the output includes a confidence interval, the harness must treat the score as a distribution, not a point estimate. Downstream systems should use the lower bound for gating decisions and the interval width for triggering human review.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro). Enable structured output mode or JSON mode with the exact schema from the prompt. The harness should validate the response against a strict schema: score_lower_bound and score_upper_bound must be numbers within the defined scale, confidence_level must be between 0 and 1, and uncertainty_sources must be a non-empty array of strings from the allowed enum. If validation fails, retry once with the validation error injected into the prompt as additional context. If the second attempt also fails, log the failure and escalate for human scoring rather than silently accepting a malformed score.

Calibration is the critical production concern. Track whether the true score (from human annotators or ground truth) falls within the model's confidence interval at the expected rate. If the model claims 90% confidence but the interval contains the true score only 70% of the time, the judge is overconfident and the intervals are too narrow. Build a calibration dashboard that plots observed coverage against expected confidence, and set a drift threshold that triggers recalibration. For high-stakes domains, require human review whenever the interval width exceeds a defined threshold or when the confidence level drops below a minimum bar (e.g., 0.7). Store the raw prompt, response, validation status, and calibration metadata in your evaluation log for audit and debugging.