Multi-Model Confidence Aggregation Prompt Template

This prompt is designed for platform teams and ML engineers operating ensemble architectures where multiple heterogeneous models generate outputs for the same input. Its core job is to produce a single, defensible aggregated confidence score that accounts for inter-model agreement, disagreement patterns, and individual model calibration curves. Use this prompt when you need a post-generation gating layer that synthesizes signals from several models into one decision-ready metric, rather than relying on a single model's self-reported confidence, which is often poorly calibrated. The ideal user is an engineering lead who already has multiple models returning responses and needs to programmatically decide whether the consensus is strong enough to proceed without human review.

Deploy this prompt in the post-generation gating layer, strictly after all models in the ensemble have returned their responses but before the final output reaches downstream systems or users. The required context includes the original user input, the full output from each model, and any available per-model calibration metadata (such as historical precision-recall curves or expected calibration error). You must also supply a defined output schema so the aggregation result can be parsed and acted upon by your application code. This prompt is not a replacement for model-level confidence scoring; it is a meta-evaluator that consumes those scores and the outputs themselves to produce a higher-quality aggregate signal.

Do not use this prompt when you only have a single model, as there is no inter-model signal to aggregate. Avoid it when the models produce fundamentally different output types that cannot be meaningfully compared (e.g., one model returns structured JSON while another returns free-text narratives with no shared evaluation surface). It is also inappropriate for real-time streaming decisions that cannot tolerate the batch aggregation latency of waiting for all models to complete. Finally, if your models have no calibration history or you cannot provide per-model performance context, the aggregated score will be unreliable and may provide a false sense of security. In those cases, invest in model calibration first, then return to this aggregation pattern.

This prompt template is designed for platform teams running ensemble or multi-model architectures who need a single, structured confidence assessment from multiple model outputs. It ingests the original input, each model's raw output, and any per-model confidence metadata, then produces an aggregated score with disagreement analysis and consensus strength indicators. Use this when you have two or more models generating responses to the same prompt and you need to decide whether the combined output is trustworthy enough to pass downstream. Do not use this for single-model confidence extraction—use the dedicated confidence extraction template instead.

textCopy
You are an expert model evaluator analyzing outputs from multiple AI models that responded to the same input. Your task is to produce an aggregated confidence assessment that combines signals from all models, identifies areas of agreement and disagreement, and provides a calibrated overall confidence score.

## INPUT

### Original User Input
[ORIGINAL_INPUT]

### Model Outputs and Metadata
For each model, you will receive the output and available confidence metadata.

[OUTPUTS_AND_METADATA]

Format: An array of objects, each containing:
- model_id: unique identifier for the model
- output: the raw text output from that model
- confidence_score: optional per-model confidence score (0.0 to 1.0)
- confidence_method: optional description of how the per-model score was derived

## OUTPUT SCHEMA

Return a valid JSON object matching this structure:

```json
{
  "aggregated_confidence": 0.0,
  "confidence_band": "high|medium|low",
  "model_count": 0,
  "agreement_level": "strong|moderate|weak|none",
  "consensus_output": "string or null",
  "disagreement_analysis": {
    "has_critical_disagreement": false,
    "disputed_claims": [],
    "disagreement_severity": "none|minor|major|critical"
  },
  "per_model_assessment": [],
  "calibration_notes": "string",
  "recommendation": "accept|review|reject"
}

Field Definitions

• aggregated_confidence: Weighted confidence score from 0.0 to 1.0, accounting for per-model scores, inter-model agreement, and output consistency.
• confidence_band: "high" if >= 0.8, "medium" if >= 0.5, "low" if < 0.5.
• model_count: Number of models evaluated.
• agreement_level: "strong" if all models substantially agree, "moderate" if majority agrees with minor variations, "weak" if no clear majority, "none" if outputs are contradictory.
• consensus_output: The agreed-upon output text if agreement is strong or moderate, otherwise null.
• disagreement_analysis: Breakdown of where models diverge.
  • has_critical_disagreement: True if models contradict on material facts or conclusions.
  • disputed_claims: Array of specific claims where models disagree, each with the claim text and which models support or oppose it.
  • disagreement_severity: Overall severity of disagreements.
• per_model_assessment: Array of objects for each model with model_id, individual_contribution_weight, and alignment_with_consensus ("aligned", "partial", "divergent").
• calibration_notes: Notes on how the aggregated score was derived, including any adjustments for known model biases or calibration issues.
• recommendation: "accept" if confidence is high and agreement is strong, "review" if moderate confidence or moderate agreement, "reject" if low confidence or critical disagreement.

CONSTRAINTS

[CONSTRAINTS]

Default constraints:

• If fewer than 2 models are provided, return an error explanation instead of an assessment.
• If per-model confidence scores are missing, estimate them from output coherence and completeness.
• Weight models equally unless calibration data suggests otherwise.
• Flag any output that appears truncated, hallucinated, or non-responsive to the original input.
• If the original input is in a regulated domain, escalate the recommendation to "review" regardless of confidence.

EXAMPLES

[EXAMPLES]

RISK LEVEL

[RISK_LEVEL]

Default: medium. For high-risk domains, require human review for any recommendation other than "accept" with strong agreement.

To adapt this template, start by populating [OUTPUTS_AND_METADATA] with a JSON array of your model responses. Each entry should include at minimum the model_id and output fields. If your models already produce confidence scores, include those as well—the aggregation logic will weight them appropriately. For [CONSTRAINTS], override the defaults if you have model-specific calibration data, domain-specific thresholds, or custom weighting rules. The [EXAMPLES] placeholder should contain at least one few-shot example showing a correct aggregation with both agreement and disagreement scenarios. Set [RISK_LEVEL] to "high" if outputs feed into clinical, legal, financial, or safety-critical systems—this will enforce stricter review requirements. After copying the template, validate that your application layer can parse the JSON output and route based on the recommendation field. Test with known agreement and disagreement cases before deploying to production.

The multi-model confidence aggregation prompt is designed to be called after you have collected raw outputs and per-model confidence signals from two or more models. In a typical production harness, you will run this prompt inside a post-processing step—after individual model calls complete but before the final output is surfaced to a user or written to a database. The prompt expects a structured input containing each model's output, its self-assessed confidence, and any available calibration metadata. You should assemble this input programmatically from your model response logs rather than asking a human operator to fill it in.

Wire the prompt into your application as a stateless function that accepts a list of model result objects and returns an aggregated confidence payload. Implement input validation before calling the LLM: check that at least two model results are present, that each result includes a confidence score on a known scale, and that the outputs being compared are semantically equivalent (i.e., they answer the same question or perform the same task). If validation fails, short-circuit with an error code rather than sending malformed input to the aggregation prompt. On the output side, parse the aggregated JSON response and validate that it contains the required fields: aggregated_confidence, inter_model_agreement, disagreement_flags, and consensus_strength. If the model returns malformed JSON, use a structured output repair prompt from the Output Repair and Validation pillar before retrying.

For retry logic, implement a maximum of two retry attempts with exponential backoff if the output fails schema validation or if the aggregated_confidence field falls outside the expected 0.0–1.0 range. Log every aggregation call—including input model scores, the raw LLM response, and the final parsed payload—to your observability platform. This trace data is essential for calibrating the aggregation prompt over time and for debugging cases where the aggregated confidence diverges significantly from human judgment. If the disagreement_flags array contains high-severity items or consensus_strength is low, route the output to a human review queue with the full aggregation payload attached as context for the reviewer.

Model choice matters here. This prompt performs analytical reasoning over numerical scores and textual outputs, so prefer a model with strong reasoning capabilities and reliable JSON output (such as Claude 3.5 Sonnet or GPT-4o with structured output mode enabled). Avoid using smaller or older models that may hallucinate agreement metrics or fail to detect subtle disagreements between model outputs. If your system already uses multiple models for the primary task, consider running the aggregation prompt on a separate, higher-capability model to avoid circular confidence assessment. Set a low temperature (0.0–0.2) to maximize consistency across repeated aggregation calls.

Before deploying, build a small evaluation harness that tests the aggregation prompt against known scenarios: two models that strongly agree, two that strongly disagree, one model with calibrated confidence and one that is overconfident, and edge cases with missing or malformed per-model confidence scores. Measure whether the aggregated confidence and disagreement flags align with your expected outcomes. If this aggregation feeds into an automated gating decision (e.g., blocking low-confidence outputs), run a shadow deployment first and compare the aggregation-based decisions against human reviewer judgments for at least 500 samples before cutting over. Monitor the rate of high disagreement flags and low consensus strength in production—spikes in either metric often indicate a model version change, a data distribution shift, or a prompt regression in one of the upstream models.