Composite Score Aggregation Prompt Template

This prompt is designed for evaluation engineers and AI pipeline architects who need to combine multiple independent dimension scores into a single, defensible quality metric. The primary job-to-be-done is automating the final step of a multi-dimensional LLM judge evaluation: taking scores for correctness, groundedness, style, and safety, and producing a weighted, normalized composite score that aligns with human holistic judgments. Use this when you have already run separate evaluation passes and need a consistent, auditable aggregation layer before feeding results into dashboards, regression tests, or model selection decisions.

Do not use this prompt as a substitute for the individual dimension evaluations themselves. It assumes the input scores are already valid and calibrated. It is not designed to re-evaluate raw outputs, detect hallucinations, or perform pairwise comparisons. The ideal user has a structured JSON array of per-dimension scores with defined scales and needs aggregation logic that handles weighting rules, dimension interaction effects (e.g., a factual error caps the maximum composite score), and normalization across judges or rubric versions. The prompt requires you to supply the raw scores, dimension definitions, weighting scheme, and any interaction or capping rules as explicit context.

Before wiring this into a production pipeline, validate the aggregated scores against a golden set of human holistic judgments. The most common failure mode is dimension weightings that don't reflect real-world quality trade-offs, leading to composites that pass the math but fail human review. Start with equal weights, calibrate against at least 50 human-scored examples, and only then introduce interaction rules or non-linear caps. If your use case involves high-stakes decisions—hiring, loan underwriting, clinical safety—add a human approval gate after aggregation and log every composite score with its input dimensions for auditability.

This template provides a production-ready prompt for combining individual dimension scores into a weighted composite score. It is designed for evaluation pipelines where multiple quality dimensions—such as accuracy, groundedness, instruction adherence, and format compliance—must be collapsed into a single metric for ranking, gating, or reporting. The prompt handles weight application, score normalization, dimension interaction effects, and confidence-weighted aggregation. It also requires the judge to output its aggregation rationale, making the composite score auditable and debuggable.

textCopy
You are an evaluation aggregation judge. Your task is to combine multiple dimension scores into a single composite quality score using the provided aggregation rules.

## INPUT

[INPUT]

## AGGREGATION CONFIGURATION

Weights:
[WEIGHTS]

Normalization Rules:
[NORMALIZATION_RULES]

Dimension Interaction Rules:
[INTERACTION_RULES]

Minimum Dimension Thresholds:
[MINIMUM_THRESHOLDS]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "composite_score": number,
  "score_range": { "min": number, "max": number },
  "dimension_scores_used": [
    {
      "dimension": string,
      "raw_score": number,
      "normalized_score": number,
      "weight": number,
      "weighted_contribution": number,
      "threshold_met": boolean
    }
  ],
  "interaction_adjustments": [
    {
      "dimensions_involved": [string, string],
      "adjustment_type": string,
      "adjustment_value": number,
      "rationale": string
    }
  ],
  "aggregation_method": string,
  "confidence_level": "high" | "medium" | "low",
  "uncertainty_sources": [string],
  "aggregation_rationale": string
}

## CONSTRAINTS

[CONSTRAINTS]

## EXAMPLES

[EXAMPLES]

## INSTRUCTIONS

1. Parse all dimension scores from the input.
2. Apply normalization rules to bring all scores to a common scale.
3. Check each dimension against its minimum threshold. Flag any dimension that falls below threshold.
4. Apply dimension interaction rules to adjust scores where dimensions influence each other.
5. Compute the weighted composite score using the formula: sum(normalized_score * weight) + interaction_adjustments.
6. If any required dimension is missing or below threshold, set confidence_level to "low" and explain in uncertainty_sources.
7. If interaction rules produce adjustments exceeding [MAX_ADJUSTMENT_CAP], cap the adjustment and note it in the rationale.
8. Output the result in the exact JSON schema specified above.

To adapt this template, replace each square-bracket placeholder with your specific configuration. The [WEIGHTS] placeholder should contain a mapping of dimension names to their numeric weights, ensuring they sum to 1.0 or 100. The [NORMALIZATION_RULES] placeholder defines how to map different score ranges to a common scale—for example, converting a 1–5 Likert score and a 0–100 percentage to a unified 0–1 range. The [INTERACTION_RULES] placeholder specifies how dimensions affect each other, such as capping the composite score when groundedness falls below a threshold regardless of other dimension scores. The [MINIMUM_THRESHOLDS] placeholder defines per-dimension floor values below which the composite score should be penalized or invalidated. The [CONSTRAINTS] placeholder should include output limits like maximum adjustment caps, required dimension presence, and rounding precision. The [EXAMPLES] placeholder should contain at least two worked examples showing input dimension scores, the aggregation calculation, and the expected output JSON. Before deploying, validate this prompt against a golden dataset of human holistic judgments to calibrate weights and thresholds, and implement automated checks that the output JSON matches the schema exactly.

The composite score aggregation prompt is not a standalone artifact—it is a processing step that sits between dimension-level judges and the final quality metric your application consumes. In a typical harness, you run multiple dimension-specific evaluators first (e.g., factual accuracy, groundedness, instruction adherence), collect their individual scores and justifications, then feed that structured output into this aggregation prompt. The harness is responsible for assembling the input payload, enforcing the aggregation schema, validating the output, and deciding what happens when aggregation fails or produces anomalous results.

Wire this prompt into your evaluation pipeline as a post-scoring aggregation stage. The input should be a structured JSON object containing dimension scores, optional weights, and any dimension-level justifications or confidence flags. Use a strict output schema that requires the final composite score, the aggregation method used, a breakdown of how each dimension contributed, and a flag for dimension interactions that influenced the final score. Validate the output programmatically before accepting it: check that the composite score falls within the expected range, that all required dimensions appear in the breakdown, that weights sum to 1.0 (or are explicitly marked as unnormalized), and that any interaction flags reference dimensions that actually exist in the input. If validation fails, retry with the same input and a stronger constraint instruction appended to the prompt—but cap retries at three attempts before escalating to a human reviewer or falling back to a simple weighted average computed in application code.

For high-stakes evaluation pipelines—such as model release gates, compliance audits, or customer-facing quality guarantees—add a human review trigger when the aggregation output shows high variance across dimensions, when dimension interactions are flagged as significant, or when the composite score falls within a narrow boundary region near a critical threshold. Log every aggregation call with the input dimension scores, the raw aggregation output, the validation result, and any retry attempts. This trace data is essential for debugging aggregation drift, recalibrating weights, and proving to auditors that your composite scores are reproducible. Avoid using this prompt for real-time scoring in latency-sensitive paths unless you've benchmarked the aggregation step and confirmed it meets your SLA; in those cases, consider caching aggregation results for common dimension score patterns or pre-computing composite scores for known input distributions.