Evidence Weight Aggregation Prompt for Multiple Signals

This prompt is for RAG pipeline architects and search ranking engineers who need to combine multiple independent evidence quality signals into a single, defensible ranking score. The job-to-be-done is aggregation: you already have sub-scores for relevance, recency, authority, specificity, and query alignment—likely produced by upstream scoring prompts or retrieval systems—and you need a final fused score that respects configurable signal importance. The ideal user is someone integrating this into a production re-ranking step where downstream answer generation or citation selection depends on a stable, explainable evidence ordering.

Use this prompt when you have heterogeneous signal sources with different scales or distributions and you need normalized aggregation with auditability. It is particularly valuable when signal importance varies by use case—for example, recency might dominate in news search while authority dominates in regulatory research. The prompt includes sensitivity analysis checks that detect when one signal overwhelms others, and dominance detection that flags cases where a single sub-score effectively determines the ranking. Do not use this prompt when you have only one signal, when sub-scores are already on a unified calibrated scale, or when you need raw score normalization without weighted aggregation. It is also not a replacement for upstream scoring prompts that generate the individual sub-scores.

Before wiring this into production, ensure each upstream sub-score is well-calibrated and that you have defined your signal importance weights based on domain requirements, not arbitrary defaults. The prompt expects structured inputs with explicit sub-scores per passage. If your upstream scorers produce inconsistent or unscaled outputs, normalize them before aggregation. For high-stakes domains like legal or medical evidence ranking, always include human review of the aggregated scores and the sensitivity analysis output before using the ranking to generate user-facing answers. The next section provides the copy-ready prompt template you can adapt.

The prompt below accepts a set of pre-computed sub-scores for a single evidence passage—relevance, recency, authority, specificity, and query alignment—and produces an aggregated weight with a detailed breakdown. It is designed to be wired into a re-ranking step of a RAG pipeline where multiple scoring models have already produced raw signals. The prompt forces the model to explain its aggregation logic, detect when one signal dominates the final score, and flag sensitivity risks where small changes in a sub-score would flip the ranking order.

textCopy
You are an evidence scoring aggregator for a retrieval-augmented generation pipeline. Your job is to combine multiple evidence quality sub-scores into a single aggregated weight and explain your reasoning.

## INPUT
- Passage ID: [PASSAGE_ID]
- Passage text: [PASSAGE_TEXT]
- Query: [QUERY]
- Relevance score (0.0–1.0): [RELEVANCE_SCORE]
- Recency score (0.0–1.0): [RECENCY_SCORE]
- Authority score (0.0–1.0): [AUTHORITY_SCORE]
- Specificity score (0.0–1.0): [SPECIFICITY_SCORE]
- Query-alignment score (0.0–1.0): [QUERY_ALIGNMENT_SCORE]
- Signal importance weights: [SIGNAL_IMPORTANCE_WEIGHTS]
  - relevance_weight: [RELEVANCE_WEIGHT]
  - recency_weight: [RECENCY_WEIGHT]
  - authority_weight: [AUTHORITY_WEIGHT]
  - specificity_weight: [SPECIFICITY_WEIGHT]
  - query_alignment_weight: [QUERY_ALIGNMENT_WEIGHT]
- Aggregation method: [AGGREGATION_METHOD]  // one of: weighted_sum, weighted_product, harmonic_mean
- Sensitivity threshold: [SENSITIVITY_THRESHOLD]  // e.g., 0.05 for flagging score changes that would alter ranking

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "passage_id": string,
  "aggregated_weight": number,  // 0.0–1.0
  "aggregation_method": string,
  "factor_contributions": {
    "relevance_contribution": number,
    "recency_contribution": number,
    "authority_contribution": number,
    "specificity_contribution": number,
    "query_alignment_contribution": number
  },
  "dominance_detected": boolean,
  "dominant_signal": string | null,  // name of signal if one contributes >50% of total weight
  "dominance_explanation": string | null,
  "sensitivity_analysis": {
    "most_sensitive_signal": string,
    "score_change_to_flip_tier": number | null,  // how much this signal must change to move passage up/down a relevance tier
    "tier_boundary_proximity": "high" | "medium" | "low"  // how close the aggregated weight is to a tier boundary
  },
  "aggregation_notes": string,  // brief explanation of the aggregation result
  "quality_flags": string[]  // any concerns: ["single_signal_dominance", "high_sensitivity", "low_confidence_input", "missing_signal"]
}

## CONSTRAINTS
1. Apply the specified aggregation method exactly. Do not substitute.
2. If any sub-score is missing or null, set aggregated_weight to null and list "missing_signal" in quality_flags.
3. Flag dominance when any single factor_contribution exceeds 50% of the aggregated_weight.
4. For sensitivity analysis, calculate how much the most_sensitive_signal must change to cross the nearest tier boundary. Use tier boundaries at 0.0–0.3 (low), 0.31–0.7 (medium), 0.71–1.0 (high).
5. If the passage text is empty or clearly unrelated to the query, set aggregated_weight to 0.0 and explain in aggregation_notes.
6. Do not invent sub-scores. Use only the values provided.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]  // "low" | "medium" | "high" — if high, add explicit human-review recommendation in aggregation_notes

Adapt this template by adjusting the [SIGNAL_IMPORTANCE_WEIGHTS] to reflect your domain priorities. For legal document retrieval, authority_weight might dominate. For news summarization, recency_weight should be elevated. The [AGGREGATION_METHOD] parameter lets you switch between weighted_sum for linear combination, weighted_product for multiplicative interaction where a single zero score collapses the result, or harmonic_mean to penalize variance across signals. The [SENSITIVITY_THRESHOLD] controls how aggressively the model flags passages that could shift tiers with minor score changes—set this lower (0.02–0.03) for high-stakes ranking where stability matters. Replace [EXAMPLES] with 2–3 worked examples showing correct aggregation for passages at different quality levels, including one edge case where a signal is missing or a passage is clearly irrelevant. Set [RISK_LEVEL] to "high" when the downstream system makes automated decisions from ranked evidence without human review; this instructs the model to include explicit cautionary language in aggregation_notes.

The Evidence Weight Aggregation prompt is not a standalone tool; it is a scoring node inside a retrieval pipeline. Wire it after your multi-signal scorers (relevance, recency, authority, specificity) have produced raw sub-scores for each passage. The prompt consumes a structured input object containing per-passage sub-scores and signal importance weights, and it returns an aggregated final score with a sensitivity analysis. In a typical RAG harness, this prompt runs as a map-reduce step: map over each passage to compute its aggregated weight, then sort the scored passages before passing the top-k to the answer generation prompt. Because this prompt performs arithmetic reasoning over multiple numerical inputs, prefer a model with strong quantitative reasoning capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) rather than a lightweight model optimized for simple classification.

Validation is the first line of defense. Before calling the prompt, validate that all sub-scores are present, numeric, and within the expected range (e.g., 0.0 to 1.0). After receiving the output, run a structural validator that checks: (1) the aggregated_score field is a number between 0.0 and 1.0, (2) the sensitivity_analysis object contains a dominant_signal field and a dominance_detected boolean, (3) each sub-score contribution is accounted for in the factor_contributions map, and (4) the confidence_interval has both lower and upper bounds. If validation fails, retry once with the same input and a stronger constraint instruction appended to the prompt. If the retry also fails, log the passage ID and raw sub-scores for manual review rather than silently passing a malformed score downstream. For high-stakes domains (legal, medical, financial), add a human review queue for any passage where dominance_detected is true and the dominant signal accounts for more than 70% of the final weight.

Logging and observability are critical because aggregated scores drive downstream answer selection. Log the full input sub-scores, the aggregated output, the model version, and the prompt template hash for every invocation. Store these in a structured format (JSON) in your observability platform so you can trace why a specific passage was ranked above another. When debugging ranking anomalies, compare the logged factor_contributions against your expected signal importance weights. A common failure mode is the model ignoring your explicit signal importance configuration and applying its own implicit weighting. To catch this, add an eval step that computes the expected contribution of each signal (sub-score × importance weight) and compares it to the model's reported factor_contributions. Flag any invocation where the deviation exceeds 0.15 for any single factor. For cost-sensitive pipelines, cache the prompt prefix containing the scoring rubric and output schema; only the per-passage sub-score object changes between calls, making this prompt highly cache-friendly when using a provider that supports prefix caching.

When integrating this prompt into a multi-source retrieval pipeline, normalize raw scores from different retrievers (vector, keyword, hybrid) before passing them as sub-scores. The prompt expects sub-scores on a consistent 0.0–1.0 scale. If your retrievers produce scores on different scales (e.g., cosine similarity 0.7–1.0, BM25 0–50), apply min-max normalization per source before aggregation. Do not rely on the model to handle incompatible score distributions; it will produce misleading aggregated weights. For production deployments, wrap the prompt call in a circuit breaker that falls back to a simple weighted average if the model API is unavailable or consistently returns invalid outputs. The fallback should be deterministic and auditable: compute sum(sub_score[i] × importance_weight[i]) / sum(importance_weights) and log that the fallback was used. This ensures your RAG pipeline degrades gracefully rather than failing entirely when the scoring model is unreachable.