Evidence Score Normalization Prompt Across Sources

This prompt is for RAG pipeline architects and search engineers who combine retrieval results from multiple backends—such as vector search, BM25 keyword search, and hybrid retrievers—and need a unified relevance scale to make downstream decisions. Each backend produces scores in its own distribution: cosine similarity might cluster between 0.7 and 0.95, while BM25 scores can span orders of magnitude. Without normalization, a passage scoring 0.8 from one source might be treated as equivalent to a passage scoring 0.8 from another, even when one is a near-perfect match and the other is marginal. The prompt takes raw scores, source metadata, and optional calibration samples, then outputs normalized scores on a consistent scale with cross-source calibration checks and outlier detection.

Use this prompt when you have at least two retrieval sources with incompatible score distributions and you need to merge or compare results before re-ranking, filtering, or answer generation. The prompt requires raw scores per passage, the source identifier for each score, and optional calibration data such as known-relevant and known-irrelevant score samples from each source. Do not use this prompt when you have a single retrieval source with a well-understood score distribution, when you can apply a fixed mathematical transform such as min-max scaling or z-score normalization without model reasoning, or when latency constraints require sub-50ms decisions that a model call cannot meet. For single-source pipelines, a deterministic normalization function in application code is faster, cheaper, and more predictable.

Before deploying this prompt, validate its output against a held-out set of passages with known relevance judgments across sources. Common failure modes include over-normalizing sparse score ranges into a narrow band where all passages appear equally relevant, misidentifying legitimate high-variance sources as outliers, and producing calibration recommendations that drift when source score distributions shift after index updates. Wire the prompt into a batch processing step that runs after retrieval and before re-ranking, with logging that captures raw scores, normalized scores, and any outlier flags for offline analysis. If the normalization output feeds directly into answer generation or user-facing ranking, add a human review gate for the first production week to confirm that cross-source score alignment matches human relevance expectations.

The prompt below is designed to be dropped into a RAG pipeline after retrieval but before evidence selection. It accepts a list of passages from different sources—each with its own raw score distribution—and normalizes them onto a common 0–1 scale. The template includes placeholders for the passages, their source identifiers, raw scores, and any domain-specific calibration rules. Use it when your retrieval architecture combines vector search, BM25, and hybrid retrievers that produce incomparable score ranges.

textCopy
You are an evidence score normalization engine for a multi-source retrieval pipeline. Your job is to convert raw retrieval scores from different backends into a unified 0–1 relevance scale, detect outliers, and flag calibration risks.

## INPUT
[PASSAGES]

Each passage includes:
- `passage_id`: unique identifier
- `source`: retrieval backend name (e.g., vector_search, bm25, hybrid)
- `raw_score`: the original score from that backend
- `text`: the passage content
- `query`: the user's original query

## NORMALIZATION RULES
1. For each source, estimate the observed score distribution from the provided passages and normalize to 0–1 using min-max scaling within that source.
2. If a source has fewer than [MIN_PASSAGES_PER_SOURCE] passages, apply a conservative penalty of [SPARSE_SOURCE_PENALTY] to all normalized scores from that source.
3. After per-source normalization, apply cross-source calibration: if one source's normalized scores are systematically higher than others for passages with similar semantic relevance, apply a calibration shift of up to [MAX_CALIBRATION_SHIFT].
4. Flag any passage whose normalized score is more than [OUTLIER_STD_THRESHOLD] standard deviations above its source mean as a potential outlier.

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "normalized_passages": [
    {
      "passage_id": "string",
      "source": "string",
      "raw_score": number,
      "normalized_score": number,
      "calibration_adjustment": number,
      "is_outlier": boolean,
      "outlier_reason": "string or null"
    }
  ],
  "source_statistics": {
    "[source_name]": {
      "count": number,
      "raw_min": number,
      "raw_max": number,
      "normalized_mean": number,
      "normalized_std": number
    }
  },
  "calibration_notes": ["string"],
  "warnings": ["string"]
}

## CONSTRAINTS
- Do not invent or modify passage text.
- If a source has only one passage, set its normalized score to [DEFAULT_SINGLE_PASSAGE_SCORE] and add a warning.
- If the raw score range for a source is zero (all identical scores), assign all passages from that source a normalized score of [UNIFORM_SCORE_DEFAULT] and add a warning.
- Preserve all input passage_ids exactly.

## RISK_LEVEL
[HIGH] — Downstream answer generation depends on these normalized scores. Incorrect normalization can promote irrelevant evidence or suppress critical context.

Adaptation guidance: Replace the square-bracket placeholders with your pipeline's specific thresholds. [MIN_PASSAGES_PER_SOURCE] should reflect the minimum sample size you trust for distribution estimation—typically 5–10. [SPARSE_SOURCE_PENALTY] is a multiplier like 0.8 that down-weights scores from under-sampled sources. [OUTLIER_STD_THRESHOLD] is usually 2.0–3.0. [MAX_CALIBRATION_SHIFT] prevents over-correction; start with 0.15. [DEFAULT_SINGLE_PASSAGE_SCORE] and [UNIFORM_SCORE_DEFAULT] are fallback values—0.5 is a neutral default. Before deploying, run this prompt against a golden dataset where you know the correct relative ordering of passages across sources, and measure whether the normalized scores preserve that ordering. If your domain has known authority hierarchies (e.g., academic sources should rank above web forums), add a post-normalization authority boost as a separate step rather than baking it into this normalization prompt.