Graded Relevance

Assigns multi-level scores (e.g., 0-4) to documents rather than binary labels. Common scales include:

• 0: Irrelevant
• 1: Marginally relevant
• 2: Relevant
• 3: Highly relevant
• 4: Perfectly authoritative This captures the reality that some documents are more useful than others, enabling metrics to reward systems that rank highly relevant documents above merely relevant ones.

Graded relevance is the foundational input for Normalized Discounted Cumulative Gain (NDCG), the dominant listwise ranking metric. NDCG uses the ordinal scores to:

• Compute discounted cumulative gain by applying a logarithmic position discount
• Normalize against the ideal ranking (IDCG) to produce a 0-1 score
• Heavily penalize systems that place highly relevant documents low in the ranked list

Graded labels are typically gathered through:

• Expert annotators: Domain specialists assign scores following detailed guidelines with examples for each level
• Crowdsourcing: Platforms like Amazon Mechanical Turk with quality control mechanisms and inter-annotator agreement checks
• Implicit signals: Deriving pseudo-grades from click-through rates, dwell time, or conversion events
• LLM-as-a-Judge: Using large language models prompted with the grading rubric to automate assessment at scale

Measuring consistency between human judges is critical for label quality. Common metrics include:

• Cohen's Kappa: Measures agreement beyond chance for two raters
• Fleiss' Kappa: Extends agreement measurement to multiple raters
• Weighted Kappa: Accounts for the ordinal nature of grades, penalizing larger disagreements (e.g., 0 vs 4) more severely than adjacent grades (e.g., 2 vs 3) Low agreement signals ambiguous grading guidelines or insufficient annotator training.

The 'discount' in NDCG reflects that users are less likely to examine documents deeper in the ranked list. The standard discount function is:

• 1 / log₂(rank + 1): Position 1 gets full gain, position 2 gets ~0.63, position 10 gets ~0.29 This models position bias in user behavior and ensures that metrics reward systems that surface highly relevant documents early, where they have the greatest impact on user satisfaction.

Binary relevance collapses all useful documents into a single category, treating a tangentially related page identically to a definitive answer. Graded relevance enables:

• Precision at multiple thresholds: Measure precision@k for 'highly relevant' vs 'any relevance'
• More discriminative evaluation: Systems that rank authoritative sources above merely topical ones receive higher scores
• Realistic user modeling: Reflects that information needs exist on a spectrum from navigational to fully informational

A neural architecture that processes a query-document pair jointly through full self-attention, enabling deep token-level interaction. Cross-encoders produce highly accurate graded relevance scores but are computationally expensive.

• Use case: Precise re-ranking of top-N candidates
• Input: Concatenated [CLS] query [SEP] document [SEP]
• Output: A single relevance score (often mapped to graded labels)
• Trade-off: High accuracy vs. low throughput; cannot pre-compute document embeddings

A paradigm where a large language model is prompted to evaluate and score output quality using graded relevance scales. The LLM acts as an automated relevance assessor, assigning ordinal scores based on detailed rubrics.

• Scoring: Likert scales (1-5) or binary with confidence
• Advantages: Captures nuanced semantic relevance beyond keyword overlap
• Calibration: Temperature scaling can align LLM confidence with human judgment distributions
• Limitations: Position bias, verbosity bias, and self-enhancement bias require mitigation

A cascade architecture where a fast bi-encoder retrieves candidate documents, and a computationally intensive cross-encoder re-ranks them using fine-grained graded relevance scoring.

• Stage 1: Approximate nearest neighbor search over dense embeddings (high recall)
• Stage 2: Cross-encoder scores top-K candidates (high precision)
• Benefit: Balances latency and accuracy by applying expensive computation only to promising candidates
• Typical K: 100-1000 candidates passed to the re-ranker