Precision at K

Precision at K is calculated as the number of relevant documents in the top-K results divided by K. For example, if 7 out of the top 10 documents are relevant, P@10 = 0.7. Unlike Mean Average Precision (MAP), P@K does not average over recall levels; it provides a single, intuitive score for a fixed cutoff. This makes it ideal for evaluating systems where users rarely look beyond the first page, such as web search and recommendation engines.

While Average Precision (AP) rewards systems that place relevant documents higher in the ranking by averaging precision at each relevant position, P@K is a simpler, cutoff-based metric. Key distinctions:

• P@K ignores the order of relevant documents within the top-K; all relevant documents contribute equally.
• AP penalizes relevant documents that appear later in the list.
• P@K is more interpretable for stakeholders: '30% of the top 10 results were useful.'
• For tasks with a single relevant answer, P@K is equivalent to Reciprocal Rank at K.

Standard Precision at K operates on a strict binary relevance model: a document is either relevant (1) or not relevant (0). This simplification has practical implications:

• It aligns with click-through data where a click is a binary signal.
• It does not account for graded relevance (e.g., 'partially relevant' vs. 'highly relevant').
• For nuanced evaluation, extensions like Discounted Cumulative Gain (DCG) or graded P@K can be used.
• In practice, relevance judgments are often collected from human annotators or inferred from user interaction logs.

In the context of BM25 and sparse lexical retrieval, P@K is a critical diagnostic tool. Since BM25 relies on exact term matching, P@K can reveal vocabulary mismatch problems. A low P@10 score on a test collection indicates that the top results contain many non-relevant documents, often because the query terms do not align with the vocabulary used in relevant documents. This metric directly motivates the adoption of query expansion and pseudo-relevance feedback techniques to improve the density of relevant documents in the top-K.

The choice of K is task-dependent and reflects user behavior:

• P@1: Critical for question-answering and voice search, where only the first result is surfaced.
• P@5: Common for mobile search interfaces with limited screen real estate.
• P@10: The standard for web search evaluation, representing the first page of results.
• P@20: Used for recall-oriented tasks like e-discovery or patent search. Benchmarks like MS MARCO and TREC routinely report P@K to compare sparse and dense retrieval systems.

P@K has known limitations that require complementary metrics for a complete evaluation:

• Ignores Recall: A system can have high P@10 but miss many other relevant documents. Pair with Recall@K.
• Cutoff Sensitivity: A system optimized for P@10 may perform poorly at P@20. Evaluate at multiple K values.
• No Rank Differentiation: A relevant document at position 1 and position 10 are weighted equally. Use MRR or nDCG to capture ranking quality.
• Judgment Incompleteness: Unjudged documents are typically treated as non-relevant, which can underestimate true precision.

The complementary metric to Precision at K, measuring the fraction of total relevant documents in the corpus that appear in the top-K results. While Precision at K focuses on result purity, Recall at K captures coverage.

• Formula: (Relevant docs in top-K) / (Total relevant docs in corpus)
• Trade-off: High recall often requires larger K values, which may reduce precision
• Use case: Critical in e-discovery and legal search where missing a relevant document is costly

An evaluation metric that measures how high the first relevant document appears in the ranked list. MRR is the average of reciprocal ranks across multiple queries, heavily penalizing systems that bury the correct answer deep in results.

• Reciprocal Rank: 1 / rank of first relevant result
• Range: 0 to 1, where 1 means the first result is always relevant
• Best for: Question-answering systems and known-item search where users need exactly one answer

A ranking-aware metric that accounts for graded relevance (not just binary relevant/not relevant) and the position of results. NDCG applies a logarithmic discount to relevance scores based on rank, reflecting that users are less likely to examine lower-ranked items.

• DCG: Sum of (relevance score / log₂(rank + 1)) for each position
• IDCG: Ideal DCG if results were perfectly ordered by relevance
• NDCG = DCG / IDCG: Normalized to [0, 1] for comparability across queries

The harmonic mean of precision and recall, providing a single balanced measure when both false positives and false negatives matter. Unlike Precision at K, F1 is typically computed over an entire classification or retrieval result set rather than a fixed cutoff.

• Formula: 2 × (Precision × Recall) / (Precision + Recall)
• Fβ variant: Weighted F-score where β > 1 favors recall, β < 1 favors precision
• Limitation: Does not account for ranking order, only set membership

The ground truth annotation that determines whether a document is relevant to a query. These judgments form the foundation of all retrieval evaluation metrics, including Precision at K.

• Binary judgments: Simple relevant/not relevant labels
• Graded judgments: Multi-level scales (e.g., 0-4 from irrelevant to highly relevant)
• Inter-annotator agreement: Measured via Cohen's Kappa to ensure judgment quality
• Pooling: Common method where top results from multiple systems are merged for human annotation