Hit Rate

Hit Rate measures the proportion of queries for which at least one relevant document is retrieved within the top K results. It is a binary, query-level metric.

• Formula: Hit Rate = (Number of queries with at least one relevant doc in top K) / (Total number of queries)
• Interpretation: A score of 0.95 means for 95% of queries, the system found something useful in its top K suggestions. It does not measure how many relevant documents were found, only if any were found.

Hit Rate is closely related to but distinct from Recall at K (R@K).

• Recall@K measures the proportion of all relevant documents retrieved for a query. It is a continuous metric (e.g., 0.67).
• Hit Rate is a binarized version of Recall@K. If Recall@K > 0, then Hit Rate = 1 for that query.
• Use Case: Hit Rate is often more critical for user-facing RAG systems. A user typically needs just one good source to answer their question; failure to find any (a Hit Rate miss) results in a complete system failure and likely a hallucinated answer.

The value of K is a crucial hyperparameter that defines the operational scope of the retrieval system and directly impacts the Hit Rate.

• Small K (e.g., 3-5): Measures the precision of the initial retriever. A high Hit Rate here indicates a very accurate dense embedding model or sparse retriever. This is typical for latency-sensitive applications where only a few passages are passed to the LLM.
• Large K (e.g., 50-100): Measures the recall of the initial "candidate generation" stage before a more expensive reranker is applied. A high Hit Rate here ensures the reranker has relevant material to work with.
• Best Practice: Set K equal to the number of passages your RAG pipeline's context window accepts. If your LLM context uses 5 passages, evaluate Hit Rate@5.

Hit Rate is the primary metric for answering: "How often does my RAG system have the necessary information to answer correctly?"

• A low Hit Rate is a fundamental retrieval problem; improving the generator (LLM) is futile if the correct context is never provided.
• It drives investigations into: embedding model quality, chunking strategies, hybrid search configuration, and the comprehensiveness of the knowledge base.
• Example: An enterprise chatbot with a Hit Rate@5 of 0.8 will hallucinate or respond "I don't know" for at least 20% of queries, regardless of LLM capability.

Hit Rate is a key Service Level Indicator (SLI) for production RAG systems, used to define Service Level Objectives (SLOs).

• SLO Example: "The answer generation service shall have a 30-day rolling Hit Rate@5 of >= 0.98."
• Monitoring Hit Rate over time detects retrieval degradation due to embedding model drift, changes in user query distribution, or expansions in the knowledge corpus that aren't properly indexed.
• A drop in Hit Rate triggers alerts for the MLOps team before users report degraded answer quality.

Hit Rate alone is insufficient for full system evaluation. It must be used with complementary metrics.

• Limitation 1: It doesn't measure ranking quality. A system could have a Hit Rate of 1.0 but place the single relevant document at position K, potentially pushing it out of the LLM's context window after truncation.
• Limitation 2: It treats all "hits" equally. One marginally relevant document scores the same as five highly relevant documents.
• Essential Complements: Precision@K (quality of the retrieved set), Mean Reciprocal Rank (MRR) (ranking of the first relevant hit), and NDCG (graded relevance). For the full RAG pipeline, Answer Faithfulness and Answer Relevance are required.

Precision at K measures the proportion of relevant documents within the top K retrieved results for a single query. It answers: 'Of the K documents I fetched, how many were good?'

• Core Focus: Quality of the retrieved set.
• Contrast with Hit Rate: While Hit Rate is binary (any relevant document present?), P@K is granular, quantifying the density of relevance in the top results.
• Example: If 3 of the top 5 retrieved documents (K=5) are relevant, P@5 = 0.6 or 60%.

Recall at K measures the proportion of all relevant documents in the corpus that are successfully found within the top K retrieved results. It answers: 'What fraction of everything relevant did I manage to capture in my top K results?'

• Core Focus: Completeness of retrieval.
• Relationship to Hit Rate: A high R@K implicitly ensures a high Hit Rate, as finding a large proportion of all relevant docs guarantees at least one is found. Hit Rate is a lower-bound, binary form of recall.
• Trade-off: Often in tension with Precision at K; retrieving more documents (higher K) can increase recall but may lower precision.

Mean Reciprocal Rank evaluates the rank of the first relevant document. For a set of queries, it averages the reciprocal of the rank position where the first relevant item appears. The reciprocal means higher scores are given to relevant documents appearing earlier (rank 1 -> 1/1=1, rank 3 -> 1/3≈0.33).

• Core Focus: Speed and rank of finding the first correct answer.
• Contrast with Hit Rate: MRR refines Hit Rate by penalizing systems where the first relevant document is buried deep in the results. A system can have a perfect Hit Rate (1.0) but a poor MRR if the first relevant doc is always at rank 10.

Context Relevance assesses the pertinence and utility of the retrieved text passages (the context) for answering the specific user query. It is a critical downstream metric for RAG, as even relevant documents can contain distracting or irrelevant sentences.

• Core Focus: Quality of the retrieved text used for generation.
• Beyond Hit Rate: Hit Rate confirms a relevant document was found; Context Relevance evaluates whether the specific text passed to the LLM is concise and on-topic. High Hit Rate with low Context Relevance leads to noisy context and potential hallucinations.
• Measurement: Often scored by LLM judges or using embedding similarity between query and retrieved chunks.

Answer Faithfulness (or Groundedness) measures the extent to which a generated answer is factually consistent with and logically deducible from the provided source context. It directly measures hallucination against the retrieved documents.

• Core Focus: Factual consistency of the generation with its sources.
• Dependency on Hit Rate: Faithfulness is only possible to evaluate if the system has a high Hit Rate, providing the necessary source material. If no relevant document is retrieved (Hit Rate=0), the model must hallucinate or refuse to answer.
• Key Metric: A cornerstone of RAG evaluation frameworks like RAGAS.

Reranking Effectiveness quantifies the improvement in retrieval quality achieved by applying a secondary, more computationally expensive model to reorder an initial set of candidate documents from a fast retriever (like a vector database).

• Core Focus: Lift in precision and recall after a second-stage ranking.
• Impact on Hit Rate: A reranker's primary job is to push the most relevant documents to the top ranks. This directly improves metrics like MRR and Precision at K. Its effect on Hit Rate is indirect but positive; by better ordering, it ensures the top-K window used for Hit Rate contains more relevant documents, making the metric more stable and meaningful.