Retrieval-Augmented Generation Score (RAG Score)

These metrics evaluate the performance of the document retrieval subsystem, which is foundational to a RAG pipeline. A high RAG Score depends on retrieving the most relevant information.

• Precision & Recall: Measures the relevance of retrieved documents. Precision@K calculates the proportion of relevant docs in the top K results. Recall@K measures the proportion of all relevant docs found in the top K.
• Normalized Discounted Cumulative Gain (NDCG): A ranking-aware metric that accounts for the graded relevance of documents and their position in the results list. It is the gold standard for evaluating ranked retrieval output.
• Context Relevance: Specifically assesses whether the text passages provided to the LLM are concise and pertinent to the query, penalizing redundant or irrelevant context.

This component measures the factual consistency between the generated answer and the provided source documents. It directly targets the prevention of hallucinations.

• Answer Faithfulness: Quantifies if all factual claims in the generated answer are logically entailed by the source context. A low score indicates fabrication or unsupported inference.
• Grounding Score: Evaluates the degree to which the output is substantiated by specific, attributable information from the source materials. It is closely related to faithfulness but may involve finer-grained attribution checks.
• Hallucination Rate: The inverse of faithfulness; the frequency of unsupported statements. It is a critical failure mode metric for production systems.

These metrics assess the utility and accuracy of the final generated answer from the user's perspective, independent of the retrieval process.

• Answer Relevance: Evaluates how directly and completely the generated answer addresses the original query. An answer can be faithful to irrelevant context but score poorly here.
• Answer Correctness: A composite assessment comparing the generated answer to a ground truth reference. It often incorporates aspects of semantic similarity (e.g., BERTScore) or token overlap (e.g., F1 Score) with an expert-provided ideal answer.
• Semantic Similarity: Uses embedding models (e.g., Sentence-BERT) to measure the meaning-based likeness between the generated and reference answers, which is more robust than lexical overlap.

For systems that provide source citations, these metrics evaluate the accuracy and completeness of the attribution, which is essential for trust and verifiability.

• Source Citation Precision: The proportion of citations in the generated answer that correctly reference the source of the stated information. Incorrect citations degrade trust.
• Source Citation Recall: The proportion of source statements or facts used in the answer that are correctly attributed to their originating documents. Missing citations obscure provenance.
• These metrics are crucial for enterprise and legal applications where audit trails and evidence are required.

While not always part of the core quality score, these operational metrics are critical for production deployment and are often tracked alongside the RAG Score.

• End-to-End Latency: The total time from query submission to answer generation, encompassing retrieval, reranking, and LLM inference. Directly impacts user experience.
• Token Efficiency: Measures the cost-effectiveness of the pipeline, including the number of tokens sent to the LLM (context + prompt) and generated in the answer.
• Throughput & Scalability: The number of queries the system can handle per second, which depends on the retrieval system's speed and the LLM's generation capacity.

Answer Faithfulness (or Factual Consistency) is a critical component of the RAG Score. It quantifies the extent to which statements in a generated answer are logically entailed by and supported by the provided source context. A low faithfulness score indicates hallucinations—information fabricated by the LLM that is not grounded in the retrieved documents. Evaluation methods include:

• Using an NLI (Natural Language Inference) model to check if the answer can be inferred from the context.
• Decomposing the answer into atomic claims and verifying each against the sources. High faithfulness is non-negotiable for enterprise RAG systems where factual accuracy is paramount.

Context Relevance measures the quality of the retrieval step, a prerequisite for a high RAG Score. It evaluates the retrieved documents for noise and redundancy. Irrelevant passages in the context can confuse the LLM, leading to lower answer quality. This metric answers: "Is all the provided context useful for answering the query?"

• It penalizes retrieved passages that contain irrelevant information.
• It is often calculated by having an LLM judge the relevance of each retrieved sentence to the query.
• High context relevance ensures the generation model operates on clean, focused information, improving both answer quality and efficiency.

Answer Relevance assesses the generated output independently of the source context. It focuses on whether the answer is a direct, complete, and appropriate response to the original user query. A answer can be perfectly faithful to irrelevant context but still score poorly on this dimension. Evaluation typically involves:

• Using the original query to interrogate the generated answer (e.g., "Given this answer, what was the question?").
• Measuring the semantic similarity between a reformulated query (based on the answer) and the original query. This metric ensures the system does not provide generic, evasive, or incomplete responses.

These are foundational information retrieval metrics that directly influence the RAG Score's potential ceiling.

• Precision at K (P@K): The fraction of retrieved documents in the top K that are relevant. High precision means less noise in the context.
• Recall at K (R@K): The fraction of all relevant documents in the corpus that are retrieved in the top K. High recall ensures critical information isn't missed. A RAG system requires a balance: high recall to find all necessary facts, and high precision to avoid overwhelming the LLM with junk. These are often measured before the generation step to isolate retrieval performance.

While a RAG Score is often reference-free, semantic similarity metrics provide a ground-truth-based evaluation of answer quality, useful for validation. They compare the generated answer to an ideal reference answer.

• BERTScore: Uses contextual embeddings (e.g., from BERT) to compute token-level similarity, capturing semantic meaning better than lexical overlap.
• Sentence Embeddings: Measures cosine similarity between dense vector representations of the candidate and reference answers.
• ROUGE & BLEU: Traditional n-gram overlap metrics, less semantically robust but useful for specific tasks like summarization. These metrics are complementary to the faithfulness/relevance axes of a RAG Score.