RAGAS

Answer Faithfulness measures the factual consistency of a generated answer with the provided source context. It quantifies hallucinations by checking if all claims in the answer can be logically inferred from the retrieved documents.

• Mechanism: Typically uses an NLI (Natural Language Inference) model or an LLM-as-a-judge to determine if each statement in the answer is entailed by the context.
• Output: A score between 0 and 1, where 1 indicates the answer contains no unsupported facts.
• Example: If the context states "The company was founded in 2010," and the answer says "Founded in 2012," the faithfulness score would be low.

Answer Relevance evaluates how directly the generated answer addresses the original query, independent of the context's factuality. It penalizes verbose, incomplete, or off-topic responses.

• Mechanism: Often calculated by using the query to interrogate the answer. An LLM judge might be asked: "Given this answer, how well is the following query addressed?"
• Focus: This metric isolates the generator's ability to be concise and on-point.
• Example: For the query "What is the capital of France?", the answer "Paris is a major European city" is partially relevant but incomplete. The answer "The capital is Paris" would score higher.

Context Precision assesses the quality of the retrieval step by measuring how many of the retrieved documents are relevant to answering the query. It rewards systems that rank relevant documents higher.

• Calculation: It is the precision of the retrieved set, weighted by the rank of each relevant document. The formula is: sum((precision at k) * rel(k)) / total relevant docs, where rel(k) is the relevance of the item at rank k.
• Purpose: Measures the signal-to-noise ratio in the context provided to the LLM. High precision means the LLM receives mostly useful information.

Context Recall evaluates the retrieval system's completeness. It measures the proportion of all relevant information in a ground truth answer that is actually present in the retrieved context.

• Mechanism: Compares ground truth statements (from a human reference) against the retrieved context to see what fraction are covered.
• Use Case: Critical for ensuring the system has access to all necessary information to form a complete answer. Low recall indicates missed key documents.
• Note: Unlike other RAGAS metrics, this one does require a ground truth answer for calculation.

The Aspect Critic Model is the underlying LLM-as-a-judge mechanism used by RAGAS to compute metrics like faithfulness and relevance without pre-trained classifiers.

• Process: The framework provides carefully crafted prompts that instruct an LLM (e.g., GPT-4) to evaluate a specific aspect (e.g., "Is this claim supported?").
• Output Parsing: The LLM's textual response (e.g., "supported" or "not supported") is parsed into a numerical score.
• Advantage: Makes the framework model-agnostic and adaptable, but introduces cost and latency from using external LLM APIs.

The Composite RAG Score is a single, aggregated metric provided by RAGAS, typically the harmonic mean of its core component scores, offering an overall performance indicator.

• Common Formula: Composite Score = (Faithfulness * Relevance * Precision)^(1/3)
• Utility: Provides a high-level benchmark for comparing different RAG pipeline configurations (e.g., changing retrievers, chunk sizes, or prompts).
• Limitation: A single score can mask trade-offs; a dip in one component (e.g., Context Recall) might be obscured by high scores in others. It is essential to analyze the component metrics individually for debugging.

Answer Faithfulness is a core metric in RAG evaluation that measures the extent to which a generated answer is factually consistent with and logically deducible from the provided source context. It directly targets hallucinations—instances where a model invents unsupported facts.

• Purpose: To ensure the LLM's output is grounded in the retrieved documents.
• Measurement: Typically scored by prompting an LLM judge to verify if all claims in the answer can be inferred from the context. A low faithfulness score indicates a problematic, ungrounded generation step.
• Relation to RAGAS: This is one of the primary reference-free metrics calculated by the RAGAS framework, often implemented using an LLM-as-a-judge approach.

Context Relevance evaluates the quality of the retrieval step by assessing how pertinent the retrieved text passages are for answering the given query. It penalizes retrieval of redundant or irrelevant information.

• Purpose: To isolate and measure the performance of the retriever component, independent of the generator.
• Measurement: Often calculated by prompting an LLM to identify and remove any sentences from the retrieved context that are not needed to answer the query. The score is based on the conciseness of the remaining text.
• Key Insight: High context relevance means the retriever is providing a dense, useful information set, which improves answer quality and reduces processing overhead for the LLM.

Answer Relevance measures how directly and comprehensively a generated answer addresses the original query, independent of its factual correctness. It assesses the answer's utility from a user's perspective.

• Purpose: To ensure the model's output is on-topic and complete, not evasive or generic.
• Measurement: Typically evaluated by using the generated answer to reconstruct the original query via an LLM. The similarity between the original and reconstructed query determines the score.
• Difference from Faithfulness: An answer can be highly relevant (directly addresses the question) but unfaithful (contains made-up details), or faithful but irrelevant (talks about a related but different topic).

A Grounding Score is a metric that quantifies the degree to which a model's generated output is substantiated by specific, attributable information from its provided source materials. It is closely related to Answer Faithfulness but often implies a stricter, token-level or claim-level attribution.

• Purpose: To enforce citational integrity, ensuring every factual statement can be traced to a source snippet.
• Measurement: Can involve decomposing an answer into individual claims and verifying each against the context, sometimes using Natural Language Inference (NLI) models or LLM judges.
• Enterprise Importance: Critical for applications in legal, medical, and financial domains where audit trails and provenance are mandatory.

The Retrieval-Augmented Generation Score (RAG Score) is a composite metric that aggregates performance across multiple evaluation dimensions (like faithfulness, relevance, and context precision) into a single, summary figure of merit for a RAG pipeline.

• Purpose: To provide a high-level, easily trackable key performance indicator (KPI) for system health and version comparisons.
• Construction: Frameworks like RAGAS calculate this by combining their core metrics (e.g., faithfulness, answer relevance, context relevance) using a defined mathematical aggregation, such as a harmonic mean.
• Utility: While the composite score is useful for monitoring, engineers must drill down into the constituent metrics to diagnose specific failures in retrieval or generation.