RAGAS (RAG Assessment)

Answer Relevance measures how directly the generated answer addresses the original query, penalizing extraneous or irrelevant information. It is calculated by generating a question from the answer using an LLM and measuring its semantic similarity to the original query.

• Purpose: Quantifies the conciseness and focus of the generated answer.
• Mechanism: Employs an LLM to perform question generation, creating a distilled version of the query implied by the answer.
• High Score Indicates: The answer is strictly pertinent to the query.
• Low Score Indicates: The answer contains hallucinations or off-topic information.

Faithfulness (or Factual Consistency) evaluates whether the facts presented in the generated answer are fully supported by the provided context. It identifies and counts unsupported statements (hallucinations).

• Purpose: Measures the factual grounding of the generation in the retrieved context.
• Mechanism: An LLM extracts all atomic statements from the answer, then judges whether each is entailed by the context.
• High Score Indicates: All claims in the answer can be inferred from the context.
• Low Score Indicates: The model introduced unsupported or contradictory facts.

Context Relevance assesses the quality of the retrieval step by measuring how much of the retrieved information is necessary to answer the query. It penalizes redundant or irrelevant passages.

• Purpose: Evaluates the precision and conciseness of the retriever.
• Mechanism: An LLM judges each sentence in the retrieved context for its necessity in answering the query.
• High Score Indicates: The retrieved context is dense with relevant information.
• Low Score Indicates: The retriever returned noisy, off-topic passages.

Context Recall measures the retriever's ability to find all information relevant to the ground truth answer. Unlike Context Relevance, it requires a ground truth answer for comparison.

• Purpose: Evaluates the recall of the retrieval system.
• Mechanism: Compares ground truth answer statements to the retrieved context to see what fraction are present or inferable.
• High Score Indicates: The retriever successfully found all necessary information.
• Low Score Indicates: Critical evidence was missed by the retriever.

RAGAS includes Aspect Critique metrics, where an LLM judge evaluates the answer against specific qualitative dimensions. These provide nuanced, subjective assessments.

• Common Aspects:
  • Harmfulness: Is the answer safe, unbiased, and non-toxic?
  • Misleading: Is the answer likely to deceive or misinform the user?
  • Coherence: Is the answer logically structured and easy to follow?
• Mechanism: An LLM acts as a critic, scoring the answer on a Likert scale (e.g., 1-5) for the specified aspect based on predefined guidelines.
• Use Case: Complements objective metrics with qualitative, human-aligned judgments.

The RAGAS Score is a single composite metric that summarizes overall system performance. It is typically computed as the harmonic mean of the core reference-free metrics: Answer Relevance, Faithfulness, and Context Relevance.

• Formula: Often implemented as RAGAS Score = 3 / (1/AR + 1/F + 1/CR) where AR, F, and CR are the scores for Answer Relevance, Faithfulness, and Context Relevance.
• Purpose: Provides a quick, high-level indicator of RAG pipeline health.
• Interpretation: A high composite score indicates a system that retrieves relevant context and generates focused, factual answers.
• Limitation: May mask trade-offs between individual components; analyzing the decomposed scores is essential for debugging.

Faithfulness is a RAGAS metric that measures the factual consistency between the generated answer and the retrieved context. It quantifies how well the LLM's output is grounded in the provided source documents, directly targeting hallucination detection.

• Calculation: Typically involves using an LLM judge to verify if all claims in the answer can be inferred from the retrieved context.
• Purpose: A low faithfulness score indicates the model is "making up" information not present in the context, a critical failure mode for enterprise RAG.

Answer Relevance evaluates how directly the generated response addresses the original query, independent of the retrieved context. It measures the conciseness and focus of the output.

• Calculation: Often assessed by using the LLM to generate a follow-up question based on the answer; a highly relevant answer should allow for a reconstruction of the original query.
• Purpose: Ensures the system does not provide verbose, generic, or off-topic responses, maintaining utility for the end-user.

Context Precision is a retrieval-oriented metric that assesses the quality of the retrieved documents. It measures the proportion of retrieved chunks that are relevant to answering the query.

• Key Insight: In RAG, high-quality generation is impossible without high-quality retrieval. This metric isolates and evaluates the retrieval component.
• Impact: A low score indicates the retrieval system is returning too much irrelevant noise, which can confuse the LLM and degrade answer quality.

Context Recall measures the retrieval system's ability to find all relevant information needed to answer the query comprehensively. It's the complement to Context Precision.

• Calculation: Compares the retrieved context against an ideal, comprehensive ground truth answer. It quantifies what percentage of necessary information was successfully retrieved.
• Purpose: Identifies failures where the system misses key facts, leading to incomplete or incorrect answers despite high precision on the chunks it did find.

This metric evaluates the semantic alignment between the generated answer and a ground truth answer, using embedding-based cosine similarity rather than exact lexical match.

• Tool: Uses sentence-transformers (e.g., all-MiniLM-L6-v2) to generate embeddings for both answers.
• Advantage: More robust than metrics like BLEU or ROUGE, as it captures paraphrasing and semantic equivalence, which is crucial for evaluating LLM-generated text.

A core principle of RAGAS is enabling reference-free or unsupervised evaluation. This means metrics are computed without requiring human-written ground truth answers, which are expensive and slow to produce.

• Mechanism: Leverages the LLM itself (as a judge) and the retrieved context as the source of truth for metrics like Faithfulness and Answer Relevance.
• Benefit: Allows for rapid, automated evaluation cycles during RAG pipeline development and continuous monitoring in production.