Factual Precision

Factual Precision is calculated as the ratio of correctly generated atomic facts to the total number of atomic facts present in the model's output.

• True Positives (TP): Correctly stated facts.
• False Positives (FP): Incorrectly stated or hallucinated facts.
• Formula: Precision = TP / (TP + FP)
• A score of 1.0 indicates zero hallucinations, but may mask poor recall.

Factual Precision must be analyzed alongside Factual Recall to avoid metric gaming. A model can achieve perfect precision by being extremely terse.

• High Precision, Low Recall: The model says very little, but everything it says is correct (safe, but incomplete).
• High Recall, Low Precision: The model covers many facts but invents details (comprehensive, but untrustworthy).
• The harmonic mean is captured by the Knowledge F1 score.

To calculate precision, long-form text must be broken into atomic facts—short, self-contained, verifiable statements.

• Example: "Apple Inc., founded by Steve Jobs in Cupertino, released the iPhone in 2007."
• Atom 1: Apple Inc. was founded by Steve Jobs.
• Atom 2: Apple Inc. was founded in Cupertino.
• Atom 3: Apple Inc. released the iPhone in 2007.
• Tools like FActScore automate this decomposition against Wikipedia.

Factual Precision is often confused with Faithfulness, but they measure different alignments.

• Factual Precision: Measures alignment with world knowledge or a verified knowledge base.
• Faithfulness: Measures alignment with a specific source document provided in the context.
• A statement can be faithful to a source document but factually imprecise if the source itself is wrong.

A critical sub-type of factual precision focuses specifically on named entities. This measures the model's ability to avoid inventing people, places, or organizations.

• Entity Error: Referring to "Tim Cook" as the CEO in 2010.
• Relation Error: Stating "Apple acquired Microsoft."
• Entity-Level Hallucination is often the most damaging type of error in enterprise settings, as it creates entirely fictional actors.

Several benchmarks specifically test factual precision by targeting common model failure modes.

• TruthfulQA: Tests resistance to generating common human misconceptions.
• FActScore: Verifies biographical generations against Wikipedia.
• HaluEval: Contains human-annotated hallucinated samples to test detection models.
• These benchmarks reveal that even high-accuracy models often have low precision on obscure facts.

An automated evaluation score, often using Natural Language Inference (NLI), that determines if a generated summary or response can be logically deduced from the input source without introducing extraneous information. It measures entailment rather than just overlap.

• NLI-Based Evaluation: Classifies the source-hypothesis relationship as entailment, contradiction, or neutral
• FaithDial: A curated dataset where hallucinated responses have been corrected to be faithful to the knowledge source
• Use Case: Essential for evaluating summarization and RAG pipelines

A human-aligned evaluation metric that breaks a long-form generation into atomic facts and verifies each against a trusted knowledge base like Wikipedia. It calculates the percentage of supported facts, providing a granular view of factual precision.

• Process: Decompose text → Extract atomic claims → Verify each claim independently
• Output: A ratio of supported facts to total facts
• Advantage: Offers a more nuanced breakdown than holistic scoring methods

The field of machine learning focused on estimating the confidence bounds of a model's predictions to identify when the model is likely to be wrong. It distinguishes between epistemic uncertainty (reducible, from lack of knowledge) and aleatoric uncertainty (irreducible, from data noise).

• Semantic Entropy: Clusters semantically equivalent outputs before calculating entropy to measure true uncertainty
• Conformal Prediction: A model-agnostic framework that generates prediction sets with a mathematical guarantee of containing the true output
• Deep Ensemble Uncertainty: Measures variance across multiple independently trained models

A prompting technique where an LLM first drafts a response, then generates a series of independent verification questions to fact-check its own work, and finally produces a corrected, verified answer. It is a zero-resource method for improving factual precision.

• Step 1: Generate baseline response
• Step 2: Plan verification questions based on the draft
• Step 3: Execute verifications independently
• Step 4: Produce the final, corrected output

A metric evaluating whether a model can correctly link a generated claim to the specific segment of a source document that supports it. It is decomposed into Citation Recall (proportion of claims with supporting citations) and Citation Precision (proportion of citations that are relevant).

• RAGTruth: A specialized benchmark for evaluating hallucination at the passage and word level in RAG systems
• Importance: Directly measures the verifiability of AI-generated content
• Application: Critical for legal and medical AI systems requiring audit trails