FActScore

FActScore breaks down complex, multi-sentence generations into indivisible, self-contained factual claims. Each atomic fact is a single piece of information that can be independently verified as true or false.

• Granularity: A paragraph about a person might yield 10-15 atomic facts
• Format: Each fact is a standalone declarative sentence
• Purpose: Eliminates ambiguity by isolating claims for precise verification

This decomposition is typically performed by a prompted LLM, with human validation studies showing high agreement on what constitutes a correct atomic breakdown.

Each atomic fact is verified against a trusted reference corpus, typically Wikipedia passages retrieved via dense retrieval. The verifier model classifies each fact as Supported, Not Supported, or Irrelevant.

• Retrieval: Uses a bi-encoder dense retriever to find the most relevant Wikipedia paragraph
• Verification: An NLI-style model determines if the passage entails the atomic fact
• Irrelevant: Facts about topics not covered in the knowledge base are excluded from scoring

This approach mirrors how a human fact-checker would consult an authoritative source, making the metric inherently interpretable.

FActScore is explicitly designed to correlate with human judgments of factual accuracy. The score is calculated as the percentage of supported atomic facts out of all facts that could be verified against the knowledge base.

• Formula: FActScore = |Supported Facts| / (|Supported| + |Not Supported|)
• Range: 0% (fully hallucinated) to 100% (fully grounded)
• Validation: Achieves high correlation with human factuality ratings across multiple LLMs

This alignment makes FActScore a reliable proxy for human evaluation, enabling scalable automated factuality assessment without manual review.

FActScore can evaluate any long-form text generator regardless of architecture or training method. It treats the evaluated model as a black box, requiring only the generated text as input.

• Evaluated Models: Works with GPT-4, Claude, PaLM, LLaMA, and custom fine-tuned models
• Task Coverage: Evaluates biographies, news articles, product descriptions, and technical explanations
• Zero-Shot: No access to model internals, training data, or output probabilities required

This flexibility makes FActScore a universal benchmark for comparing factual reliability across different LLM providers and deployment scenarios.

Beyond a single aggregate score, FActScore enables fine-grained error categorization by identifying exactly which atomic facts failed verification and why.

• Error Types: Distinguishes between factual contradictions, unsupported claims, and unverifiable statements
• Entity Tracking: Reveals which named entities (people, places, organizations) are most prone to hallucination
• Topic Analysis: Identifies subject domains where the model consistently underperforms

This diagnostic capability helps model developers target specific weaknesses in their factuality training pipelines rather than applying blanket fixes.

The foundational preprocessing step for FActScore that breaks long-form text into self-contained, verifiable claims. Each atomic fact must be a single piece of information that can be independently verified against a knowledge base.

• Strips pronouns and resolves coreferences into explicit entities
• Transforms complex sentences into simple subject-predicate-object triples
• Enables granular precision scoring rather than holistic judgments

Example: "Einstein, born in Ulm, won the Nobel Prize in 1921" becomes two facts: (1) Einstein was born in Ulm, (2) Einstein won the Nobel Prize in 1921.