Knowledge F1

Knowledge F1 is calculated as the harmonic mean of Factual Precision and Factual Recall, not a simple arithmetic average. This mathematical choice penalizes extreme imbalances between the two metrics.

• Formula: 2 * (Precision * Recall) / (Precision + Recall)
• Penalty Effect: A model with 100% precision but only 10% recall will score a low F1 (~18%), reflecting incomplete extraction
• Why Harmonic: Prevents models from gaming the system by being overly conservative (high precision, low recall) or overly verbose (high recall, low precision)
• Range: Always between 0.0 and 1.0, where 1.0 represents perfect extraction

Before Knowledge F1 can be computed, both the generated text and the ground-truth source must be decomposed into atomic factual units—individual, verifiable statements.

• Atomic Fact: A single, self-contained claim that can be independently verified (e.g., 'The Eiffel Tower is 330 meters tall')
• Decomposition Methods: Can use rule-based NLP parsers, fine-tuned extractive models, or LLM-based factoid extraction
• Granularity Control: The definition of an 'atomic fact' must be consistent across both prediction and reference to ensure valid comparison
• Example: The sentence 'Paris, founded in the 3rd century BC, is the capital of France' decomposes into three atomic facts

Knowledge F1 explicitly surfaces the inherent tension between exactness and completeness in factual extraction, forcing evaluation beyond single-dimensional accuracy.

• High Precision, Low Recall: The model extracts few facts but all are correct—indicative of a conservative, risk-averse generation strategy
• Low Precision, High Recall: The model extracts many facts but includes errors—indicative of hallucination-prone behavior
• Operational Interpretation: A system used for medical summarization may prioritize precision over recall, while a legal discovery tool may require high recall
• Threshold Setting: Organizations can set minimum acceptable F1 thresholds based on their domain's risk tolerance

Knowledge F1 can be computed at different semantic depths, distinguishing between simple entity extraction and complex relational fact verification.

• Entity-Level F1: Measures whether the correct named entities (people, dates, locations) were extracted, ignoring relationships between them
• Relation-Level F1: Measures whether the correct subject-predicate-object triples were extracted, requiring both entity correctness and relational accuracy
• Example: Extracting 'Tim Cook' and 'Apple' is entity-level; extracting 'Tim Cook is the CEO of Apple' is relation-level
• Hierarchical Reporting: Mature evaluation pipelines report both levels to provide granular diagnostic insight into failure modes

Knowledge F1 is designed to correlate with human evaluator assessments of factual quality, making it a reliable proxy for manual review in automated evaluation pipelines.

• Benchmark Correlation: Studies using datasets like FActScore and HaluEval show strong Pearson correlation (>0.7) between Knowledge F1 and human factuality ratings
• Inter-Annotator Agreement: The atomic decomposition step reduces subjectivity, improving consistency across human evaluators
• Limitation: F1 does not capture the severity of errors—a critical medical misstatement and a minor date error are weighted equally
• Complementary Metrics: Should be used alongside Critical Error Rate and Semantic Entropy for a complete risk picture

Acceptable Knowledge F1 scores vary dramatically by application domain, and no universal threshold exists for 'good enough' factual accuracy.

• Medical Summarization: Typically requires F1 > 0.95 due to patient safety implications
• News Article Generation: Often targets F1 > 0.85, balancing completeness with acceptable minor errors
• Creative Writing Assistance: May accept F1 as low as 0.60, where factual precision is secondary to stylistic quality
• Continuous Monitoring: F1 thresholds should be monitored in production as model drift or data distribution shifts can degrade performance over time

Measures the exactness of a model's output by calculating the ratio of correct factual statements to the total number of factual statements generated.

• Formula: True Positives / (True Positives + False Positives)
• A high precision score indicates the model rarely inserts incorrect facts when it makes a claim
• Low precision signals the model is over-generating or confabulating details
• Often paired with Factual Recall to compute the Knowledge F1 harmonic mean

Quantifies the completeness of information extraction by measuring the ratio of correctly generated facts to the total number of facts present in the ground-truth source.

• Formula: True Positives / (True Positives + False Negatives)
• High recall means the model successfully surfaced most available factual knowledge
• Low recall indicates the model is omitting critical information from the source
• The harmonic mean of Precision and Recall produces the Knowledge F1 score

A human-aligned evaluation metric that decomposes long-form generation into atomic facts and verifies each against a trusted knowledge base like Wikipedia.

• Breaks paragraphs into discrete, verifiable factual units
• Each atomic fact is checked independently for support
• Produces a percentage score representing the proportion of supported facts
• Directly addresses the granularity problem in hallucination measurement

An automated evaluation score using Natural Language Inference (NLI) to determine if a generated summary can be logically deduced from the input source.

• Classifies each claim as entailment, contradiction, or neutral
• A faithful output contains only entailed statements with no introduced information
• Serves as a foundational building block for more complex composite metrics like Knowledge F1
• Commonly used in summarization and RAG evaluation pipelines

The frequency at which a language model generates nonsensical or factually incorrect text relative to source material, expressed as a percentage of total tokens or sentences.

• Provides a high-level health indicator for production LLM systems
• Can be decomposed into entity-level, relation-level, and sentence-level hallucinations
• Knowledge F1 offers a more nuanced alternative by separating precision and recall failures
• Critical for LLMOps monitoring dashboards and risk thresholds

Evaluates whether a model can correctly link generated claims to the specific source segments that support them, measured through Citation Recall and Citation Precision.

• Citation Recall: Proportion of claims supported by a cited source
• Citation Precision: Proportion of citations that actually support their associated claim
• Complements Knowledge F1 by adding a provenance dimension to factual accuracy
• Essential for RAG systems where traceability is a compliance requirement