FEVER Dataset

FEVER defines a strict ternary label space that forces models to distinguish between definitive truth, definitive falsehood, and epistemic uncertainty.

• SUPPORTS: The retrieved evidence confirms the claim is true.
• REFUTES: The retrieved evidence confirms the claim is false.
• NOT ENOUGH INFO: Wikipedia does not contain sufficient information to verify or refute the claim.

This tripartite structure prevents models from guessing when evidence is absent, a critical requirement for real-world fact-checking systems where silence is preferable to hallucination.

FEVER is not a single classification task but a composite pipeline requiring two distinct stages:

• Document Retrieval: Models must first search Wikipedia to find relevant documents, simulating real-world information-seeking behavior.
• Textual Entailment: Given retrieved sentences, models perform Natural Language Inference to determine if the evidence logically entails or contradicts the claim.

This decomposition mirrors production fact-checking architectures where retrieval and verification are separate, evaluable components. The dataset provides gold-standard evidence annotations for supervised training of both stages.

Claims in FEVER were created by human annotators who were instructed to mutate sentences from Wikipedia into factual assertions, ensuring natural linguistic diversity.

• Annotators rephrased, negated, or generalized original sentences to produce claims that sound like real-world statements.
• This process avoids the artificiality of template-generated claims and introduces genuine lexical variation, negation patterns, and syntactic complexity.
• Claims cover a broad range of topics including history, science, entertainment, and geography, testing domain generalization.

The human-in-the-loop design ensures that models trained on FEVER encounter the same ambiguities and phrasing variations present in actual misinformation.

FEVER employed a rigorous multi-annotator validation process to ensure label quality and minimize subjective bias in veracity judgments.

• Each claim was independently labeled by multiple annotators.
• Claims with disagreement were adjudicated or discarded, resulting in high inter-annotator agreement.
• The final dataset achieved a Fleiss' Kappa score indicating near-perfect agreement on the SUPPORTS/REFUTES/NOTENOUGHINFO distinction.

This methodological rigor makes FEVER a trusted benchmark where evaluation metrics reflect genuine model capability rather than annotation noise, a common pitfall in crowdsourced NLP datasets.

The official evaluation metric, the FEVER Score, jointly measures both evidence retrieval quality and veracity prediction accuracy in a single scalar value.

• A prediction is considered correct only if the model provides the right label AND cites at least one complete evidence set from the gold-standard annotations.
• This penalizes models that guess the correct label without proper evidence grounding.
• The metric enforces evidence fidelity: a model cannot achieve a high score by memorizing claim-label correlations; it must demonstrate genuine retrieval capability.

The FEVER Score has become a standard for evaluating end-to-end fact verification systems beyond the original dataset.

The prerequisite step to using FEVER. Claim Detection is the NLP task of identifying check-worthy factual assertions within unstructured text. Before a model can retrieve evidence, it must first segment a document into discrete, verifiable claims.

• Filters out opinions, questions, and subjective statements
• Often uses a check-worthiness classifier to prioritize claims
• Essential for scaling fact-checking to large document volumes

The first stage of the FEVER task itself. Evidence Retrieval involves searching a corpus (like Wikipedia) to find the most relevant text passages that can support or refute a claim. This is typically performed using dense passage retrieval or BM25 algorithms.

• Converts claims into queries for a document index
• Ranks passages by relevance before passing them to the NLI model
• Poor retrieval is the primary bottleneck in fact verification accuracy

The core reasoning engine behind FEVER. Natural Language Inference (NLI) determines whether a hypothesis (the claim) can be logically inferred from a premise (the evidence). The FEVER dataset uses a three-way classification: Supports, Refutes, or Not Enough Info.

• Distinct from semantic similarity; requires logical reasoning
• Foundation models like RoBERTa fine-tuned on MNLI are common baselines
• Handles the final veracity judgment after evidence is retrieved

A directional relationship between text fragments where the truth of a premise logically implies the truth of a hypothesis. Textual Entailment is the formal logic framework underlying NLI and is the mechanism by which FEVER models establish Supports verdicts.

• Example: Premise 'The cat is sleeping on the mat' entails Hypothesis 'A cat is on a mat'
• Contrast with contradiction and neutral relationships
• Used in legal and medical AI for high-stakes reasoning

The final machine learning task that FEVER models are evaluated on. Veracity Prediction classifies a claim as true, false, or mixed based on aggregated evidence and source reliability signals. In FEVER, this is the end-to-end output after joint retrieval and NLI.

• Aggregates multiple evidence sentences into a single judgment
• Real-world systems add source reliability scoring as a weighting factor
• Outputs are used to populate fact-check dashboards and knowledge bases

The explainability layer missing from the original FEVER task but critical for real-world deployment. Justification Production is the NLG step that summarizes the evidence and reasoning behind a veracity decision into human-readable text.

• Generates a concise explanation citing specific evidence sentences
• Transforms opaque model decisions into auditable justifications
• Required for platforms like Google Fact Check Tools and ClaimReview markup