AIDA CoNLL-YAGO is the canonical evaluation dataset for the entity linking task, created by annotating the CoNLL-2003 shared task corpus with links to the YAGO knowledge base. It contains 1,393 Reuters news articles partitioned into training, validation, and test sets, with each named entity mention manually disambiguated to a unique YAGO identifier by human annotators.
Glossary
AIDA CoNLL-YAGO
The standard benchmark dataset for evaluating entity linking systems, consisting of Reuters news articles with hand-labeled mentions linked to YAGO entities.
BENCHMARK DATASET
What is AIDA CoNLL-YAGO?
The AIDA CoNLL-YAGO dataset is the standard benchmark for evaluating entity linking systems, consisting of Reuters news articles with hand-labeled mentions linked to YAGO entities.
The dataset evaluates a system's ability to perform disambiguation by linking ambiguous surface forms to the correct entity in a large-scale taxonomy. Performance is measured using micro and macro precision, recall, and F1 score, with a dedicated NIL prediction mechanism for mentions that lack a corresponding entry in the knowledge base.
BENCHMARK ARCHITECTURE
Key Characteristics
The AIDA CoNLL-YAGO dataset is the definitive benchmark for entity linking systems, providing a rigorous evaluation framework based on hand-annotated newswire text mapped to the YAGO knowledge base.
01
Corpus Composition
The dataset is built on Reuters newswire articles from the CoNLL 2003 shared task, repurposed for entity linking evaluation. It contains 1,393 documents with 34,956 annotated mentions spanning diverse topics including politics, sports, and business. Each mention is manually labeled by human annotators and linked to a specific YAGO entity identifier, providing a gold-standard evaluation set. The corpus is divided into three official splits: a training set (946 documents), a development set (216 documents), and a test set (231 documents), enabling standardized model comparison.
34,956
Annotated Mentions
1,393
Reuters Documents
03
NIL Detection Requirement
A critical and challenging aspect of the benchmark is NIL prediction. Not every mention in the text has a corresponding entity in the YAGO knowledge base. Systems must correctly identify these out-of-knowledge-base mentions and label them as NIL rather than forcing an incorrect link. This evaluates a model's ability to recognize the boundaries of its own knowledge, a crucial capability for production systems operating on real-world text where knowledge bases are inherently incomplete.
~35%
NIL Mentions in Test Set
04
Evaluation Metrics
Performance is measured using micro-averaged accuracy and macro-averaged accuracy across all mentions. The primary metric is strong annotation match: a prediction is correct only if the system links a mention to the exact YAGO entity specified in the gold standard. Partial matches or linking to a related but incorrect entity are counted as errors. This strict scoring ensures that systems are evaluated on precise disambiguation rather than approximate topical relevance, setting a high bar for state-of-the-art performance.
05
Candidate Generation Protocol
The benchmark provides a standard candidate list for each mention, generated using a combination of surface form matching against Wikipedia anchor texts and YAGO entity labels. This protocol ensures fair comparison between systems by controlling for differences in candidate retrieval. Systems must rank these candidates and select the correct one or predict NIL. The candidate sets are deliberately constructed to include challenging confusable entities that share identical or highly similar surface forms, testing the depth of a model's semantic understanding.
AIDA CONLL-YAGO BENCHMARK
Frequently Asked Questions
Essential questions about the foundational dataset used to evaluate entity linking and disambiguation systems, covering its structure, annotation methodology, and role in advancing semantic search.
The AIDA CoNLL-YAGO dataset is the standard benchmark corpus for evaluating entity linking and disambiguation systems. It consists of 1,393 Reuters newswire articles (946 for training, 216 for validation, and 231 for testing) where every named entity mention has been manually annotated and linked to its corresponding unique entry in the YAGO knowledge base. Its importance stems from being the first large-scale, high-quality dataset that enabled rigorous, reproducible comparison between entity linking approaches. The dataset contains 34,956 hand-labeled mentions spanning diverse entity types including persons, organizations, locations, and miscellaneous entities, with annotators achieving high inter-annotator agreement. Researchers use AIDA to measure micro and macro F1 scores, precision@k, and NIL prediction accuracy, making it the de facto standard for tracking progress in the field since its introduction at the 2011 CoNLL Shared Task.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us