The CoNLL-2012 Shared Task was a competitive evaluation focused on modeling multilingual unrestricted coreference and semantic role labeling using the OntoNotes 5.0 corpus. It required systems to jointly predict entity mentions, coreference chains, and predicate-argument structures across English, Chinese, and Arabic, establishing a unified end-to-end information extraction pipeline benchmark.
Glossary
CoNLL-2012 Shared Task
A landmark benchmarking challenge based on the OntoNotes corpus that drove significant progress in end-to-end coreference resolution and semantic role labeling systems.
BENCHMARKING CHALLENGE
What is CoNLL-2012 Shared Task?
A landmark benchmarking challenge based on the OntoNotes corpus that drove significant progress in end-to-end coreference resolution and semantic role labeling systems.
This task drove the adoption of joint inference models that resolve syntactic and semantic dependencies simultaneously, rather than as isolated pipeline stages. Its official evaluation metric, the average F1 score across coreference, semantic role labeling, and named entity recognition subtasks, remains a standard for measuring holistic document-level natural language understanding.
CoNLL-2012 Shared Task
Key Features of the Benchmark
The CoNLL-2012 Shared Task established a unified evaluation framework for end-to-end coreference resolution and semantic role labeling using the multi-genre OntoNotes corpus, driving a decade of architectural innovation.
01
Unified OntoNotes Corpus
The task standardized training and evaluation on OntoNotes 5.0, a large-scale, multi-genre corpus. This forced models to generalize across:
- Newswire (Wall Street Journal)
- Broadcast Conversation (ABC, CNN)
- Web Text (Blogs)
- Telephone Speech (Switchboard) The diverse genres tested robustness against informal syntax and disfluencies.
02
Dual-Task Evaluation
Unlike prior shared tasks, CoNLL-2012 required systems to perform two interdependent tasks simultaneously:
- Coreference Resolution: Clustering mentions into entity chains.
- Semantic Role Labeling: Assigning predicate-argument structures. The official evaluation metric, CEAF for coreference and F1 for SRL, measured performance on gold-standard syntactic parses.
03
Closed vs. Open Track
The task featured two distinct competition tracks to isolate the impact of external resources:
- Closed Track: Systems could only use the provided OntoNotes data and specified preprocessing tools.
- Open Track: Participants could leverage any external corpora, lexicons (e.g., FrameNet, VerbNet), or unannotated text. This design rigorously measured the value of feature engineering versus raw data scale.
04
Mention Detection Scoring
A critical sub-task evaluated was gold mention detection. Systems were scored on their ability to identify the exact spans of entity mentions before resolving them. This decomposed the pipeline into:
- Boundary Identification: Finding the start and end tokens.
- Type Classification: Labeling the mention as Named Entity, Nominal, or Pronominal. High precision here was a prerequisite for accurate coreference chains.
05
Legacy of Span-Based Architectures
The CoNLL-2012 dataset became the primary benchmark for neural coreference resolution. The winning system by Fernandes et al. (2012) used deterministic clustering, but the task later catalyzed:
- SpanBERT: Pre-training on span-boundary objectives.
- Higher-order Inference: Iteratively refining mention representations.
- End-to-end Neural Models: Replacing syntactic parsers with raw text encoders.
UNDERSTANDING THE BENCHMARK
Frequently Asked Questions
Explore the foundational questions about the CoNLL-2012 Shared Task, the landmark challenge that standardized the evaluation of end-to-end coreference resolution and semantic role labeling on the OntoNotes corpus.
The CoNLL-2012 Shared Task was a landmark benchmarking challenge focused on modeling multilingual unrestricted coreference in the OntoNotes corpus. It required participating systems to perform end-to-end coreference resolution, identifying all noun phrase mentions in a text and clustering them into equivalence classes that refer to the same real-world entity. The task uniquely integrated semantic role labeling (SRL) as a supporting annotation layer, pushing systems to jointly model entity identity and predicate-argument structure across English, Chinese, and Arabic. The official evaluation metric was the MUC, B³, and CEAF-based CoNLL F1 score, which averages these three complementary measures to assess both mention detection and coreference linking quality.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us