CoNLL-2012

The corpus is built directly from the OntoNotes 5.0 dataset, a large-scale, multi-genre corpus. It provides rich, integrated annotations including syntax, predicate-argument structure, word senses, and coreference chains. The data spans English, Chinese, and Arabic, though the shared task focused on English. Genres include newswire, broadcast news, broadcast conversation, telephone conversation, weblogs, and pivot text from the Bible.

The official evaluation scenario provides models with gold (human-annotated) mentions. This separates the task of mention detection from the core task of linking. Systems are scored purely on their ability to correctly cluster these pre-identified spans into entities. This paradigm allows for a focused evaluation of the coreference linking algorithm itself, isolating it from the noise of imperfect mention detection.

Performance is evaluated using the average of three complementary metrics:

• MUC (Message Understanding Conference): A link-based metric that heavily penalizes errors in large chains.
• B³ (Bagga & Baldwin): A mention-based precision and recall metric that evaluates the correctness of individual mention assignments.
• CEAFₑ (Constrained Entity-Alignment F-Measure): An entity-based metric that finds the optimal one-to-one alignment between predicted and gold entities using a mention-similarity criterion. The final score is the CoNLL F1, the unweighted average of these three metrics.

A secondary evaluation track requires systems to perform end-to-end coreference, starting from raw text. Models must first detect all candidate mentions (mention detection) and then cluster them into coreference chains. This is a significantly harder task, as errors in mention identification cascade directly into linking errors. Systems are evaluated using the same MUC, B³, and CEAFₑ metrics, but on the full system output.

A critical design choice is that singleton entities are excluded from the official evaluation. A singleton is an entity mentioned exactly once in a document. The gold data contains these, but they are removed for scoring. This means a model is not penalized for failing to cluster a lone mention, focusing the evaluation on the resolution of multi-mention entities and the quality of the chains that are formed.

The dataset defines canonical train, development, and test splits to ensure reproducible research. The English portion is segmented into:

• Train: 2,802 documents
• Dev: 343 documents
• Test: 348 documents These splits are stratified by genre to ensure a representative distribution of linguistic phenomena across all sets, preventing genre-specific overfitting.

The dominant neural architecture for CoNLL-2012 tasks. Instead of making independent pairwise decisions, this model scores all candidate antecedents for a given mention and selects the highest-ranked one.

• Uses learned span representations to encode mentions
• Employs biaffine attention for pairwise scoring
• Enables higher-order inference for transitive reasoning across chains
• Achieved state-of-the-art results on the CoNLL-2012 shared task

A pre-training method specifically optimized for span-level tasks like coreference resolution on CoNLL-2012. Unlike standard BERT, SpanBERT masks contiguous spans of tokens rather than individual tokens.

• Predicts masked spans using span boundary representations
• Trained on a span boundary objective that forces the model to encode span-level information
• Significantly outperforms standard BERT on CoNLL-2012 benchmark
• Provides the foundational encoder for modern coreference systems

The fundamental output structure evaluated in CoNLL-2012. A coreference chain is the complete ordered set of all mentions within a document that refer to a single entity.

• Begins with the first mention (often a proper name or full noun phrase)
• Includes all subsequent pronominal references (he, she, it)
• May contain bridging anaphora inferentially linked to the discourse referent
• CoNLL-2012 metrics (MUC, B³, CEAF) evaluate chain quality

An iterative refinement technique critical for CoNLL-2012 performance. Span representations are updated based on the representations of their predicted antecedents, enabling transitive reasoning.

• First-order: independent pairwise scoring
• Second-order: spans attend to their top-ranked antecedents
• Enables resolution of split antecedents where plural pronouns refer to multiple entities
• Dramatically improves singleton entity identification
• Key innovation in the e2e-coref architecture by Lee et al.

The prerequisite subtask for CoNLL-2012 evaluation. Identifies all spans of text that refer to an entity before coreference chains can be built.

• Span pruning filters low-likelihood candidate spans to reduce computational cost
• Head-finding heuristics identify the syntactic head word of each mention
• Modern systems perform joint mention detection and coreference scoring
• CoNLL-2012 gold mentions include named entities, pronouns, and nominal phrases
• Quality of mention detection directly impacts downstream coreference metrics

A pronoun disambiguation challenge that tests the commonsense reasoning capabilities required for CoNLL-2012. Two sentences differ by a single word that flips the pronoun's antecedent.

• Example: "The trophy doesn't fit in the suitcase because it is too big." vs. "...because it is too small."
• Requires deep world knowledge beyond syntactic patterns
• Exposes limitations of models trained purely on CoNLL-2012 statistical patterns
• Used as a diagnostic benchmark for genuine language understanding