Neural Coreference

Unlike traditional pipeline systems that separate mention detection from coreference linking, neural models jointly learn both tasks. The e2e-coref architecture enumerates all possible text spans up to a maximum length, computes a span representation for each using bidirectional LSTMs or transformers, and scores them for both mention likelihood and coreference linkage simultaneously. This joint optimization eliminates cascading errors where a missed mention in the detection phase permanently breaks a coreference chain.

Neural models encode each candidate span into a dense fixed-length vector by attending over its constituent token representations. The canonical approach concatenates:

• The hidden states of the span's start and end tokens
• A head-finding attention weighted sum over all tokens in the span
• An embedding of the span width to capture length priors This representation captures both the internal syntactic structure and semantic content of the phrase, enabling the model to recognize that "the tech giant" and "Apple" refer to the same entity without explicit string matching.

Rather than making independent pairwise coreference decisions, neural systems employ a mention-ranking model that scores all candidate antecedents for each mention and selects the highest-ranked one. The scoring function typically uses biaffine attention—a learned bilinear transformation that computes compatibility between the mention and antecedent span representations. This architecture naturally handles the asymmetry of coreference decisions and allows the model to compare multiple candidates before committing, significantly outperforming earlier mention pair models that classified each pair in isolation.

A critical innovation in neural coreference is higher-order inference, where span representations are iteratively updated based on their predicted antecedents. After an initial scoring pass, each span's representation is recomputed by attending over the representations of its top-ranked antecedents. This process repeats for multiple iterations, enabling transitive reasoning: if mention A corefers with B, and B corefers with C, the model can infer that A and C are coreferent even if they share no direct lexical overlap. This mechanism dramatically improves long-range coreference chain consistency.

SpanBERT is a transformer pretraining objective specifically designed for span-level tasks like coreference. Unlike standard BERT which masks random individual tokens, SpanBERT masks contiguous spans of tokens and trains the model to predict the masked content using only the boundary tokens' representations. Additionally, it employs a span boundary objective that forces the model to predict each token in the masked span from the boundary representations alone. This pretraining strategy produces representations that are inherently better at capturing phrase-level semantics, yielding significant gains when fine-tuned on coreference benchmarks like CoNLL-2012.

Naively considering all possible spans in a document is computationally prohibitive—a document with n tokens yields O(n²) candidate spans. Neural systems apply aggressive span pruning and antecedent pruning to maintain tractability:

• Span pruning: A lightweight mention score filters out spans unlikely to be mentions, typically keeping only the top λ spans per token position
• Antecedent pruning: For each mention, only the K nearest preceding spans are considered as candidate antecedents, exploiting the strong locality prior in coreference These heuristics reduce complexity from quadratic to near-linear while retaining over 95% of gold mentions.

The dominant neural architecture that scores all candidate antecedents for a given mention and selects the highest-ranked one. Unlike pairwise models, this approach considers the entire candidate set jointly.

• Uses biaffine attention to compute pairwise scores between mention and antecedent representations
• Enables higher-order inference by iteratively refining span representations based on predicted antecedents
• Outperforms mention-pair models by avoiding independent classification errors

A pre-training method optimized for span-level tasks like coreference resolution. Instead of masking random tokens, SpanBERT masks contiguous spans and predicts them using span boundary representations.

• Trains the model to reason about entire entity mentions rather than individual tokens
• Achieved state-of-the-art results on CoNLL-2012 when combined with e2e-coref
• Span boundary objective forces the model to learn span-level semantics

An iterative refinement technique where span representations are updated based on the representations of their predicted antecedents. This enables transitive reasoning across coreference chains.

• If mention A corefers with B, and B corefers with C, the model learns that A corefers with C
• Typically implemented through multiple iterations of the mention-ranking scorer
• Critical for resolving long coreference chains spanning multiple sentences

Efficiency methods that reduce the candidate mention space before scoring. Neural models consider all possible spans up to a maximum length, creating a quadratic explosion of candidates.

• Span pruning: Filter spans with low mention likelihood scores using a lightweight scorer
• Antecedent pruning: Restrict candidate antecedents by distance, syntactic constraints, or agreement features
• Reduces computational cost from O(n⁴) to O(n³) or lower while maintaining accuracy

The standard evaluation dataset for coreference resolution derived from OntoNotes 5.0. It provides gold-standard annotations across multiple genres including newswire, broadcast, and telephone conversations.

• Evaluates using MUC, B³, and CEAF metrics averaged into the CoNLL F1 score
• Contains annotated coreference chains, named entities, and syntactic parses
• The definitive benchmark for comparing neural coreference architectures

The canonical end-to-end neural coreference model by Lee et al. (2017) that jointly performs mention detection and coreference scoring using learned span representations.

• Computes span embeddings by concatenating BiLSTM hidden states of start and end tokens with attention-pooled internal representations
• Uses a head-finding heuristic to identify syntactic heads for feature extraction
• Foundation architecture that subsequent neural coreference systems build upon and improve