e2e-coref

Unlike traditional pipeline systems that separate mention detection from coreference resolution, e2e-coref performs both tasks simultaneously. The model considers all possible spans up to a maximum width, scores each for mention likelihood, and then scores pairs of spans for coreference. This joint approach allows the model to learn that certain spans are better mentions precisely because they participate in coreference chains, and vice versa.

• Eliminates cascading errors from separate mention detection modules
• Considers all n-grams up to length L as candidate mentions
• Uses a shared span representation for both subtasks

Each candidate span is encoded into a fixed-length vector by concatenating the bidirectional LSTM hidden states of its first and last tokens, along with an attention-weighted sum over all tokens in the span. An additional feature vector encoding the span's width is appended. This representation captures both boundary information and internal content without relying on syntactic parse trees.

• Start token embedding: h_start
• End token embedding: h_end
• Soft head attention: weighted sum over span tokens
• Span width feature: learned embedding of span length

Coreference decisions are made using a mention-ranking approach rather than pairwise classification. For each mention, the model scores all candidate antecedents using a learned scoring function and selects the highest-scoring one. The scoring function combines a biaffine attention mechanism with a coarse pairwise feature vector that captures speaker and genre metadata.

• Score(m, a) = s_m(m) + s_m(a) + s_pair(m, a)
• s_pair uses a biaffine transformation: span_m^T * W * span_a
• Includes a dummy antecedent for first mentions or non-coreferent spans
• Trained with a marginal log-likelihood objective over correct antecedents

The model applies iterative span representation updates where each span's embedding is recomputed as a weighted average of its original representation and the representations of its top-scoring antecedents from the previous iteration. This enables transitive reasoning: if mention A corefers with B, and B corefers with C, the model can learn that A should also corefer with C.

• N iterations of refinement (typically 2-3)
• Span i at iteration t: f(span_i^(t-1), antecedent_span^(t-1))
• Enables implicit chain-level reasoning without explicit clustering
• Improves resolution of long-distance coreference chains

The model is trained to maximize the marginal log-likelihood of all correct antecedents for each mention. The loss function sums over all mentions and considers the set of all spans that are true coreferent antecedents. This objective naturally handles the case where a mention has multiple correct antecedents (e.g., in a chain) and gracefully manages the dummy antecedent for non-coreferent or first mentions.

• Loss: sum over mentions of log(sum over true antecedents of exp(score))
• No explicit negative sampling required
• Jointly optimizes mention detection and coreference accuracy
• Trained on the CoNLL-2012 shared task dataset (OntoNotes 5.0)

To manage the quadratic explosion of candidate spans, e2e-coref applies aggressive span pruning after mention scoring. Only the top-k spans by mention score are retained for the coreference scoring phase. Additionally, antecedent pruning restricts each mention to consider only the nearest k antecedents, based on the observation that most coreference links are local.

• Top-λ spans retained after mention scoring (λ ≈ 0.4 * tokens)
• Maximum antecedent distance: typically 50-100 mentions back
• Reduces complexity from O(n^4) to approximately O(n^2)
• Pruning decisions are learned, not heuristic

The foundational technique for encoding arbitrary text spans into fixed-length vectors, enabling the model to score mentions and antecedents without relying on pre-parsed syntactic trees.

• Endpoint Concatenation: The representation of span i to j is computed by concatenating the hidden states of its start and end tokens from a biLSTM or Transformer.
• Head-Finding Attention: A learned attention mechanism over the tokens within a span, weighted toward the syntactic head word, is often added to the endpoint representation for richer features.
• Feature Enrichment: The final span vector is typically concatenated with hand-crafted features like span length and genre metadata before scoring.

An iterative refinement technique that allows the model to perform transitive reasoning across coreference chains, moving beyond first-order antecedent decisions.

• Iterative Refinement: Span representations are updated based on the representations of their predicted antecedents from the previous iteration.
• Transitive Closure: If mention A is coreferent with B, and B with C, higher-order inference allows the model to implicitly learn that A and C are also coreferent.
• Implementation: Typically involves running the antecedent scoring step multiple times, using the weighted sum of antecedent representations as the new span representation for the next iteration.

A critical computational efficiency step that drastically reduces the number of candidate mention spans considered by the model from O(n²) to a manageable constant.

• Mention Scoring: A lightweight feedforward network assigns a 'mention-ness' score to every possible span up to a maximum width.
• Top-K Filtering: Only the top K spans (e.g., 0.4 * document length) by mention score are retained as candidates for the coreference scoring stage.
• Impact: This step is essential for making end-to-end models computationally feasible on long documents without sacrificing recall on the most likely entity mentions.

A pre-training method specifically designed to improve performance on span-level tasks like coreference resolution, serving as a powerful text encoder for the e2e-coref architecture.

• Span Masking: Masks contiguous random spans of tokens rather than individual subword tokens, forcing the model to predict the entire span content.
• Span Boundary Objective (SBO): A novel training objective that requires the model to predict each masked token using only the representations of the tokens at the span's boundary.
• Performance Gain: Replacing the original biLSTM encoder with SpanBERT significantly boosts coreference resolution accuracy, particularly on long-distance dependencies.