Higher-Order Inference

Unlike first-order models that score mention pairs independently, higher-order inference repeatedly updates span representations using the embeddings of their predicted antecedents. In each iteration N, a span's representation is recomputed as a weighted combination of its original embedding and the representation of its antecedent from iteration N-1. This allows information to propagate along the coreference chain, so that a pronoun's representation eventually encodes features of the entity's first mention, even if they are separated by multiple sentences.

The key advantage of higher-order inference is its ability to perform implicit transitive closure. If mention A corefers with B, and B corefers with C, a first-order model might fail to link A directly to C if they share few surface features. Higher-order models solve this by propagating information: after refinement, A's representation will have been pulled toward B's, and B's toward C's, making A and C more similar in the learned embedding space. This enables the model to correctly cluster all three mentions into a single coreference chain.

During each refinement step, a span's new representation is computed by attending over its top-K candidate antecedents. The model computes soft attention weights over these candidates using a scoring function, then produces a weighted sum of their representations. This aggregated vector is concatenated with the span's current representation and passed through a feedforward network to produce the refined embedding. This mechanism allows the model to integrate information from multiple potential antecedents simultaneously, rather than committing to a single hard decision early in the process.

Higher-order inference is typically embedded within an end-to-end architecture that jointly performs mention detection and coreference resolution. The model scores all possible spans up to a maximum width, prunes low-scoring candidates, and then iteratively refines the surviving spans. Because refinement updates are applied to all spans in parallel, the model can simultaneously resolve multiple coreference chains within a document. The final clustering is produced by linking each mention to its highest-scoring antecedent after the last refinement iteration.

While higher-order inference significantly improves accuracy on long-distance coreference and Winograd-style challenges, it introduces a linear increase in computational cost with each refinement iteration. Typical implementations use 2-3 iterations, as accuracy gains diminish beyond this point. To manage complexity, models employ aggressive span pruning before refinement and restrict the antecedent candidate pool to a fixed window of preceding mentions. The biaffine attention mechanism used for scoring is also designed for efficient batched computation on GPU hardware.

The introduction of higher-order inference in models like e2e-coref and its successors produced a step-change improvement on the CoNLL-2012 shared task. Average F1 scores increased by 2-3 absolute points compared to first-order baselines, with the largest gains observed on pronominal resolution and long-distance coreference links. The technique has since become a standard component in state-of-the-art neural coreference systems and has been adapted for related tasks including entity linking and cross-document coreference.

The foundational architecture that enables higher-order inference. Instead of making independent pairwise decisions, this model scores all candidate antecedents for a mention and selects the highest-ranked one. Higher-order inference iteratively refines these scores by conditioning span representations on the representations of their predicted antecedents, allowing transitive reasoning across the coreference chain.

A fixed-length vector encoding a contiguous token sequence, typically computed from the hidden states of a Transformer like SpanBERT. In higher-order inference, these representations are not static. After an initial antecedent is predicted, the span's representation is updated by attending over its antecedent's representation, enabling the model to capture long-range dependencies and resolve complex split antecedents.

The complete ordered set of all mentions in a discourse that refer to a single entity. Higher-order inference is designed to build these chains with greater accuracy by propagating information across links. A correct chain resolves anaphora and cataphora seamlessly, linking a pronoun to its antecedent and that antecedent to an earlier mention, forming a coherent entity cluster.

A scoring mechanism that computes a pairwise score between a mention and a candidate antecedent using a learned bilinear transformation. In iterative higher-order models, this attention is applied repeatedly. After span representations are updated based on a coarse antecedent distribution, the biaffine scorer re-evaluates all pairs, allowing the model to correct initial errors and converge on a stable clustering.

A computational efficiency technique that restricts the candidate antecedent search space using heuristic filters based on distance, syntactic constraints, or agreement features. This is critical for making higher-order inference tractable. By pruning the search space before each iterative refinement step, the model can apply expensive, deep scoring functions only to the most plausible candidates.

A pre-training method that masks contiguous spans of tokens and predicts them using span boundary representations, optimized for span-level tasks. Its boundary-focused representations provide a superior starting point for the span vectors that higher-order inference models iteratively refine. The strong initial representations reduce the number of refinement iterations needed for convergence.