Span Representation

Span representations are typically computed by concatenating or attending over the hidden states of the span's start token and end token. This boundary-focused approach captures the syntactic and semantic boundaries of a phrase without requiring explicit segmentation. Models like SpanBERT optimize this by masking entire contiguous spans during pre-training, forcing the model to predict the masked content using only the boundary tokens. This produces representations inherently sensitive to phrase-level constituency, making them ideal for tasks like mention detection where identifying precise entity boundaries is critical.

Beyond simple concatenation of endpoints, sophisticated span representations use soft head-finding attention to compute a weighted sum over all token hidden states within the span. This mechanism learns to focus on the syntactic head word of the phrase without requiring explicit dependency parse features. The attention weights are computed as:

• Token-level scoring: Each token receives a learned score based on its hidden state
• Normalized weighting: Scores are softmax-normalized across the span
• Weighted aggregation: The final representation is a weighted sum of all token vectors This approach is more robust than rigid head-finding heuristics and adapts to the linguistic properties of different mention types.

Effective span representations often incorporate explicit span width embeddings to encode the length of the token sequence. These learned embeddings capture important priors:

• Short spans (1-3 tokens) typically correspond to named entities and pronouns
• Medium spans (4-7 tokens) often represent common noun phrases
• Long spans (8+ tokens) may indicate nested or complex syntactic constituents By concatenating a width embedding with the boundary-derived representation, models learn to distinguish between spans of different lengths that might otherwise have similar semantic content, improving mention detection precision.

Static span representations are insufficient for coreference because an entity's representation should incorporate information from its coreference chain. Higher-order inference iteratively refines span representations by attending over the representations of predicted antecedents:

• First pass: Compute initial span representations and antecedent scores
• Refinement step: Update each span's representation by attending over its top-k predicted antecedents
• Iteration: Repeat the refinement process, allowing transitive information to propagate through the chain This enables the model to learn that 'the company' and 'it' should have similar representations if they corefer, improving resolution accuracy for long-distance dependencies.

Span representations are designed to be compatible with biaffine attention scoring for efficient pairwise comparison. Given a span representation g_i for mention i and g_j for candidate antecedent j, the coreference score is computed as:

• Bilinear transformation: s(i,j) = g_i^T W g_j + U^T [g_i; g_j] + b
• The matrix W captures interaction patterns between mention and antecedent features
• The concatenated term provides a direct compatibility score This scoring function is more expressive than simple dot-product or cosine similarity, allowing the model to learn complex coreference patterns such as appositive relationships and predicate-nominative constructions.

Enumerating all possible spans in a document is computationally prohibitive, as the number grows quadratically with document length. Span representation systems employ aggressive pruning strategies:

• Span pruning: Score all possible spans up to a maximum width, then retain only the top-k spans by mention likelihood score for downstream coreference scoring
• Antecedent pruning: For each retained mention, consider only the k nearest preceding mentions as candidate antecedents, plus a dummy NA antecedent
• Distance-based filtering: Apply hard constraints on maximum antecedent distance These optimizations reduce the candidate space from O(n^2) to O(nk), making end-to-end coreference feasible on long documents while maintaining high recall.