SpanBERT

Unlike standard BERT which masks random individual tokens, SpanBERT masks contiguous random spans of text. This forces the model to rely on the structural context of the entire span rather than memorizing local collocations. By masking full noun phrases or entity mentions, the model learns to predict the semantic content of a whole segment based on its boundaries, directly mimicking the requirements of mention detection and coreference linking.

The core innovation of SpanBERT is the Span Boundary Objective. Rather than predicting each masked token independently using its own position's output vector, SBO predicts each token in the masked span using only the output representations of the tokens immediately outside the span (the start and end boundaries). This explicitly teaches the model to encode the meaning of a phrase into its boundary tokens, which is precisely the information needed to build high-quality span representations for coreference scoring.

SpanBERT abandons the Next Sentence Prediction (NSP) objective used in original BERT. Instead, it pre-trains on single contiguous text segments of up to 512 tokens, sampled to respect sentence boundaries. This design choice was driven by the finding that NSP is detrimental to span-level tasks. By removing the artificial sentence-pair format, the model learns long-range discourse dependencies within a single document, which is critical for resolving coreference chains that cross sentence boundaries.

SpanBERT established a new state-of-the-art on the CoNLL-2012 shared task for coreference resolution upon its release. The combination of span masking and the SBO objective proved uniquely effective. Key results include:

• Significant F1 gains over standard BERT on the OntoNotes 5.0 benchmark.
• Strong performance without task-specific architectural modifications, demonstrating that the pre-training objective alone was responsible for the improvement.
• The learned representations proved to be robust, general-purpose features for any task requiring entity-level understanding.

The core pre-training innovation of SpanBERT. Instead of masking random individual tokens like standard BERT, contiguous spans of tokens are masked. The model must predict the entire masked span using only the representations of the boundary tokens (the tokens immediately before and after the span). This forces the model to learn span-level semantics rather than relying on local token co-occurrence statistics.

• Example: In 'The quick brown fox jumps', masking 'quick brown' forces the model to predict the phrase from 'The' and 'fox'.
• Benefit: Directly optimizes for span representation quality, crucial for tasks like coreference resolution.

A novel auxiliary training loss used alongside the standard Masked Language Model (MLM) loss. For each masked span, the SBO uses the output representations of the boundary tokens (the tokens immediately outside the masked span) to predict each token inside the span. This explicitly teaches the model to encode span content into its boundary representations.

• Mechanism: A position embedding marks the relative offset of the target token from the left boundary.
• Impact: Eliminates the need for the model to attend to individual masked tokens, relying purely on the structural boundary.

The primary downstream task where SpanBERT achieves state-of-the-art performance. This NLP task involves identifying all mentions (spans of text) that refer to the same real-world entity and clustering them into coreference chains. SpanBERT excels here because its span boundary representations naturally capture the semantic content of a phrase.

• Example: In 'Alice went to the store. She bought milk.', SpanBERT links 'She' to 'Alice'.
• Benchmark: Achieved significant F1 score gains on the CoNLL-2012 shared task.

Unlike standard BERT, which often uses two concatenated segments with a Next Sentence Prediction (NSP) objective, SpanBERT is trained on single, long contiguous sequences of up to 512 tokens. This design choice is deliberate: NSP is not beneficial for span-level tasks, and single sequences prevent the model from creating spurious correlations between unrelated text segments.

• Advantage: Maximizes the length of coherent context available for predicting long spans.
• Efficiency: Simplifies the data preprocessing pipeline by removing the need for sentence segmentation.

A fixed-length vector encoding the meaning of a contiguous token sequence. In SpanBERT, a span's representation is typically computed by concatenating the hidden states of its start and end tokens along with a learned attention-based soft head vector. This representation is then fed into a feedforward network for scoring.

• Formula: g_i = [x_{START(i)}; x_{END(i)}; \hat{x}_i]
• Usage: These vectors are scored in a mention-ranking model to determine antecedent links.

The neural architecture typically paired with SpanBERT for coreference tasks. Instead of making independent pairwise decisions, this model generates candidate spans, computes their span representations using SpanBERT, and then scores each candidate antecedent for a given mention. The highest-scoring antecedent is selected, or a dummy 'NA' antecedent is chosen if no link exists.

• Process: Mention Detection → Span Pruning → Antecedent Scoring.
• Integration: SpanBERT provides the high-quality span embeddings that make this architecture highly effective.