SpanBERT

Unlike standard BERT which masks random individual tokens, SpanBERT masks contiguous random spans of text. This forces the model to predict entire sequences of masked words based on surrounding context, teaching it to reason about phrase-level semantics rather than relying on local collocation statistics. The span lengths are sampled from a geometric distribution, with a bias toward shorter spans to maintain a balanced training signal.

The SBO is a key auxiliary loss function that requires the model to predict each masked token using only the representations of the tokens at the boundaries of the masked span. This explicitly teaches the model to encode span-level information at its edges, which is critical for downstream tasks that rely on identifying the start and end of entity mentions or argument spans. The final representation for a masked token is a function of the boundary tokens and its relative position within the span.

SpanBERT abandons the Next Sentence Prediction (NSP) objective used in original BERT. Instead, it pre-trains on single, long contiguous text sequences of up to 512 tokens, sampled from document boundaries. This design choice was motivated by the finding that NSP is detrimental or neutral for span-oriented tasks. By removing the artificial 50/50 sentence-pair construction, the model learns better long-range discourse-level dependencies within a single coherent context.

SpanBERT achieves state-of-the-art results on several span selection benchmarks without task-specific architectural modifications. On TACRED, a challenging relation extraction dataset, SpanBERT substantially outperforms BERT-base and BERT-large. This performance gain is directly attributed to the span masking and SBO objectives, which provide a better inductive bias for identifying the entity spans that serve as arguments to a relation classifier.

The architecture demonstrates exceptional performance on coreference resolution, the task of clustering all mentions that refer to the same real-world entity. On the OntoNotes benchmark, SpanBERT-large sets a new standard by learning to represent entire mention spans effectively. The SBO is particularly well-suited here, as coreference decisions often hinge on the semantic content concentrated at the head and tail of a noun phrase.

The key differentiators from original BERT are threefold:

• Masking Strategy: Contiguous spans vs. random subword tokens
• Auxiliary Loss: Span Boundary Objective vs. Next Sentence Prediction
• Input Format: Single contiguous sequences vs. concatenated sentence pairs These changes are conceptually simple but represent a fundamental shift in pre-training philosophy from token-level language modeling to span-level semantic understanding, making SpanBERT a specialized tool for information extraction pipelines.