SpanBERT

Unlike standard BERT which masks random individual tokens, SpanBERT masks contiguous random spans of text. This forces the model to predict entire multi-token sequences based on surrounding context, directly training it to reason about phrase-level semantics. For example, in the sentence 'The New York Stock Exchange closed higher,' SpanBERT might mask the entire span 'New York Stock Exchange' rather than masking 'York' and 'Stock' independently. This is critical for NER where entities are often multi-word expressions.

The SBO is a novel auxiliary training task that requires the model to predict each masked token using only the representations of the tokens at the boundaries of the masked span. Specifically:

• The model observes the hidden states of the external boundary tokens (immediately before and after the span)
• It also uses the relative position of the target token within the span
• This compels the model to encode span-level information directly into the boundary token representations This objective teaches the model that boundary tokens are information-rich anchors for understanding entire phrases.

SpanBERT departs from BERT's practice of concatenating two half-length documents with a 50% chance. Instead, it trains exclusively on single, full-length sequences without the Next Sentence Prediction (NSP) objective. This design choice ensures that masked spans are always contiguous and contextually coherent within a single document, preventing the model from learning spurious cross-document correlations. The removal of NSP also simplifies the training pipeline and focuses the model entirely on span-level understanding within a unified context.

Span lengths are sampled from a geometric distribution rather than a uniform distribution. This means shorter spans are sampled much more frequently than longer ones, but the model still occasionally encounters very long spans. The distribution is parameterized by a prior probability p (typically 0.2), which controls the expected span length. This approach:

• Mirrors the natural distribution of entity lengths in real text
• Ensures the model is robust to both short entities (e.g., 'Apple') and long nominal phrases (e.g., 'the United States Department of Justice')
• Prevents the model from overfitting to a fixed span length

SpanBERT achieved significant performance gains over BERT-base and BERT-large on core span selection benchmarks without task-specific architectural modifications:

• CoNLL-2003 NER: Outperformed BERT by 2.8 F1 points
• SQuAD 1.1: Improved exact match by 2.0 points
• SQuAD 2.0: Improved F1 by 3.3 points
• Coreference Resolution (OntoNotes): Achieved a 3.3 F1 point gain These results demonstrate that span-level pre-training provides a stronger inductive bias for any downstream task requiring the identification and classification of text spans.

SpanBERT is not a new architecture but a pre-training methodology applied to the standard Transformer encoder. It is fully compatible with the BERT architecture and can be used as a drop-in replacement for any BERT-based model. Key implications:

• No additional parameters or inference latency compared to BERT
• Can be fine-tuned using the same pipelines and hyperparameters
• Compatible with standard token classification heads (e.g., linear layer + CRF) for NER
• The boundary representations learned via SBO can be directly leveraged for span enumeration architectures like Global Pointer or Biaffine Classifiers

Unlike standard BERT which masks random individual tokens, SpanBERT masks contiguous random spans of text. This forces the model to predict entire multi-token sequences based on surrounding context, directly training it to reason about phrase-level semantics rather than just individual word identities. The geometric distribution used for span length sampling ensures a bias toward shorter, linguistically meaningful phrases.

A novel auxiliary training loss that teaches the model to predict each masked token using only the representations of the tokens at the span's boundary (the tokens immediately before and after the masked span). This structural inductive bias explicitly encodes the intuition that span content is heavily determined by its immediate context, improving performance on tasks requiring precise boundary detection like Named Entity Recognition and Extractive Question Answering.

SpanBERT departs from BERT's Next Sentence Prediction (NSP) objective by training exclusively on single, long contiguous sequences (up to 512 tokens). This design choice eliminates the artificial noise from NSP and allows the model to learn richer, cross-sentence dependencies within a document. The removal of NSP was proven to be crucial for span-level tasks, as the model focuses entirely on intra-textual coherence rather than sentence-pair relationships.

The key architectural divergence lies in the pre-training objectives. While both share the Transformer encoder backbone, SpanBERT replaces BERT's token-level masking and NSP with span-level masking and the Span Boundary Objective. This results in significantly superior performance on span selection tasks like CoNLL-2003 NER and SQuAD 2.0, while maintaining comparable performance on GLUE benchmarks, proving the targeted efficacy of its design.