Span Categorization

Unlike BIO Tagging schemes that assign a label to every token, span categorization models directly enumerate all possible text segments (spans) within a sentence. The model scores each candidate span for every entity type, bypassing the need for a Conditional Random Field (CRF) decoding layer. This allows the architecture to naturally handle Nested NER structures, where one entity is contained within another, without complex label encoding.

A core architectural component is the Biaffine Classifier. This layer applies a bilinear transformation to pairs of input vectors representing the start and end tokens of a candidate span. The output is a score matrix for all possible start-end token pairs, which directly represents the model's confidence that a specific token sequence constitutes a valid entity of a given type. This mechanism is central to models like Global Pointer.

Span categorization models typically construct a span representation by concatenating the contextualized embeddings of the span's start and end tokens, often derived from a transformer like BERT. A learned attention-weighted sum of all token representations within the span may also be included. This composite vector is then fed into a feedforward network to predict the entity type, providing a holistic view of the segment.

A primary advantage of this paradigm is its native ability to resolve Nested NER. Because the model independently scores every possible span, it can simultaneously identify 'Bank of America' as an Organization and 'America' as a Location within the same text. Token-level sequence labeling models struggle with this, as a single token cannot be assigned two different BIO tags simultaneously.

During training, the model computes a loss over all enumerated spans, often using Label Smoothing to prevent overconfidence. At inference time, a simple threshold is applied to the classification scores to select final entities. This eliminates the computationally expensive Viterbi Decoding step required by linear-chain CRFs, significantly speeding up prediction while maintaining high Mention-Level F1 scores.

Span categorization is conceptually related to MRC-NER (Machine Reading Comprehension for NER). In MRC-NER, a question is posed for each entity type, and the answer is extracted as a span. Span categorization generalizes this by simultaneously querying for all entity types in a single pass, using the entity type embeddings as implicit queries to score every candidate span in the text.

A pre-training method specifically designed to improve span-level representations rather than individual token representations. Unlike standard BERT's random token masking, SpanBERT masks contiguous random spans and introduces a span-boundary objective (SBO) that predicts masked tokens using only the representations of the boundary tokens. This forces the model to learn span-level semantics, making it particularly effective for span categorization tasks where accurately identifying entity boundaries is critical. Fine-tuning SpanBERT on NER benchmarks consistently yields state-of-the-art results.

A NER architecture that directly identifies entity spans by scoring all possible start-end token pairs using a multiplicative attention mechanism. For a sequence of length n, it computes an n×n scoring matrix where each cell represents the confidence that a span from token i to token j constitutes an entity of a specific type. This elegantly handles nested entities and flat NER without requiring a CRF decoding layer or BIO tagging scheme. The model uses a relative position encoding to capture distance-aware span representations.

A deep learning layer that applies a bilinear transformation to pairs of input vectors, commonly used in span categorization to score the relationship between candidate start and end tokens. Given two distinct feedforward networks that process token representations into start and end vectors, the biaffine classifier computes a score for every pair using a learned bilinear weight matrix plus linear terms. This architecture is central to span-based parsing and NER systems because it explicitly models the interaction between span boundaries rather than treating them independently.

A segment-level sequence model that directly models the probability of entire entity spans rather than individual tokens. Unlike linear-chain CRFs that operate at the token level with BIO tags, Semi-Markov CRFs assign scores to variable-length segments, naturally handling segment-level features such as span width, internal consistency, and boundary patterns. This makes them theoretically aligned with span categorization, as they bypass the need for token-level tagging schemes and can model non-contiguous or overlapping structures when extended.

The primary evaluation metric for span categorization and NER that computes the harmonic mean of precision and recall based on exact matches of both entity span boundaries and their type classifications. A predicted entity is considered correct only if its start offset, end offset, and type label all match the ground truth exactly. This metric directly validates the core objective of span categorization: simultaneously getting the boundaries and the category right. Partial credit is not awarded, making it a strict measure of end-to-end extraction quality.

The subtask of NER focused solely on identifying the start and end points of entity mentions in text, evaluated independently of type classification accuracy. This isolates the span localization problem from the categorization problem. In span categorization architectures, boundary detection is often handled by the same scoring mechanism that predicts types, but evaluating it separately reveals whether errors stem from span proposal failures or type confusion. Strong boundary detection with weak typing suggests the span enumeration mechanism is sound but the type classifier needs improvement.