BERT-NER

Unlike static word embeddings, BERT generates dynamic representations where the vector for a word changes based on its surrounding context. The word 'Apple' receives a different embedding in 'Apple released a new iPhone' versus 'Apple is a healthy fruit'. This is achieved through the bidirectional Transformer encoder which applies self-attention across all tokens in the input sequence simultaneously, allowing each token to attend to both its left and right context. The final hidden states from the last Transformer layer serve as the contextualized features fed into the classification head.

The architecture appends a simple yet effective linear classification layer on top of BERT's output. For each token representation h_i produced by the final Transformer layer, the model computes a softmax probability over the NER label space:

• B-PER: Beginning of a person entity
• I-ORG: Inside an organization entity
• O: Outside any entity This layer is trained end-to-end with the entire BERT model during fine-tuning, allowing the pre-trained representations to adapt specifically to the NER task while maintaining the rich linguistic knowledge acquired during pre-training.

BERT-NER uses WordPiece tokenization, which splits rare or unknown words into smaller subword units. For example, 'Transformers' might be tokenized as ['Transform', '##ers']. This creates a mismatch between tokens and entity labels since NER annotations typically operate at the word level. The standard solution is to:

• Assign the entity label only to the first subword token of a word
• Label subsequent subword tokens with a special ignore tag or propagate the same label This subword handling enables the model to generalize to unseen words and handle morphological variations effectively.

BERT-NER employs the fine-tuning paradigm rather than using BERT as a frozen feature extractor. During training, all parameters of both the pre-trained BERT encoder and the newly initialized classification layer are updated via backpropagation. This joint optimization allows the model to:

• Adapt attention patterns to focus on entity-relevant context
• Adjust intermediate representations for sequence labeling objectives
• Achieve superior performance compared to feature-based extraction Fine-tuning typically requires only 2-4 epochs on NER datasets like CoNLL-2003 to reach state-of-the-art results.

BERT's standard maximum sequence length is 512 tokens, which can be limiting for long documents. BERT-NER implementations address this through:

• Sliding window approaches that process overlapping text segments
• Strided inference where predictions from overlapping windows are merged using majority voting or confidence scoring
• Document-level aggregation that maintains entity consistency across sentence boundaries For production systems, models like Longformer or BigBird with extended attention mechanisms can replace BERT to handle longer contexts natively without windowing.

While the base BERT-NER architecture uses a simple softmax classifier, many implementations add a Conditional Random Field (CRF) layer on top for structured prediction. The CRF models dependencies between adjacent label predictions, preventing invalid transitions like I-PER following B-ORG. This joint decoding approach:

• Enforces label sequence consistency through learned transition probabilities
• Uses Viterbi decoding at inference to find the globally optimal label path
• Typically yields a 0.5-1.5 F1 point improvement on standard benchmarks The trade-off is increased computational complexity during both training and inference.

An alternative to token-level BIO tagging that directly enumerates and classifies arbitrary text spans. Instead of predicting a label for every token, the model scores all possible start-end token pairs using a biaffine classifier or global pointer mechanism. This approach naturally handles nested NER and avoids the label inconsistency issues of sequence labeling.

A paradigm where BERT-NER models generalize from only a handful of labeled examples per entity type. Techniques include:

• Prompt-based NER: reformulating extraction as a masked language modeling task using templates.
• Prototypical networks: computing distances between token embeddings and class prototypes in representation space.
• Contrastive pre-finetuning: training on diverse NER datasets to learn a general entity extraction capability before adapting to the target domain.

The process of adjusting a BERT-NER model trained on a source domain (e.g., newswire text) to perform on a target domain (e.g., clinical notes) with distinct entity types like medication and dosage. Strategies include:

• Continued pre-training on in-domain text before fine-tuning.
• Multi-task learning with auxiliary objectives like masked language modeling.
• Adversarial training to learn domain-invariant token representations.