Contextualized Embeddings

The defining capability of contextualized embeddings is assigning different vectors to the same word depending on usage. For example, the word 'bank' receives distinct embeddings in 'river bank' versus 'savings bank'. This is achieved by passing the entire sentence through a bidirectional language model like BERT or ELMo, where the hidden state at each position captures both left and right context. This eliminates the single-vector bottleneck of models like Word2Vec, where all senses of a polysemous word are averaged into one static representation.

Contextualized models derive their power from bidirectional processing. Unlike unidirectional models that read text left-to-right or right-to-left, architectures like BERT use a Masked Language Model (MLM) objective. During pre-training, random tokens are masked, and the model learns to predict them by attending to both preceding and following words simultaneously. This forces the hidden representations to encode full-sentence context, capturing long-range dependencies and syntactic structure that shallow or unidirectional models miss.

Different layers of a deep contextualized model encode different linguistic properties:

• Lower layers: Capture local syntax, morphology, and part-of-speech information.
• Middle layers: Encode phrase-level constituency and grammatical dependencies.
• Upper layers: Represent task-specific semantics and high-level meaning. This hierarchical structure allows downstream tasks to extract the most relevant features. For named entity recognition, lower-layer features are often more critical, while sentiment analysis benefits from upper-layer semantic abstractions.

Contextualized embeddings operate on subword units rather than full words, using algorithms like Byte-Pair Encoding (BPE) or WordPiece. This allows the model to handle out-of-vocabulary words by decomposing them into known fragments. For instance, 'unhappiness' becomes ['un', 'happiness'] or ['un', 'happi', '##ness']. The final word embedding is typically the aggregate of its subword representations, enabling the model to generalize to rare, misspelled, or novel terms without a fixed vocabulary ceiling.

Static embeddings (Word2Vec, GloVe, FastText):

• One vector per word, regardless of context.
• Cannot distinguish homonyms.
• Trained on local co-occurrence windows.

Contextualized embeddings (ELMo, BERT, GPT):

• Unique vector per word per occurrence.
• Resolves polysemy dynamically.
• Trained on full sentence or document context.
• Capture syntax, semantics, and pragmatics in a single representation. This shift from static to contextualized representations is a primary driver of the performance leap in modern NLP systems.

Contextualized embeddings are the backbone of the pre-train then fine-tune paradigm. A large transformer model is pre-trained on massive unlabeled corpora using self-supervised objectives (e.g., MLM, NSP). The resulting contextualized representations form a universal language understanding base. For a specific downstream task like question answering or textual entailment, a thin task-specific layer is added on top, and the entire model is fine-tuned end-to-end. This approach drastically reduces the need for large, task-specific labeled datasets.

The predecessor to contextualized models. Static embeddings assign a single, fixed vector to each word regardless of context. For example, the word 'bank' receives the identical vector whether referring to a river bank or a financial bank. Models like Word2Vec (2013) use shallow neural networks trained on co-occurrence prediction tasks, while GloVe (2014) factorizes a global word-word co-occurrence matrix. These models cannot resolve polysemy, making them unsuitable for tasks requiring fine-grained semantic distinctions.

The preprocessing step critical to contextualized embedding quality. Modern models use subword tokenization algorithms like Byte-Pair Encoding (BPE) or WordPiece to split rare words into meaningful subunits. For example, 'contextualized' might become ['context', '##ual', '##ized']. This approach:

• Eliminates out-of-vocabulary issues
• Allows the model to share representations across morphologically related words
• Enables handling of novel compounds and misspellings The embedding for a word is then the aggregation of its subword token embeddings, typically via mean pooling or taking the first subtoken representation.