Token Embedding

Each token ID is mapped to a fixed-size, dense vector of real numbers (e.g., 768 dimensions for BERT-base). Unlike sparse one-hot encodings, dense embeddings pack rich semantic information into every dimension. The model learns these values during training, positioning tokens with similar meanings close together in the vector space. This compact representation allows neural networks to efficiently process and compare linguistic units.

The primary power of token embeddings lies in capturing distributional semantics—the principle that words appearing in similar contexts have similar meanings. In the embedding space, king and queen are closer to each other than to car. This geometric property enables models to generalize across synonyms and related concepts without explicit programming. Cosine similarity between embedding vectors is the standard metric for quantifying this semantic closeness.

Well-trained embeddings exhibit linear substructures that encode relational patterns. The classic example: king - man + woman ≈ queen. This vector arithmetic demonstrates that embeddings capture not just similarity but also abstract relationships like gender, tense, or capital-country pairs. This emergent property allows models to perform analogical reasoning by manipulating vectors in the latent space.

A critical distinction exists between static embeddings (Word2Vec, GloVe) and contextual embeddings (BERT, GPT). Static models assign a single, fixed vector to each token regardless of context—so bank has one representation. Contextual models generate dynamic vectors where bank in 'river bank' differs from bank in 'savings bank'. This context-dependence resolves polysemy and captures nuanced syntactic roles.

Token embeddings serve as the initial input layer of Transformer architectures. The raw token ID is used to look up its corresponding vector from the embedding matrix—a learned parameter of shape (vocabulary_size, embedding_dimension). This vector is then summed with positional encodings before being fed into the self-attention layers. The quality of this initial representation profoundly impacts the model's downstream performance.

Embeddings are shaped by the training objective. Masked Language Modeling (BERT) forces embeddings to encode bidirectional context to predict missing tokens. Causal Language Modeling (GPT) trains embeddings to predict the next token from prior context. Contrastive objectives (SimCSE) explicitly pull embeddings of semantically similar sentences together while pushing dissimilar ones apart, directly optimizing the vector space structure.