Vector Embedding

Embeddings compress high-dimensional sparse representations (like one-hot encoded vocabularies of 50,000+ tokens) into dense vectors typically ranging from 256 to 4096 dimensions. This compression forces the model to learn latent semantic features where semantically similar concepts occupy nearby regions in the continuous vector space.

• BERT embeddings: 768 or 1024 dimensions
• OpenAI text-embedding-3-large: up to 3072 dimensions
• Image embeddings (CLIP): typically 512 dimensions

The reduced dimensionality enables efficient storage in vector databases and fast approximate nearest neighbor search.

Vector embeddings exhibit linear semantic relationships in their latent space. The classic example is the vector arithmetic: king - man + woman ≈ queen. This property emerges because the training objective forces the model to encode relational information as directional vectors.

• Analogy solving: The offset vector between word pairs encodes the relationship itself
• Cross-modal alignment: CLIP embeddings align text and image vectors in a shared space where image_of_dog ≈ text_embedding('dog')
• Concept interpolation: Moving along a direction in embedding space smoothly transitions between semantic concepts

This linearity enables algebraic manipulation of meaning without retraining the underlying model.

The geometric arrangement of embeddings preserves semantic neighborhoods. Objects with similar meaning cluster together, while dissimilar concepts are pushed apart. This is enforced during training through contrastive loss functions that explicitly minimize distance between positive pairs and maximize distance between negative pairs.

• Cosine similarity: Measures the angle between vectors, ignoring magnitude differences
• Euclidean distance: Measures straight-line distance in the embedding space
• Dot product: Used when vector magnitudes carry meaningful information (e.g., recommendation systems)

The choice of distance metric directly impacts retrieval quality and should match the loss function used during embedding model training.

Pre-trained embeddings capture universal features that transfer across downstream tasks without fine-tuning. An embedding model trained on general text can effectively represent domain-specific documents because it has learned fundamental linguistic patterns, syntactic structures, and world knowledge.

• Zero-shot classification: Compare document embeddings to label text embeddings
• Cross-lingual transfer: Multilingual embedding models map different languages into a shared semantic space
• Modality bridging: Joint embedding spaces allow text-to-image and image-to-text retrieval

This transferability is what makes vector databases powerful: the same embedding infrastructure serves search, clustering, classification, and anomaly detection simultaneously.

The dimensionality reduction inherent in embeddings introduces information loss. Not all semantic nuances survive compression into a fixed-length vector. Rare entities, fine-grained distinctions, and long-tail concepts may be poorly represented or collapsed into nearby unrelated concepts.

• The 'cabbage problem': Visually similar but semantically distinct objects may map to nearby vectors
• Context collapse: Embeddings of polysemous words average multiple meanings into a single point
• Length truncation: Long documents compressed into a single embedding lose fine-grained passage-level semantics

Mitigation strategies include late interaction models (ColBERT), multi-vector representations, and hierarchical chunking strategies that preserve granularity.

Well-trained embeddings tend toward isotropy: vectors are distributed uniformly in all directions rather than concentrating in a narrow cone. Anisotropic embeddings (where all vectors point in a similar direction) degrade the effectiveness of cosine similarity because all items appear artificially similar.

• Post-processing whitening: Applying a linear transformation to enforce isotropy
• Contrastive training: Naturally encourages vectors to spread across the hypersphere
• Normalization: L2-normalizing embeddings projects them onto the unit hypersphere, making cosine similarity equivalent to dot product

Isotropy ensures that the full representational capacity of the embedding space is utilized and that similarity scores are meaningfully distributed.