Vector Embedding

Embeddings map data into a high-dimensional space where each dimension represents a latent feature. Modern models like OpenAI's text-embedding-3-large produce vectors with 3,072 dimensions, while efficient models may use 384–768 dimensions. Higher dimensionality captures more nuanced semantic relationships but increases storage and computational cost.

• Dense vectors: Every dimension contains a non-zero value, encoding rich semantic information
• Sparse vectors: Mostly zero values, typical of traditional TF-IDF representations
• The trade-off between dimensionality and performance is governed by the curse of dimensionality, where excessive dimensions can degrade nearest-neighbor search accuracy

The fundamental operation in vector space is measuring cosine similarity—the cosine of the angle between two vectors. Unlike Euclidean distance, cosine similarity is insensitive to vector magnitude, focusing purely on directional alignment. A score of 1.0 indicates identical orientation (perfect semantic match), 0 indicates orthogonality (no relationship), and -1 indicates opposition.

• The query "capital of France" and the passage "Paris is the seat of French government" will exhibit high cosine similarity despite sharing few lexical terms
• This property enables cross-lingual retrieval, where semantically equivalent phrases in different languages map to proximate vectors

Embedding models fall into two categories based on how they handle linguistic context:

• Static embeddings (e.g., Word2Vec, GloVe): Each word has a single, fixed vector regardless of surrounding text. The word "bank" receives the same embedding whether referring to a financial institution or a riverbank, conflating distinct meanings
• Contextual embeddings (e.g., BERT, text-embedding-3): The vector for a token is dynamically generated based on its surrounding context. This disambiguates polysemous words and captures phrase-level semantics, making contextual models the standard for modern RAG and semantic search systems

Advanced models like CLIP (Contrastive Language-Image Pre-training) project different data modalities—text and images—into a shared embedding space. This enables cross-modal retrieval: searching a database of images using natural language queries, or vice versa.

• A text embedding for "a red sports car on a mountain road" will exhibit high cosine similarity with the image embedding of a photograph matching that description
• Multi-modal spaces are foundational for Vision-Language Models (VLMs) and Retrieval-Augmented Multi-Modal systems, where an LLM can reason over both textual documents and relevant imagery retrieved from a unified vector store

A metric that measures the cosine of the angle between two vectors in a high-dimensional space. It quantifies semantic similarity independent of vector magnitude, making it the standard distance function for comparing embeddings.

• Value ranges from -1 (opposite) to 1 (identical)
• A score of 0 indicates orthogonal, unrelated concepts
• Preferred over Euclidean distance for text embeddings because it ignores document length

The process of compressing high-dimensional embedding vectors into fewer dimensions while preserving their semantic relationships. Essential for visualization and computational efficiency.

• PCA: Linear projection preserving global variance
• t-SNE: Non-linear technique optimized for local neighborhood preservation
• UMAP: Faster alternative that better preserves global structure
• Commonly used to project 768- or 1536-dimensional embeddings into 2D or 3D for cluster analysis

A neural network trained to map discrete data—text, images, or audio—into continuous vector representations where semantic proximity equals geometric proximity. Modern embedding models are typically transformer-based encoders.

• Text: OpenAI text-embedding-3, Cohere Embed, BGE, E5
• Image: CLIP, DINOv2, SigLIP
• Multi-modal: Models that embed different data types into a shared vector space
• Output dimensionality typically ranges from 384 to 3072 dimensions

An information retrieval paradigm that matches queries to documents based on conceptual meaning rather than exact keyword overlap. It relies on embedding both the query and the document corpus into the same vector space and performing nearest-neighbor retrieval.

• Handles synonyms, paraphrases, and cross-lingual queries
• Often combined with lexical search in hybrid retrieval systems
• Foundational to RAG architectures and modern enterprise search

The strategic segmentation of long-form content into discrete, self-contained blocks optimized for embedding and vector retrieval. Chunk size and overlap directly impact retrieval precision and recall.

• Fixed-size chunking: Splits text by token count with overlap windows
• Semantic chunking: Splits at natural boundaries like paragraphs or sections
• Recursive chunking: Hierarchical splitting that respects document structure
• Chunk size typically ranges from 256 to 1024 tokens for optimal embedding performance