Embedding Space

The number of axes in the vector space, typically ranging from 768 (BERT-base) to 4096 or higher. Higher dimensionality allows for richer semantic encoding but increases memory footprint and computational cost.

• BERT embeddings: 768 or 1024 dimensions
• OpenAI ada-002: 1536 dimensions
• E5-large-v2: 1024 dimensions
• Dimensionality is a hyperparameter balancing expressiveness against the curse of dimensionality, where distances between points become less meaningful in extremely high dimensions.

A property where embeddings are uniformly distributed in all directions without dominant axes of variation. Anisotropic spaces—common in raw transformer outputs—suffer from a degenerate cosine similarity problem where all vectors cluster in a narrow cone.

• Post-processing techniques like whitening or flow-based normalization correct anisotropy
• Isotropic spaces improve semantic differentiation and retrieval recall
• Models like BGE and E5 are explicitly trained to encourage isotropic representations

The property that small perturbations in the input text produce proportionally small displacements in the embedding space. This ensures that paraphrases and semantically equivalent expressions map to nearby points.

• Enables fuzzy matching without exact keyword overlap
• Measured by the Lipschitz constant of the encoder function
• Contrastive training explicitly enforces locality by pulling positive pairs together
• Violations of locality indicate representation collapse or poor generalization

The emergent property that vector arithmetic corresponds to semantic operations. The classic example: king - man + woman ≈ queen. This linearity enables analogical reasoning and zero-shot transfer.

• Encoded as consistent vector offsets between concept pairs
• Supports operations like attribute manipulation and concept interpolation
• Strongest in well-trained word-level and sentence-level embedding models
• Degrades when the space is overly compressed or anisotropic

The distance function defining 'closeness' in the space. While cosine similarity is standard for dense retrieval, the choice of metric fundamentally shapes the geometry of neighborhoods.

• Cosine similarity: Measures angular distance, insensitive to vector magnitude
• Euclidean distance (L2): Sensitive to both direction and magnitude; requires normalized vectors for semantic tasks
• Inner product (dot product): Used in MIPS for maximum efficiency in production retrieval
• Metric choice must be consistent between training and indexing phases

The observation that high-dimensional embeddings lie on a lower-dimensional manifold embedded within the full space. Real semantic variation occupies far fewer effective dimensions than the nominal dimensionality.

• Explains why dimensionality reduction techniques like PCA and t-SNE produce meaningful visualizations
• Intrinsic dimension estimates guide compression strategies like Product Quantization
• Training objectives that flatten the manifold improve interpolation between known points
• Violations create holes in the space where queries return poor results

The fundamental metric for navigating the embedding space. It measures the cosine of the angle between two vectors, ignoring their magnitude. In high-dimensional spaces, this effectively quantifies semantic similarity—a score of 1 means identical direction, 0 means orthogonal (unrelated), and -1 means opposite. It is the default scoring function for most dense retrieval systems.

Exhaustively comparing a query vector against billions of points is computationally prohibitive. ANN algorithms trade a small, controlled amount of accuracy for massive speedups. They build vector indexes that partition or graph the space, enabling sub-linear search times. Key algorithms include:

• HNSW: Multi-layered navigable graphs.
• IVF: Clustering-based inverted files.
• PQ: Compression via product quantization.

The training paradigm that shapes the embedding space. Models are trained to pull positive pairs (a query and its relevant passage) close together while pushing negative pairs (irrelevant passages) far apart. The contrastive loss function defines the topology of the space. Techniques like in-batch negatives and hard negative mining are critical for preventing the space from collapsing and ensuring it captures fine-grained semantic distinctions.

The standard neural architecture for mapping text into the embedding space. A query encoder and a passage encoder independently process inputs into dense vectors. This asymmetric search setup allows passage embeddings to be pre-computed and indexed offline. At query time, only the query vector is generated, enabling extremely fast Maximum Inner Product Search (MIPS) against the static index.

A paradigm that challenges the single-vector-per-passage assumption. Instead of pooling all token vectors into one point, multi-vector encoding stores every token's embedding. At query time, a MaxSim operation computes relevance via fine-grained token-level interactions. This retains more expressiveness than a bi-encoder while being far faster than a full cross-encoder, representing a distinct region in the cost-accuracy space.

Raw 768-dimensional float32 vectors consume massive memory. Product Quantization (PQ) decomposes vectors into sub-vectors and clusters them independently, while Scalar Quantization (SQ) maps floats to 8-bit integers. These techniques shrink the embedding space's memory footprint by 4x-16x, enabling billion-scale indices to reside in RAM. The trade-off is a slight reduction in recall precision.