Anisotropy

In anisotropic spaces, all word or sentence vectors occupy a restricted angular region of the hypersphere rather than distributing uniformly. This means the average cosine similarity between any two random vectors is significantly greater than zero, often exceeding 0.3 or 0.5. The consequence is that even semantically unrelated concepts appear artificially similar, reducing the discriminative power of cosine similarity metrics.

• Random vectors should ideally have near-zero cosine similarity
• In anisotropic BERT embeddings, average pairwise similarity can reach 0.6
• The phenomenon is most pronounced in pre-trained transformer models before fine-tuning

Anisotropy manifests mathematically as a highly skewed singular value spectrum. A small number of principal components capture a disproportionate amount of the total variance in the embedding space. These dominant directions often encode syntactic or frequency-based information rather than semantic meaning, acting as noise that overwhelms the subtler dimensions where genuine semantic distinctions reside.

• Top singular values can be orders of magnitude larger than the rest
• Removing the first few principal components often improves semantic task performance
• This skew is a direct consequence of the training objective rather than data distribution

The clustering of vectors degrades the reliability of cosine similarity as a semantic metric. When all points are crowded into a narrow cone, the distance between genuinely related concepts and unrelated ones compresses, making it difficult to establish meaningful similarity thresholds. This directly undermines retrieval-augmented generation systems, semantic search, and clustering algorithms that depend on well-separated vector representations.

• Semantic similarity scores become inflated and less discriminative
• Nearest neighbor retrieval returns false positives from unrelated concepts
• Clustering algorithms struggle to identify genuine semantic groupings

The primary mitigation for anisotropy is whitening, a linear transformation that normalizes the covariance matrix of the embedding space to the identity matrix. This operation equalizes variance across all dimensions and decorrelates features, effectively spreading vectors uniformly across the hypersphere. Common approaches include ZCA whitening, PCA whitening, and learned flow-based normalizing transformations.

• ZCA whitening preserves maximal original orientation while removing anisotropy
• Post-whitening, average pairwise cosine similarity drops to near zero
• Whitening can be applied as a post-processing step without retraining the model

Research has shown that anisotropy is strongly correlated with word frequency. High-frequency words like 'the', 'of', and 'and' dominate the principal components of the embedding space, creating a frequency bias that overshadows semantic content. This explains why removing the first few principal components—which capture frequency information—often improves performance on semantic textual similarity benchmarks.

• Frequency accounts for a large portion of variance in static embeddings
• Contextualized models like BERT exhibit anisotropy even in context-specific representations
• The bias is an artifact of the masked language modeling objective

Modern embedding models explicitly optimize for isotropy—the uniform distribution of vectors—as a quality objective. Techniques include contrastive learning with hard negative mining, regularization terms that penalize dominant singular values, and architectural innovations like layer normalization placement. Models trained with these objectives, such as SimCSE and E5, exhibit significantly more isotropic spaces and superior retrieval performance.

• Contrastive objectives naturally encourage uniform distribution on the hypersphere
• The InfoNCE loss implicitly optimizes for isotropy
• Evaluation benchmarks like MTEB reward isotropic embedding spaces