Embedding Normalization

After L2 normalization, every embedding vector has a magnitude of exactly 1.0, projecting all points onto the surface of a unit hypersphere. This eliminates magnitude variance as a confounding factor in similarity calculations.

• The normalization operation divides each vector component by the Euclidean norm: v̂ = v / ||v||₂
• All vectors now lie on a manifold with constant curvature
• Magnitude information is discarded, preserving only angular relationships
• This constraint is fundamental to making cosine similarity equivalent to dot product

For L2-normalized vectors, cosine similarity becomes computationally identical to dot product. Since both vectors have unit length, the denominator in cosine similarity (product of magnitudes) equals 1.0, reducing the formula to a simple inner product.

• cos(θ) = (A·B) / (||A|| × ||B||) simplifies to A·B when both norms equal 1
• This eliminates a division operation per similarity computation
• Enables fused multiply-add hardware acceleration on GPUs and TPUs
• Critical for high-throughput ANN retrieval where billions of dot products are computed per query

Normalization stabilizes training by bounding the gradient magnitudes flowing through the network. Without normalization, embeddings with large L2 norms produce disproportionately large gradients, causing unstable updates and loss spikes.

• The gradient through L2 normalization has a bounded Jacobian
• Prevents exploding gradients in deep embedding architectures
• Enables higher learning rates without divergence
• Particularly important for contrastive learning objectives like InfoNCE loss, where unnormalized embeddings can cause loss collapse

Well-trained normalized embeddings tend toward uniform distribution on the hypersphere, maximizing the representational capacity of the embedding space. This property emerges naturally from contrastive objectives that push dissimilar samples apart.

• Uniform distribution prevents dimensional collapse, where embeddings cluster in a narrow cone
• Maximizes the effective capacity of the embedding dimension
• Measured by metrics like the hyperspherical uniformity gap
• Related to the Thomson problem of distributing electrons evenly on a sphere
• Uniformity combined with tolerance to semantically similar items defines the ideal embedding geometry

Normalized embeddings enable additive angular margin losses such as ArcFace, CosFace, and SphereFace. These loss functions operate directly on the geodesic distance (angle) between vectors on the unit hypersphere, enforcing intra-class compactness and inter-class separation.

• ArcFace adds an angular margin penalty: cos(θ + m)
• Requires embeddings to lie on a hypersphere for the angular penalty to be geometrically meaningful
• Produces highly discriminative representations for user and item embeddings
• Widely adopted in face recognition and increasingly in recommendation systems for learning separable user interest clusters

After normalization, a temperature parameter τ is often applied to scale the logits before softmax. This controls the concentration of the similarity distribution without affecting vector directions, acting as an entropy regulator for the predicted probability distribution.

• Lower temperature (τ < 1) sharpens the distribution, making the model more confident and discriminative
• Higher temperature (τ > 1) flattens the distribution, encouraging exploration and softer assignments
• The normalized embedding magnitude is fixed at 1, so temperature becomes the sole sharpness control knob
• Critical in contrastive learning where temperature modulates the concentration of the von Mises-Fisher distribution on the hypersphere