Matryoshka Representation Learning

MRL structures embeddings so that truncated prefixes of the full vector remain semantically meaningful. The first 8, 16, 32, or 64 dimensions each form a valid, lower-fidelity representation of the input. This is achieved by optimizing a sum of losses computed on multiple nested dimensions simultaneously during training.

• Full 768-dim vector: highest fidelity
• First 128 dims: ~95% of full accuracy
• First 64 dims: ~90% of full accuracy
• First 32 dims: usable for coarse filtering

A single MRL model replaces multiple fixed-dimension models. At inference time, you select the dimensionality based on latency budget, storage constraints, or accuracy requirements. This eliminates the operational overhead of maintaining separate 128-dim, 256-dim, and 768-dim embedding models.

• Coarse filtering: use 64-dim vectors for fast candidate retrieval
• Re-ranking: use full 768-dim vectors for precision scoring
• Edge deployment: store compact 32-dim vectors on device

MRL modifies the standard contrastive loss by applying it to multiple prefix lengths of the embedding vector. For a max dimension D, the loss is computed at dimensions d₁, d₂, ..., D where each d is a power of 2 or a predefined granularity level. The total loss is a weighted sum:

• L_total = Σ w_i × L(f(x)[:d_i], y)
• Higher weights typically assigned to larger dimensions
• Lower dimensions learn to capture the most salient features first
• Forces the model to front-load critical semantic information

MRL enables progressive retrieval architectures. A vector database can store only the first k dimensions for each document, performing an initial ANN search at low memory cost. Candidate documents are then re-ranked using their full vectors loaded from a separate store.

• Memory reduction: 64-dim index uses ~92% less RAM than 768-dim
• Speed improvement: smaller vectors mean faster distance computations
• No accuracy ceiling: full vectors always available for final scoring
• Compatible with FAISS, HNSW, and PQ-based indexes

Unlike post-hoc techniques like PCA or random projection, MRL learns to structure information hierarchically during training. Post-hoc reduction discards information that was distributed uniformly across dimensions. MRL explicitly concentrates the most discriminative features in the lowest dimensions.

• PCA: projects trained embeddings, loses information in discarded components
• Random projection: computationally cheap but unpredictable quality
• MRL: bakes hierarchical structure into the representation itself
• Result: truncated MRL vectors consistently outperform PCA-reduced vectors of the same size

In legal AI, MRL addresses the tension between precision and scale. Legal corpora often contain millions of documents where exact clause matching is critical, but rapid filtering is also necessary.

• First-pass retrieval: 64-dim vectors scan 10M+ legal documents in milliseconds
• Clause-level matching: 256-dim vectors identify semantically similar provisions
• Citation verification: full 768-dim vectors confirm precise authority references
• Enables single-model deployment across research, review, and compliance workflows