Sentence-BERT (SBERT)

SBERT employs a siamese network structure where two identical BERT models process sentence pairs in parallel with tied weights. This architecture enables the model to learn sentence representations directly optimized for semantic similarity rather than next-sentence prediction. During training, the network processes pairs like (query, positive) and (query, negative), adjusting embeddings so similar sentences map closer together in vector space.

• Fixed-size embeddings from mean pooling of token outputs
• Enables efficient cosine similarity comparison
• Eliminates the need for cross-encoder inference during search

SBERT transforms BERT's token-level outputs into a single fixed-size sentence embedding through configurable pooling operations. The default strategy uses mean pooling across all token embeddings, which captures the aggregate semantic content of the entire sentence. Alternative strategies include CLS token pooling (using only the special classification token) and max pooling (taking element-wise maximums).

• Mean pooling: averages all token vectors for robust representation
• CLS pooling: uses the dedicated sentence-level token
• Max pooling: captures the most activated features per dimension

SBERT supports multiple training objectives depending on the available data and target task. Classification objective adds a softmax layer on top of concatenated embeddings (u, v, |u-v|) for NLI datasets. Regression objective computes cosine similarity between embeddings and minimizes mean squared error against labeled similarity scores. Triplet objective uses anchor, positive, and negative sentences to minimize distance to positives while maximizing distance to negatives.

• Softmax classification for entailment/contradiction tasks
• Cosine similarity regression for STS-B benchmark training
• Triplet margin loss for relative similarity ranking

Multiple pre-trained SBERT variants are available through the sentence-transformers library, each optimized for different use cases. all-MiniLM-L6-v2 is a lightweight general-purpose model mapping to 384-dimensional embeddings, while all-mpnet-base-v2 provides higher quality at 768 dimensions. Domain-specific variants like Legal-BERT fine-tuned with SBERT objectives exist for specialized legal semantic search.

• all-MiniLM-L6-v2: 384 dimensions, fast inference
• all-mpnet-base-v2: 768 dimensions, highest quality
• multi-qa-mpnet-base-dot-v1: optimized for dot-product similarity

Once sentences are encoded into fixed-size vectors, SBERT enables cosine similarity computation between any pair of embeddings. This allows building semantic search systems where a query is encoded once and compared against a pre-computed index of document embeddings. The similarity score ranges from -1 to 1, with higher values indicating greater semantic relatedness. This approach scales to millions of documents using approximate nearest neighbor (ANN) indexes like FAISS or HNSW.

• Query encoded once, compared against pre-indexed corpus
• Cosine similarity: measure of angular distance between vectors
• Compatible with ANN libraries for billion-scale retrieval

SBERT models are evaluated on the Semantic Textual Similarity (STS) benchmark, which measures how well embeddings correlate with human similarity judgments. SBERT with mean pooling achieves Spearman rank correlations exceeding 0.80 on STS tasks, dramatically outperforming averaging BERT token embeddings (which produces poor sentence representations). This benchmark validates that SBERT embeddings capture genuine semantic relatedness rather than superficial lexical overlap.

• STS-B: human-annotated sentence pair similarity scores
• Spearman correlation measures ranking agreement
• SBERT significantly outperforms raw BERT embeddings on STS tasks

The mathematical objective that makes SBERT work. Contrastive loss trains the model by minimizing the distance between embeddings of semantically similar sentence pairs (e.g., a query and a relevant clause) while maximizing the distance between dissimilar pairs. In legal contexts, this teaches the model to distinguish between a binding indemnification clause and a superficially similar limitation of liability provision, ensuring high precision in contract review.

A data curation strategy critical for legal SBERT performance. Hard negatives are documents that are lexically similar to a query but jurisprudentially irrelevant—such as a case citing the same statute but reaching an opposite holding. Training with these difficult examples sharpens the embedding space, preventing the retrieval of false positives that waste attorney time during due diligence.

A two-stage retrieval architecture where SBERT acts as the efficient first-pass bi-encoder, and a cross-encoder provides the final precision filter. The cross-encoder jointly processes the query and candidate document through full transformer attention, computing a fine-grained relevance score. This pipeline is essential for legal research platforms where SBERT retrieves 100 candidate cases, and the cross-encoder identifies the 10 that are directly on-point.

The preprocessing strategy that determines SBERT's indexing granularity. Rather than splitting legal documents by arbitrary token counts, semantic chunking respects structural boundaries like sections, articles, or paragraphs. This ensures that an SBERT embedding represents a complete legal concept—such as an entire force majeure clause—rather than a fragmented sentence fragment, preserving retrieval coherence.

An architecture that fuses SBERT's dense semantic retrieval with sparse lexical search like BM25. Legal queries often contain precise terms of art (e.g., 'Rule 12(b)(6)') that semantic search might miss, while BM25 excels at exact matching. Reciprocal Rank Fusion (RRF) merges the result lists, ensuring that a search for 'summary judgment standard' retrieves both conceptually related cases and documents containing the exact procedural phrase.

A training method that produces SBERT embeddings where truncated vector prefixes remain useful for similarity search. A 768-dimensional legal embedding can be reduced to 128 dimensions with minimal accuracy loss. This enables flexible deployment: high-precision retrieval at full dimensionality for litigation support, and low-latency search at reduced dimensionality for real-time contract review interfaces.