Sentence-BERT (SBERT)

The foundational architecture of SBERT where two identical BERT encoders with tied weights process sentence pairs simultaneously. During training, the network learns to map semantically similar sentences close together in the embedding space. Unlike standard BERT's cross-encoder approach requiring both sentences as input, the siamese setup enables pre-computation of embeddings for efficient retrieval.

• Weight-tied twin networks process inputs independently
• Enables offline indexing of millions of sentences
• Training uses contrastive loss or triplet loss objectives

SBERT converts variable-length token sequences into fixed-size sentence embeddings using pooling operations on the transformer output. The default MEAN pooling averages all token vectors, capturing overall semantic content. Alternative strategies include:

• CLS token pooling: Uses only the special classification token's representation
• MAX pooling: Takes element-wise maximum over time dimension
• Attention-weighted pooling: Learns importance weights for each token

MEAN pooling consistently outperforms CLS token extraction for semantic similarity tasks.

SBERT employs specialized loss functions to learn semantic similarity. Classification objective concatenates embeddings u and v with their element-wise difference |u-v|, then feeds through a softmax classifier for entailment tasks. Regression objective computes cosine similarity between embeddings and minimizes mean squared error against human similarity scores.

• Triplet loss uses anchor, positive, and negative examples
• Multiple Negatives Ranking loss leverages in-batch negatives
• Trained on SNLI and Multi-Genre NLI datasets for broad semantic understanding

SBERT's key innovation is decoupling encoding from comparison. Once sentences are embedded, similarity computation reduces to cosine similarity between fixed vectors—an O(1) operation. This enables:

• Real-time semantic search over millions of documents
• Clustering with standard algorithms like k-means on embeddings
• Semantic deduplication by thresholding cosine similarity

A single V100 GPU can encode approximately 3,000 sentences per second with a base SBERT model.

SBERT is a bi-encoder: sentences are encoded independently, enabling pre-computation but sacrificing some accuracy. A cross-encoder processes sentence pairs jointly through full self-attention, achieving higher accuracy at the cost of combinatorial complexity.

• Bi-encoder: O(n) encoding, O(1) comparison per pair
• Cross-encoder: O(n²) full attention passes for n sentences
• Production systems often use bi-encoder retrieval followed by cross-encoder re-ranking
• SBERT provides both architectures in its framework

Pre-trained SBERT models can be fine-tuned on domain-specific data to capture specialized semantics. The TSDAE (Transformer-based Sequential Denoising Auto-Encoder) method enables unsupervised adaptation using only unlabeled target domain text.

• Domain-specific NLI datasets improve legal or medical search
• Knowledge distillation transfers cross-encoder quality to bi-encoder speed
• Multilingual SBERT variants support 50+ languages via teacher-student training
• Fine-tuning requires as few as 1,000 labeled pairs for significant improvement