Knowledge Distillation

The core framework involves a teacher model (typically a large cross-encoder or ensemble) and a student model (a lightweight bi-encoder). The student is trained not on hard labels, but on the soft labels produced by the teacher. These soft labels—probability distributions over classes—contain rich information about inter-class similarities that one-hot labels lack.

• Teacher: High-capacity model frozen during distillation
• Student: Compact model trained to mimic teacher outputs
• Soft Targets: Probability vectors capturing dark knowledge

A critical application in modern search systems. A cross-encoder processes the query and document concatenated together with full self-attention, yielding high accuracy but prohibitive latency. The distilled bi-encoder encodes query and document independently, enabling pre-computed passage embeddings and fast Maximum Inner Product Search (MIPS).

• Teacher: Cross-encoder with full query-document attention
• Student: Bi-encoder for independent encoding
• Result: Cross-encoder precision at bi-encoder speed

A temperature parameter (T) controls the softness of the teacher's output probabilities. Higher temperatures produce softer distributions, revealing more of the dark knowledge—the relative similarities between incorrect classes. The student is trained with the same high temperature, but evaluated at T=1.

• T > 1: Softens probability distribution, exposes class relationships
• T = 1: Standard softmax for inference
• Loss Function: Typically Kullback-Leibler divergence between softened distributions

Distillation can use the original labeled dataset, a large unlabeled corpus, or synthetically generated queries. Using unlabeled data allows the student to learn from the teacher's predictions on a much larger scale than the original training set, improving generalization.

• Labeled Data: Original supervised training set
• Unlabeled Corpus: Teacher generates pseudo-labels for massive datasets
• Synthetic Queries: Augment training with generated query-passage pairs

To improve the student's discriminative power, training often incorporates hard negatives—passages that are top-ranked by a basic retriever but are actually irrelevant. The teacher model scores these negatives, and the student learns to push them away from the query in the embedding space, sharpening the decision boundary.

• Hard Negatives: Superficially similar but irrelevant passages
• Source: Retrieved from BM25 or a preliminary dense retriever
• Impact: Significantly improves Recall@K metrics

The student is optimized using a combination of losses. The distillation loss (KL divergence) aligns the student's softened outputs with the teacher's. An optional student loss (cross-entropy with hard labels) can be added to ground the model in true labels. A weighted sum balances imitation and ground truth.

• L_KD: KL divergence between teacher and student soft labels
• L_CE: Cross-entropy loss against ground truth labels
• L_total = α * L_KD + (1-α) * L_CE

A critical data curation strategy for dense retrieval distillation where training examples that are superficially similar to the query but irrelevant are deliberately selected. These hard negatives force the student model to learn fine-grained distinctions. In a distillation pipeline, the teacher cross-encoder identifies passages with high BM25 lexical overlap but low semantic relevance.

• Source: Retrieved from sparse methods like BM25 or top-K ANN results
• Teacher filtering: Cross-encoder scores identify false positives
• Impact: Dramatically improves student model discriminative power
• Mining strategies: Static pre-mining vs. dynamic in-batch hard negative selection

Two distinct distillation objectives for training student retrievers. Score distillation uses the teacher's absolute relevance scores as regression targets, training the student to predict exact scores via MSE loss. Rank distillation focuses on preserving the relative ordering of passages, using listwise losses like ListNet or LambdaRank to optimize ranking metrics directly.

• Score distillation: Pointwise approach, MSE or KL divergence loss
• Rank distillation: Listwise approach, optimizes NDCG or MRR
• Trade-off: Score distillation is simpler; rank distillation better aligns with retrieval metrics
• Hybrid: Combining both losses for optimal performance

A training efficiency technique where other positive passages within a mini-batch serve as negative examples for each query, eliminating the need for explicit negative sampling. Combined with knowledge distillation, the teacher cross-encoder scores all in-batch pairs, providing dense supervision signals. This scales training to large batch sizes on multiple GPUs.

• Mechanism: For query Q_i, passages P_j (j ≠ i) become negatives
• Teacher role: Scores all query-passage pairs in the batch
• Gradient caching: Stores teacher scores to avoid recomputation
• Memory optimization: Requires large batch sizes (1024-4096) for effective training

A specialized loss function for retrieval distillation that enforces a minimum margin between the scores of relevant and irrelevant passages. The student is penalized when the score gap falls below a threshold τ. This produces well-separated embeddings optimized for Maximum Inner Product Search (MIPS).

• Margin parameter (τ): Minimum required score separation
• Loss: max(0, τ - score_pos + score_neg)
• Advantage: Prevents embedding collapse and improves index quality
• Variants: Adaptive margins based on passage difficulty or teacher confidence