Knowledge Distillation for Re-Ranking

The specific process of training a faster Bi-Encoder student model to mimic the softmax score distribution or the margin between positive and negative pairs produced by a slower Cross-Encoder teacher. The student learns to approximate the teacher's full-attention scoring without performing token-level interactions at inference time. Training typically uses KL divergence loss to align the student's probability distribution with the teacher's, or Margin-MSE to replicate the teacher's ranking margins. This enables the student to achieve near-teacher precision at Bi-Encoder latency.

A loss function that measures the difference between two probability distributions. In distillation, it quantifies how much the student model's predicted relevance distribution diverges from the teacher's soft target distribution. The teacher produces a softened probability vector over candidate documents using a high temperature parameter T, which reveals dark knowledge about inter-class similarities. The student is trained to minimize KL(teacher_softmax || student_softmax), learning richer signals than hard binary labels alone.

A training paradigm where a large, computationally expensive teacher model (typically a Cross-Encoder like MonoBERT) generates high-quality relevance scores or rankings, and a compact student model (typically a Bi-Encoder) learns to replicate these outputs. The teacher processes query-document pairs with full self-attention, capturing rich token-level interactions. The student encodes queries and documents independently into dense vectors. During distillation, the student learns to produce embeddings whose dot-product similarity mirrors the teacher's fine-grained relevance judgments.

An alternative distillation objective that trains the student to replicate the margin between positive and negative document scores produced by the teacher. For each query, the teacher scores a positive and a negative document, producing a margin m_teacher = score(q, d+) - score(q, d-). The student computes its own margin m_student using dot-product similarity. The loss is the mean squared error between m_teacher and m_student. This approach directly transfers the teacher's ranking discrimination ability without requiring full probability distribution matching.

A data curation strategy critical to effective distillation. The teacher Cross-Encoder identifies hard negatives—documents that receive high similarity scores from the current student Bi-Encoder but are actually irrelevant to the query. These challenging examples are added to the distillation training set, forcing the student to learn fine-grained discriminative boundaries. Without hard negatives, the student easily separates obvious non-relevant documents and fails to capture the teacher's precision on borderline cases. Often combined with in-batch negatives for efficiency.

The phenomenon where a teacher model's softened output probabilities contain valuable information beyond the hard correct label. When a Cross-Encoder scores a set of candidate documents, the relative probabilities assigned to incorrect documents reveal which ones are semantically closer to the correct answer. This dark knowledge encodes the teacher's internal similarity structure. By training on these soft targets rather than binary relevance labels, the student Bi-Encoder learns a richer embedding space that better preserves the teacher's nuanced understanding of document relationships.