Teacher-Student Architecture

The architecture is defined by a deliberate imbalance in model complexity. The teacher is a high-capacity model—often a deep ensemble or a massive neural network—with millions or billions of parameters, optimized solely for predictive performance. The student is a compact, low-latency model, frequently chosen for its inherent structural interpretability, such as a decision tree, a Generalized Additive Model (GAM), or a shallow neural network. This asymmetry is not a bug but the core feature, enabling the student to act as a high-fidelity, transparent proxy for the teacher's complex decision logic.

Instead of training solely on hard, one-hot encoded ground-truth labels, the student learns from the teacher's output probability distribution, known as soft targets. These soft targets are generated by applying a high temperature scaling parameter to the teacher's softmax layer, which smooths the probability distribution and exposes the dark knowledge—the relative probabilities assigned to incorrect classes. This reveals inter-class similarities and the teacher's generalization structure, providing a much richer, higher-entropy supervisory signal than hard labels alone.

The student is optimized using a composite loss function that balances two objectives:

• KL-Divergence Loss: Minimizes the Kullback-Leibler divergence between the softened probability distributions of the teacher and student, ensuring the student mimics the teacher's internal representation of class relationships.
• Student Loss: A standard cross-entropy loss against the true ground-truth labels, which anchors the student to reality and prevents it from merely replicating the teacher's errors. A hyperparameter, typically denoted α, weights the contribution of each loss term.

The success of a teacher-student architecture is measured not just by accuracy on a held-out test set, but by fidelity—the degree to which the student's predictions match the teacher's predictions on unseen data. A high-fidelity student is a faithful surrogate. This metric is critical for interpretability use cases: a student model that achieves high accuracy but low fidelity has learned a different decision boundary than the teacher, defeating its purpose as an explanatory proxy. Fidelity is often reported as the percentage of test instances where the student and teacher agree.

The standard teacher-student framework operates in a post-hoc and offline manner. The teacher model is fully trained, frozen, and treated as a fixed oracle before the student's training begins. The student never updates the teacher. This decoupled, two-phase workflow—first train the teacher for performance, then distill the student for interpretability—is a key architectural constraint. It simplifies deployment but means the student can only approximate the teacher's learned function and cannot correct fundamental flaws or biases already baked into the teacher's weights.

The framework is fundamentally agnostic to the specific architecture of the student model, which is its greatest strength for interpretability. The student can be any differentiable or non-differentiable model that can be trained on input-output pairs. Common interpretable students include:

• Decision Tree Surrogates: For rule extraction and flowchart-like logic.
• Explainable Boosting Machines (EBMs): For glass-box, additive feature functions.
• Linear Proxy Models: For global feature importance scores.
• Rule Lists: For sparse, stable if-then explanations. This allows engineers to select the exact form of interpretability required by a specific use case.

The overarching compression technique where a compact student model is trained to replicate the behavior of a high-capacity teacher model. The student minimizes a composite loss function that combines the standard cross-entropy with ground-truth labels and the Kullback-Leibler divergence against the teacher's softened probability outputs. This process transfers generalization ability without transferring the computational cost.

The probability distributions over classes produced by the teacher model, smoothed by a high temperature parameter. Unlike hard labels (e.g., [0, 1, 0]), soft targets reveal the dark knowledge of inter-class similarities. For example, a teacher classifying a car might assign a probability of 0.001 to 'truck' and 0.0001 to 'apple', teaching the student that a truck is a more similar concept than an apple.

A hyperparameter T applied to the softmax function: softmax(z_i / T). When T=1, the output is the standard softmax. As T increases, the probability distribution softens, amplifying the relative importance of incorrect classes. This reveals the teacher's dark knowledge about the data's similarity structure. During inference, the student's temperature is reset to 1 for crisp predictions.

A composite objective function that guides student training:

• Hard Loss: Standard cross-entropy between student logits and ground-truth labels.
• Soft Loss: Kullback-Leibler divergence between the student's softened outputs and the teacher's soft targets. The total loss is a weighted sum: L = α * L_hard + (1-α) * L_soft, where α balances mimicking the teacher against fitting the original data.

The implicit information encoded in the relative probabilities of incorrect classes within a teacher's softmax output. This knowledge captures the generalization and similarity structure learned from the data. For instance, a teacher trained on MNIST might assign a higher probability to '3' than to '7' for an image of '8', encoding the visual similarity between '8' and '3'. Distillation transfers this rich structure to the student.

A statistical measure of how one probability distribution Q (the student) diverges from a reference distribution P (the teacher). Defined as D_KL(P || Q) = Σ P(i) log(P(i)/Q(i)). In distillation, minimizing the KL divergence between the teacher's and student's soft targets forces the student to match the teacher's entire output distribution, not just the predicted class. It is asymmetric and non-negative, reaching zero only when the distributions are identical.