Post-Hoc Distillation

The foundational training framework where a pre-existing, frozen teacher model generates supervisory signals for a transparent student model to mimic. The teacher is typically a high-capacity neural network, while the student is an inherently interpretable model such as a decision tree or generalized additive model. The student learns solely from the teacher's input-output mappings, never accessing the original training data or ground-truth labels directly.

• Teacher remains frozen and unmodified during distillation
• Student architecture is chosen for native interpretability
• Training uses only the teacher's predictions as targets

The probability distributions produced by the teacher model's softmax output layer, typically softened using a high temperature parameter. Unlike hard labels that only indicate the predicted class, soft targets reveal the dark knowledge encoded in the relative probabilities assigned to incorrect classes. This rich signal captures inter-class similarities and the teacher's uncertainty structure, enabling the student to learn a more nuanced decision boundary.

• Contain class similarity information absent from hard labels
• Produced by applying temperature scaling to logits
• Provide richer supervision than one-hot ground truth

A composite objective function that guides student training by combining two components: the Kullback-Leibler divergence between teacher and student soft targets, and an optional cross-entropy loss against ground-truth labels. The KL divergence term ensures the student replicates the teacher's output distribution, while the ground-truth term anchors the student to actual labels when available.

• KL divergence measures distributional alignment
• Weighted combination: α * KL_loss + (1-α) * CE_loss
• Temperature must be consistent between teacher and student during loss computation

A student model whose quality is measured by fidelity—the degree to which its predictions match the teacher model's outputs on unseen data—rather than solely by accuracy against ground-truth labels. High fidelity indicates the student has successfully captured the teacher's decision logic. This metric is critical in post-hoc distillation because the goal is explaining the teacher's behavior, not necessarily achieving optimal task performance.

• Fidelity = agreement rate between teacher and student predictions
• Complements standard accuracy metrics
• Essential for validating the surrogate's explanatory power

A globally interpretable tree-based model trained on the input-output pairs of a black-box teacher to provide a faithful, high-level approximation of its overall decision logic. Algorithms like CART or C4.5 are commonly used. The resulting tree can be visualized as a flowchart, making the teacher's decision boundaries accessible to non-technical stakeholders. The tree's depth and leaf count control the interpretability-accuracy tradeoff.

• Produces human-readable if-then rules
• Depth constraints manage complexity vs. fidelity
• Suitable for global explanation of the entire model

A training method that adds a penalty term to the distillation loss to encourage the student model's decision boundaries to be simple and amenable to extraction as a compact set of logical rules. This regularization biases the student toward learning axis-aligned or sparse decision surfaces that can be expressed concisely. The approach bridges the gap between high-fidelity approximation and human-comprehensible explanation.

• Penalizes complex, non-linear decision boundaries
• Encourages sparse feature usage
• Facilitates direct rule extraction post-training