Distillation for Decision Trees

The decision tree is trained as a global surrogate, meaning it approximates the teacher model's entire decision boundary across the full input space. Unlike local explanation methods that explain a single prediction, this provides a complete, high-level map of the model's logic.

• Training Data: The student is trained on a synthetic dataset generated by querying the teacher model with unlabeled or randomly sampled inputs.
• Fidelity Metric: Quality is measured by fidelity—the percentage of test inputs where the tree's prediction matches the teacher's—not just accuracy against ground truth.
• Trade-off: Deeper trees achieve higher fidelity but sacrifice interpretability, requiring careful depth constraints.

Instead of training on hard class labels, the decision tree student learns from the teacher's soft targets—the full probability distribution over all classes. This transfers the 'dark knowledge' of inter-class similarities.

• Rich Supervision: A soft target reveals that an image of a '3' is more similar to an '8' than a '1', providing more information per example than a one-hot label.
• Temperature Scaling: A high temperature parameter is applied to the teacher's softmax to soften the distribution, exposing subtle relationships that would otherwise be crushed near zero.
• Loss Function: The tree is optimized using Kullback-Leibler divergence between its output distribution and the teacher's softened distribution.

The trained decision tree is directly translated into a set of human-readable if-then rules. Each path from the root node to a leaf node represents a conjunctive rule that defines a specific region of the input space.

• Example Rule: IF age > 35 AND income < $50k AND credit_score <= 620 THEN risk = 'High'
• Rule Pruning: Redundant or overly specific rules can be pruned to improve readability without significant fidelity loss.
• Auditability: These explicit rules allow compliance officers and domain experts to inspect, validate, and challenge the logic without any machine learning expertise.

Standard decision tree algorithms like CART (Classification and Regression Trees) or C4.5 are repurposed as student models. The distillation process is algorithm-agnostic to the teacher architecture.

• CART: Uses Gini impurity to recursively partition the feature space, producing binary trees well-suited for distillation.
• C4.5: Employs information gain ratio for splitting and can handle both continuous and categorical features natively.
• SIRUS Algorithm: A specialized distillation method that extracts a stable rule list from a random forest of shallow trees, providing robustness against small data perturbations.

Decision tree distillation explicitly navigates the fundamental tension between predictive performance and transparency. The depth and complexity of the tree directly control this balance.

• Shallow Trees (Depth 3-5): Highly interpretable with limited fidelity, suitable for high-level explanation and regulatory compliance.
• Moderate Trees (Depth 6-10): Balance fidelity and readability, often used for debugging model behavior.
• Deep Trees (Depth 10+): Approach the teacher's accuracy but become unwieldy, defeating the purpose of distillation.
• Ensemble Distillation: Multiple trees can be distilled into a single, shallower tree to capture more complex boundaries while maintaining a single interpretable structure.

Beyond rule extraction, the distilled tree provides a natural mechanism for global feature importance. The reduction in impurity at each split point quantifies how critical each feature is to the model's decision process.

• Gini Importance: Calculated as the total decrease in node impurity weighted by the probability of reaching that node, averaged over all trees in an ensemble.
• Split Count: The number of times a feature is used for splitting across the tree provides an intuitive importance ranking.
• SHAP Integration: The tree structure allows for fast, exact Shapley Additive Explanations computation, combining global surrogate logic with game-theoretic local attribution.