Rule-Regularized Distillation

The training objective combines two competing forces: a fidelity term (e.g., Kullback-Leibler divergence) ensuring the student mimics the teacher's soft targets, and a rule regularization term that penalizes decision boundary complexity. This penalty is often implemented as a differentiable approximation of the number of logical rules or the depth of a decision tree required to describe the student's learned function. The hyperparameter lambda controls the trade-off between prediction accuracy and interpretability.

Unlike standard distillation which only cares about output matching, rule-regularized distillation actively shapes the student's internal geometry. The regularization term encourages the student to learn axis-aligned decision boundaries rather than complex, non-linear manifolds. This is achieved by penalizing weight matrices that would create intricate feature interactions, effectively pushing the student toward a functional form that can be trivially decomposed into a compact set of if-then conditions.

The core value proposition is that the resulting student model is amenable to rule extraction by construction. Standard surrogates often suffer from low fidelity when their complex internals are approximated by rules. Here, the regularization ensures the student's logic is already simple. Post-training, algorithms like CART or SIRUS can extract a high-fidelity, sparse rule set directly from the student's weights or by querying its decision surface, with minimal approximation error.

The regularization term frequently incorporates an L1 (Lasso) penalty on the student's weights or on the parameters of an attention mechanism. This drives many weights to exactly zero, performing automatic feature selection. By forcing the student to rely on a minimal subset of input features, the resulting model not only becomes computationally efficient but also yields rules that reference only a handful of critical variables, making the explanations concise and actionable for human auditors.

The distillation process often employs a dynamic temperature scaling schedule. Training begins with a high temperature in the softmax, allowing the student to absorb the rich 'dark knowledge' of inter-class similarities from the teacher. As training progresses and the rule regularization term gains influence, the temperature is gradually reduced. This annealing process shifts the focus from soft probability matching to crisp, high-confidence decision boundaries that are easier to capture as discrete logical rules.

By varying the regularization strength lambda, practitioners can trace out a Pareto frontier that explicitly quantifies the trade-off between fidelity to the teacher and rule-set complexity. This provides a rigorous, quantitative framework for selecting a model: a compliance officer might choose a high-interpretability point with 5 rules and 92% fidelity, while a less critical application might accept 20 rules for 98% fidelity. This moves the selection process from subjective judgment to objective optimization.

The foundational model compression technique where a smaller student model is trained to replicate the behavior of a larger teacher model. The student learns from soft targets—probability distributions smoothed by a temperature parameter—which encode richer inter-class similarity information than hard labels alone. This transfer of dark knowledge enables compact models to approach the performance of their computationally expensive teachers.

The process of deriving a set of symbolic, human-readable if-then rules from a trained neural network. By systematically querying the model and analyzing its decision boundaries, rule extraction algorithms produce logical expressions that approximate the network's behavior. When combined with distillation, the extracted rules serve as a transparent-by-design student, offering global interpretability without sacrificing fidelity to the original model.

A composite objective function that guides student training. In standard distillation, it combines:

• Kullback-Leibler divergence between teacher and student soft targets
• Standard cross-entropy loss against ground-truth labels Rule-regularized distillation extends this by adding a regularization penalty that encourages the student's decision boundaries to be simple, axis-aligned, or otherwise amenable to logical rule extraction.

A globally interpretable tree-based model trained on the input-output pairs of a black-box teacher. Algorithms like CART or C4.5 produce a flowchart-like structure where each node represents a feature test and each leaf a prediction. When used as a distillation student, the tree's inherent axis-aligned splits naturally translate into if-then rules, making it an ideal architecture for rule-regularized approaches.

A glass-box model that combines additive feature functions with automatic interaction detection. EBMs are inherently interpretable—each feature's contribution can be visualized as a shape function—while achieving performance competitive with black-box models. As a distillation student, an EBM provides high-fidelity approximation of the teacher while maintaining the transparency required for rule extraction and regulatory compliance.

A student model whose quality is measured by fidelity—the degree to which its predictions match the teacher on unseen data—rather than accuracy against ground-truth labels alone. High fidelity ensures the extracted rules faithfully represent the teacher's decision logic. Metrics include:

• Fidelity score: percentage of matching predictions
• Rule coverage: proportion of input space covered by extracted rules
• Rule complexity: number and length of extracted logical conditions