Distillation for Generalized Additive Models

The GAM student learns an equation of the form g(E[Y]) = f_1(x_1) + f_2(x_2) + ... + f_p(x_p). Each shape function f_i captures the univariate relationship between a single feature and the target, making the model transparent-by-design. Unlike a black-box teacher, there are no cross-feature interactions unless explicitly added, ensuring that the marginal effect of changing one input can be inspected in complete isolation.

The primary interpretability mechanism. After distillation, each feature's learned shape function is plotted as a curve, showing exactly how the model's prediction changes as that feature varies. For example, a shape function for 'age' might reveal a U-shaped risk curve, while 'income' might show a logarithmic saturation. These plots are the definitive audit artifact, allowing a compliance officer to verify that the model has learned sensible, monotonic, or domain-expected relationships from the teacher.

The student GAM is trained to minimize a composite loss: L = α * KL(softmax(z_T/T) || softmax(z_S/T)) + (1-α) * CrossEntropy(y, softmax(z_S)). The temperature parameter T softens the teacher's output distribution, exposing dark knowledge about inter-class similarities. The Kullback-Leibler divergence term forces the GAM's additive score to match the teacher's rich probability landscape, transferring knowledge that hard labels alone cannot convey.

Fidelity is measured by how closely the GAM student's predictions match the teacher's on unseen data. A high-fidelity distilled GAM achieves global surrogate status, meaning its additive decomposition faithfully approximates the teacher's full decision boundary. This is quantified using R-squared for regression or AUC agreement for classification. A drop in fidelity signals that important feature interactions captured by the teacher are being lost and may need to be added as explicit two-way terms in the GAM.

An Explainable Boosting Machine is a modern, high-performance GAM that serves as an ideal student. It uses bagging and gradient boosting to learn each shape function as a set of piecewise-constant bins. EBMs also include a fast, cyclical algorithm to automatically detect and rank pairwise interaction terms. When distilling a deep neural network teacher, the EBM student can optionally include the top-k interactions to boost fidelity while maintaining full transparency for every additive component.

A distilled GAM provides a complete audit trail for model decisions. For any single prediction, the final logit is decomposed as: Prediction = Intercept + f_1(x_1_value) + f_2(x_2_value) + .... Each term is a scalar lookup from its shape function. This satisfies right to explanation requirements under regulations like GDPR and the EU AI Act, as the exact numerical contribution of each feature to a denied loan application or a flagged transaction can be presented to a human reviewer without ambiguity.

The probability distributions over classes produced by a teacher model, typically smoothed by a high temperature parameter. Unlike hard labels (e.g., [0,1,0]), soft targets (e.g., [0.1,0.8,0.1]) reveal the teacher's dark knowledge about class similarities. When distilling into a GAM, these rich signals guide the shape functions to learn nuanced feature contributions that mirror the teacher's internal representations.

A hyperparameter applied to the softmax function during distillation that controls the softness of the output probability distribution. Higher temperatures produce softer distributions, exposing more of the teacher's dark knowledge. The formula is:

• q_i = exp(z_i / T) / Σ_j exp(z_j / T)
• T=1: standard softmax
• T>1: softer, reveals class similarities
• T→∞: uniform distribution

A composite objective function that combines two components:

• KL Divergence between teacher and student soft targets (soft loss)
• Cross-entropy against ground-truth labels (hard loss) The total loss is typically: L = α * L_soft + (1-α) * L_hard, where α balances the influence of the teacher's dark knowledge against the true labels. For GAM distillation, this loss guides each feature's shape function to faithfully reproduce the teacher's decision boundary.

A student model whose quality is measured by its fidelity—the degree to which its predictions match those of the teacher model on unseen data. Key metrics include:

• Fidelity: Percentage of test samples where student and teacher agree
• R² between logits: How well student logits predict teacher logits
• Shape function correlation: How closely each GAM shape function reflects the teacher's marginal feature behavior Fidelity, not just accuracy, is the primary success criterion in distillation.