Distillation for Logistic Regression

The logistic regression student serves as a global surrogate model, approximating the teacher's entire decision boundary with a single linear equation. Unlike local explanation methods that explain one prediction at a time, this approach provides a complete, transparent model that can be inspected, validated, and deployed independently.

• Each feature receives a single coefficient representing its global importance
• The decision boundary is a simple hyperplane in the feature space
• Perfect for regulated industries requiring fully auditable models
• Can be expressed as a simple scorecard or formula

Instead of training on hard binary labels, the logistic regression student learns from the teacher's soft targets—probability distributions over classes. These softened outputs, controlled by a temperature parameter in the teacher's softmax, encode rich information about class similarities and decision boundary confidence.

• Soft targets reveal which classes the teacher considers plausible alternatives
• A high temperature smooths the distribution, exposing more dark knowledge
• The student learns not just what the teacher predicts, but how confidently
• Results in better-calibrated probability estimates than direct training on labels

The training objective combines two components: the Kullback-Leibler (KL) divergence between teacher and student soft targets, and the standard cross-entropy loss against ground-truth labels. A weighting parameter α balances these two signals.

• KL divergence term: Aligns student probabilities with teacher soft targets
• Cross-entropy term: Ensures the student remains grounded in actual labels
• Typical α values range from 0.1 to 0.5, favoring the distillation signal
• The combined loss produces students that often outperform those trained on labels alone

A key advantage of distillation to logistic regression is the production of well-calibrated probability estimates. The soft targets from the teacher encode uncertainty information that helps the linear student avoid overconfident predictions on ambiguous inputs.

• Student probabilities better reflect true empirical likelihoods
• Reduces the need for post-hoc calibration like Platt scaling
• Critical for risk-sensitive applications in finance and healthcare
• Maintains interpretability while improving reliability of confidence scores

Once trained, the logistic regression coefficients provide direct, quantitative feature importance scores. Each coefficient represents the change in log-odds for a unit increase in the corresponding feature, making the model's reasoning completely transparent.

• Positive coefficients indicate features that increase the predicted probability
• Negative coefficients indicate features that decrease it
• Coefficient magnitudes reflect relative importance after standardization
• Can be directly presented to regulators or used in adverse action notices

The student model is evaluated primarily on fidelity—how closely its predictions match the teacher's—rather than accuracy against ground truth alone. A high-fidelity logistic regression student successfully captures the teacher's learned decision logic in a transparent form.

• Fidelity measured by agreement rate or KL divergence on held-out data
• High fidelity indicates the teacher's behavior is largely linear in the feature space
• Low fidelity suggests the teacher relies on non-linear interactions the student cannot capture
• Guides decisions about whether a linear explanation is sufficient for the use case

The probability distributions over classes produced by a teacher model, typically smoothed by a high temperature parameter. Unlike hard labels (0 or 1), soft targets reveal the teacher's 'dark knowledge' about inter-class similarities. For a logistic regression student, these targets provide a richer, more nuanced training signal than ground-truth labels alone, enabling the linear model to learn a decision boundary that better approximates the complex teacher's generalization.

A hyperparameter T applied to the softmax function that controls the softness of the output probability distribution. As T increases, the distribution flattens, revealing the dark knowledge of the teacher's internal representations. For logistic regression distillation, a well-tuned temperature is critical: it prevents the student from simply memorizing the teacher's overconfident predictions and instead exposes the relative similarity structure between classes that a linear model can capture.

A composite objective function that combines the Kullback-Leibler (KL) divergence between teacher and student soft targets with the standard cross-entropy loss against ground-truth labels. For a logistic regression student, this dual loss ensures the model remains faithful to the teacher's knowledge while still being grounded in reality. The mixing ratio between these two losses is a key hyperparameter that balances fidelity to the teacher against accuracy on the original task.

A statistical measure of how one probability distribution diverges from a second, reference distribution. In distillation, it quantifies the difference between the teacher's soft targets and the student's predicted probabilities. Minimizing the KL divergence forces the logistic regression student to replicate the teacher's output distribution, effectively transferring the complex model's knowledge into a globally interpretable linear form.

A simple linear model, such as LASSO or logistic regression, trained to mimic a complex model's predictions locally or globally. As a student in distillation, it provides global feature-level importance scores through its learned coefficients. Each weight directly quantifies the contribution of a feature to the prediction, offering a transparent, auditable explanation of the teacher's decision logic that satisfies regulatory and compliance requirements.

A student model whose quality is measured by its fidelity—the degree to which its predictions match those of the teacher model on unseen data, rather than solely by accuracy on ground-truth labels. For a distilled logistic regression, high fidelity means the linear model faithfully reproduces the teacher's decision boundary. This metric is crucial for ensuring the interpretable surrogate is a trustworthy proxy for the original black-box model.