Model Distillation

Instead of training on hard binary labels (fraud/not fraud), the student model learns from the teacher's output probabilities (e.g., 0.87 fraud, 0.13 not fraud). These soft labels encode the teacher's uncertainty and the subtle relationships it discovered between classes. For fraud detection, this means the student learns that a transaction with a 0.6 fraud score is fundamentally different from one with a 0.99 score, preserving the granular risk assessment that a simple binary label would destroy.

A temperature parameter (T) is applied to the teacher's softmax output before distillation. Higher temperatures (T > 1) soften the probability distribution, revealing the dark knowledge of the teacher—the relative similarities between incorrect classes.

• T=1: Standard softmax, one class dominates
• T=5-20: Softened distribution exposes class relationships
• Effect: The student learns that a fraudulent transaction resembles money laundering more than a false positive, even if neither is the top prediction

The student is optimized using a composite loss that combines two objectives:

• Distillation Loss (soft loss): Kullback-Leibler divergence between the softened teacher and student outputs, weighted by T²
• Student Loss (hard loss): Standard cross-entropy against the ground truth labels

This dual objective ensures the student both mimics the teacher's nuanced reasoning and maintains accuracy on the original task. The balance between these losses is controlled by a hyperparameter α (alpha).

The student model is typically a fraction of the teacher's size, enabling deployment in resource-constrained environments:

• Teacher: Deep ensemble of 10 XGBoost models or a large transformer
• Student: A single, shallow decision tree or a pruned neural network
• Compression ratios: 10x to 1000x reduction in parameters is common

For real-time fraud scoring, a distilled student can execute inference in sub-millisecond latency on CPU, compared to the teacher's GPU-dependent runtime.

A critical advantage for regulated financial environments: distillation can transfer knowledge into an intrinsically interpretable student architecture. A complex, black-box teacher (e.g., a deep neural network) can be distilled into a Explainable Boosting Machine (EBM) or a shallow decision tree. This provides:

• Auditable logic: Every decision path is traceable
• Regulatory compliance: Satisfies model governance requirements
• Reason code generation: Direct extraction of adverse action reasons for flagged transactions

Multiple diverse teacher models can be distilled simultaneously into a single student. The student learns from the aggregated output distribution of the entire ensemble, capturing the collective wisdom and reducing the variance of any single model. This is particularly powerful for fraud detection where ensembles of graph neural networks, gradient boosting machines, and sequence models each capture different fraud patterns. The distilled student inherits this multi-perspective understanding in a single, deployable artifact.

An interpretable model, such as a linear regression or a shallow decision tree, trained to approximate the predictions of a complex, black-box teacher model. In fraud detection, a surrogate model provides global insight into the teacher's behavior, allowing compliance officers to understand the general decision boundaries without needing to inspect the opaque teacher's internal weights. The student model in distillation is a form of surrogate model.

A property of models that are inherently simple and understandable due to their structure. Distillation often targets an intrinsically interpretable student, such as a Explainable Boosting Machine (EBM) or a single decision tree. This allows the final deployed fraud model to provide direct, auditable reason codes for each blocked transaction, satisfying regulatory requirements without post-hoc analysis.

The application of interpretation methods like SHAP or LIME to a trained, complex model after it has been built. Distillation can be seen as a way to bypass the need for post-hoc explainability on a black-box teacher by transferring its knowledge into a student that is either intrinsically interpretable or more amenable to post-hoc analysis, reducing computational overhead for real-time explanations.

A set of concise, human-readable statements that provide the primary reasons for a model's adverse action, such as flagging a transaction as fraudulent. A distilled, interpretable student model can generate adverse action reason codes directly from its structure. For example, a decision tree student can output the exact rule path that led to a fraud alert, providing a clear audit trail.

The institutional frameworks required to govern the lifecycle of financial fraud detection models. Distillation directly supports model governance by creating a simpler, more auditable model from a complex one. This simplifies validation, ensures compliance with regulations like SR 11-7, and reduces the model risk associated with deploying an uninterpretable deep learning system.

Defending fraud detection models against adversarial attacks. Distillation was originally popularized as a defensive technique against adversarial examples. By training a student on the teacher's soft probability outputs, the student learns smoother decision boundaries, making it harder for fraudsters to craft transactions that cause a sudden, sharp change in the model's classification.