Knowledge Distillation

The core framework involves a two-model setup. A large, pre-trained teacher model generates predictions on a dataset. A smaller student model is then trained not just on the ground-truth labels, but to mimic the teacher's output distribution.

• Hard Targets: The original one-hot encoded labels.
• Soft Targets: The teacher's probability distribution over all classes, which captures dark knowledge about inter-class similarities.
• The student minimizes a combined loss of both hard and soft targets.

A critical hyperparameter, temperature (T), is applied to the softmax function to control the softness of the teacher's output probabilities.

• A higher T produces a softer probability distribution, revealing more information about which classes the teacher finds similar to the correct one.
• The same temperature is used when training the student on soft targets, but T=1 is used for hard targets.
• This process extracts the 'dark knowledge' hidden in the teacher's confidence ratios.

The most common form of knowledge transfer, where the student learns solely from the final output layer of the teacher.

• The student is trained to minimize the Kullback-Leibler (KL) divergence between its softened output and the teacher's softened output.
• This is highly effective for tasks like medical text classification, where a smaller BioBERT student can mimic a massive clinical LLM teacher for ICD-10-CM coding on a CPU.

Instead of only matching final outputs, the student learns to replicate the intermediate feature representations of the teacher's hidden layers.

• A distillation loss penalizes the difference between the student's hidden states and the teacher's hidden states, often using a learned linear projection to align dimensions.
• This is crucial for deep semantic tasks like medical NER, where mimicking the teacher's internal entity boundary representations leads to a more robust student model.

This technique transfers the structural relationships learned by the teacher among data samples or feature maps.

• The student learns to preserve the pairwise similarity between outputs in a batch, ensuring that two radiology reports deemed similar by the teacher are also similar for the student.
• This captures higher-order structural knowledge, improving the student's ability to generalize the semantic topology of a clinical corpus.

The primary driver for distillation in healthcare is deploying models on edge devices with limited compute, memory, and power.

• A massive teacher model (e.g., a 70B-parameter clinical LLM) can distill its diagnostic reasoning into a student model small enough to run on a hospital tablet.
• This enables low-latency, private inference for applications like bedside clinical decision support without transmitting PHI to the cloud.

The foundational framework where a large, high-capacity teacher model generates soft labels—probability distributions over classes—that encode richer information than hard labels. The compact student model is trained to mimic these soft targets, capturing the teacher's generalization patterns. In clinical NLP, a massive model like GatorTron can serve as the teacher, transferring its medical knowledge to a student model deployable on hospital edge servers.

A temperature parameter (T) in the final softmax layer controls the softness of the teacher's output probabilities. Higher temperatures produce softer distributions that reveal inter-class similarities learned by the teacher. For example, a teacher might indicate that a chest X-ray finding is 70% likely to be atelectasis and 25% likely to be consolidation, teaching the student about diagnostic ambiguity that hard labels would miss.

The student model is optimized using a composite loss combining:

• Distillation loss: Kullback-Leibler divergence between the softened teacher and student outputs
• Student loss: Standard cross-entropy with ground-truth hard labels This dual objective ensures the student learns both the teacher's nuanced knowledge and the correct clinical classifications. The weighting between these losses is a critical hyperparameter.

A combined optimization approach where knowledge distillation is performed simultaneously with quantization-aware training (QAT). The student model learns to replicate the teacher's behavior while accounting for the precision loss introduced by INT8 or INT4 quantization. This produces models that are both architecturally compact and numerically efficient, ideal for deployment on edge devices like portable ultrasound systems or bedside monitors.

Beyond matching final output probabilities, feature-based distillation transfers knowledge from the teacher's intermediate hidden layers. The student is trained to minimize the distance between its internal representations and the teacher's at corresponding layers. In clinical models, this can preserve the hierarchical feature extraction that identifies medical entities—from character-level morphology to document-level diagnostic reasoning.

A technique where a model distills knowledge into itself through iterative training cycles. The model's own predictions from previous epochs serve as soft targets for subsequent training, progressively refining its internal representations without requiring a separate teacher. This is particularly valuable when a larger clinical model is unavailable, allowing a BioBERT-scale model to bootstrap its own performance improvements on medical NER tasks.