Logits Distillation

Logits are the raw, unnormalized scores a neural network generates for each possible class before applying the final activation function (like softmax). In logits distillation, these values are the primary knowledge vector transferred from teacher to student.

• Key Property: They contain richer information than a one-hot label, including the model's relative confidence across all classes.
• Example: For a 3-class image task, a teacher's logits might be [5.2, 1.1, -0.3], indicating strong confidence in class 1, weak in class 2, and very weak in class 3.

Temperature Scaling is a critical hyperparameter applied to the softmax function to control the 'softness' of the teacher's output distribution.

• Mechanism: The logits are divided by a temperature parameter T before softmax: softmax(logits / T).
• Effect: A T > 1 (e.g., T=4) smoothens the probability distribution, amplifying the dark knowledge—the subtle relationships between non-predicted classes.
• Purpose: It provides a more informative training signal for the student than the hard, peaked distribution (T=1) used for final inference.

The distillation loss quantifies the discrepancy between the teacher and student's softened output distributions. The Kullback-Leibler (KL) Divergence is the standard objective.

• Formula: L_KD = T^2 * KL( softmax(z_s / T) || softmax(z_t / T) ), where z_s and z_t are student and teacher logits.
• Why T^2?: This scaling factor ensures the gradient magnitudes remain consistent when T changes.
• Combined Loss: In practice, this is often combined with a standard cross-entropy loss using a ground-truth label: L_total = α * L_CE + (1-α) * L_KD.

A defining feature of logits distillation is its architecture-agnostic nature. The teacher and student models can have radically different internal structures.

• Constraint: They must only share the same output space (e.g., the same set of classification labels).
• Flexibility: This allows distilling a massive transformer-based teacher (e.g., BERT) into a tiny convolutional student (e.g., MobileNet) for the same task.
• Contrast: This differs from feature mimicking or attention transfer, which require architectural alignment or adaptation layers to match intermediate representations.

Dark Knowledge is the implicit, relational information embedded within a trained model's softened output distribution. It is the core 'knowledge' transferred during logits distillation.

• Content: It encodes the teacher's learned similarities between classes. For example, an image of a 'cat' might produce slightly elevated probabilities for 'lynx' and 'tiger' versus 'truck'.
• Transfer: By matching the teacher's softened probabilities, the student learns not just which class is correct, but the structure of the label space as understood by the more powerful teacher.

This defines the temporal relationship between teacher training and distillation.

• Offline Distillation: The standard approach. A static, pre-trained teacher model provides logits. The student is trained separately. This is simple and stable.
• Online Distillation: The teacher and student are trained jointly from scratch. Their logits are compared in each training step. This can yield a better, co-adapted student but is more complex to optimize.
• Self-Distillation: A sub-case of online distillation where the same model acts as both teacher and student, often distilling from deeper to shallower layers within the same network.

Logits distillation is a cornerstone for deploying capable models on smartphones, IoT devices, and microcontrollers. By training a small student to mimic a large cloud-based teacher's logits, the resulting model achieves high accuracy with a drastically reduced memory and compute footprint, enabling real-time inference without a network connection.

• Key Benefit: Enables complex AI features (like on-device translation or object detection) on hardware with limited RAM and CPU power.
• Example: A 500M parameter teacher model distilled to a 50M parameter student for offline speech recognition on a mobile device.

This is the most direct application for reducing inference latency and cloud compute costs. Companies use logits distillation to shrink large, expensive-to-serve models (e.g., for sentiment analysis, search ranking, or recommendation) into compact versions that maintain performance.

• Process: A high-accuracy but slow teacher model in production provides soft labels to train a faster, cheaper-to-run student model.
• Outcome: Significant reduction in API latency and infrastructure cost while preserving model quality for end-users.

Logits distillation enables collaborative learning across decentralized devices (e.g., hospitals, personal phones) without sharing raw data. Instead of exchanging sensitive training data or full model weights, devices share only the soft targets (temperature-scaled logits) from their local teacher models. A central server aggregates these distilled knowledge signals to train a global student model.

• Privacy Guarantee: Raw user data never leaves the local device.
• Use Case: Training a next-word prediction model across millions of phones while keeping all personal typing data private.

Logits distillation can amalgamate knowledge from multiple specialized teacher models into a single, unified student. Each teacher (e.g., one for classification, one for named entity recognition) provides its logits as a supervisory signal. The student learns to approximate all tasks simultaneously, avoiding the combinatorial cost of running multiple large models.

• Advantage: Creates a versatile, general-purpose model from a set of experts.
• Application: A single on-device assistant model capable of intent classification, entity extraction, and sentiment analysis, distilled from separate large teachers.

When the original training data is unavailable (due to privacy, licensing, or loss), logits distillation can proceed using synthetically generated data. A generator creates inputs that maximize the response diversity of the teacher model's logits. The student is then trained on these synthetic samples, using the teacher's logits as the sole learning signal.

• Solves: Data scarcity and privacy regulations (e.g., GDPR).
• Process: Known as Data-Free Distillation, it allows legacy model knowledge to be transferred long after the original dataset is gone.

Logits distillation is often combined with quantization techniques for extreme compression. In Quantization-Aware Distillation (QAD), the student model is trained with simulated low-precision (e.g., INT8) arithmetic during the distillation process. The teacher's full-precision logits guide the student to be robust to the numerical errors introduced by quantization.

• Result: A model that is both small from distillation and optimized for efficient integer math on NPUs/TPUs.
• Deployment Target: Ultra-low-power chips where every bit of precision and memory counts.

Knowledge Distillation is the overarching model compression paradigm where a compact student model is trained to replicate the behavior of a larger, more complex teacher model. Logits distillation is a primary method within this framework, focusing on matching the teacher's final pre-softmax outputs. The core goal is to transfer the teacher's learned representations and decision boundaries to a smaller, more efficient model suitable for deployment.

The Teacher-Student Framework is the foundational architecture for knowledge distillation. It consists of:

• A pre-trained, high-capacity Teacher Model that provides supervisory signals.
• A smaller, more efficient Student Model that is trained to mimic the teacher.
• A distillation loss function (like KL divergence) that measures the discrepancy between their outputs. This framework formalizes the one-way transfer of 'dark knowledge' and is the structural basis for logits distillation.

Soft Targets are the probability distributions output by the teacher model after applying a temperature-scaled softmax. Unlike hard, one-hot labels (e.g., [0, 0, 1, 0]), soft targets (e.g., [0.01, 0.04, 0.9, 0.05]) contain richer information about inter-class similarities—the 'dark knowledge.' In logits distillation, the student is trained to match these softened distributions, learning not just the correct class but the relative relationships between all classes.

Temperature Scaling is a critical hyperparameter technique in logits distillation. A temperature parameter T is introduced into the softmax function: softmax(z_i / T).

• T = 1: Standard softmax.
• T > 1: 'Softens' the probability distribution, amplifying the dark knowledge in non-target class probabilities.
• T → ∞: Outputs approach a uniform distribution. By using T > 1 during distillation, the teacher produces a more informative training signal for the student, which is crucial for effective knowledge transfer.

Dark Knowledge refers to the implicit, relational information encoded within a trained neural network's output distribution. It represents the model's understanding of similarities between different classes (e.g., that a 'cat' is more similar to a 'dog' than to an 'airplane'). Logits distillation, via temperature scaling, is designed specifically to extract and transfer this dark knowledge from the teacher's softened logits to the student, enabling the smaller model to generalize better than if trained on hard labels alone.

Kullback-Leibler (KL) Divergence Loss is the most common objective function for logits distillation. It quantitatively measures the difference between the softened output probability distributions of the teacher (P) and the student (Q). The distillation loss is often formulated as: L_KD = T^2 * KL(P || Q), where T is the temperature. Minimizing this KL divergence forces the student's logits to produce a probability distribution that is statistically similar to the teacher's, thereby transferring the dark knowledge.