Glossary

Logits

Logits are the raw, unnormalized output scores from a neural network's final linear layer, which are transformed into probability distributions via a softmax function.

Get in touch Learn more

AI evaluator reviewing output quality on laptop, comparison metrics visible, casual evaluation session.

MODEL DISTILLATION

What are Logits?

The raw, unnormalized scores output by a neural network's final linear layer before activation.

Logits are the unnormalized output vectors from the final linear layer of a neural network, representing the model's raw scores for each possible class or token before any probability transformation. In knowledge distillation, these logits are the primary vehicle for transferring 'dark knowledge'—the rich, inter-class relationships learned by a large teacher model—to a smaller student model. The teacher's logits are softened using temperature scaling to create informative soft targets for the student to mimic.

The student model is trained using a distillation loss, typically the Kullback-Leibler Divergence Loss, which minimizes the statistical difference between the softened logit distributions of the teacher and student. This process allows the compact student to achieve accuracy comparable to the larger teacher, making logits a fundamental concept for inference optimization and latency reduction through techniques like model distillation and the creation of efficient variants such as DistilBERT and TinyBERT.

MODEL DISTILLATION

Key Characteristics of Logits

In knowledge distillation, logits are the raw, unnormalized scores output by a neural network's final linear layer. Their properties are central to transferring 'dark knowledge' from a teacher model to a student.

Unnormalized Output Vectors

Logits are the raw, unnormalized scores produced by the final linear (fully connected) layer of a neural network before any activation function is applied. They represent the model's evidence for each possible class.

Key Property: They are not probabilities; their values can range from negative to positive infinity.
Example: In a 3-class problem, a logit vector might be [2.1, -1.7, 0.5], where the first class has the strongest evidence.

Source of Dark Knowledge

The relative magnitudes and ordering of logits contain rich information about the teacher model's learned inter-class relationships, known as dark knowledge. This is the core signal used in distillation.

A high logit for the correct class and a moderately high logit for a visually similar class (e.g., 'cat' and 'lynx') provides more learning signal than a simple one-hot label.
This relational information helps the student model generalize better and become more calibrated.

Transformation via Softmax & Temperature

To be used as training targets, logits are converted into a probability distribution using the softmax function. In distillation, a temperature parameter (T) is introduced to control the smoothness of this distribution.

Formula: softmax(logits_i) = exp(logits_i / T) / sum(exp(logits_j / T))
Effect of T:
- T = 1: Standard softmax, producing the teacher's native probabilities.
- T > 1: 'Softens' the distribution, amplifying the dark knowledge in non-top logits.
- T → ∞: Approaches a uniform distribution.

Contrast with Hard Labels

Logits/Soft Targets differ fundamentally from hard labels (one-hot encoded ground truth). This distinction is why distillation often outperforms training on hard labels alone.

Aspect	Hard Label	Soft Target (from Logits)
Format	`[0, 1, 0]`	`[0.05, 0.90, 0.05]`
Information	Only the correct class.	Relative confidence across all classes.
Gradient Signal	Sparse.	Dense, providing a richer learning signal per example.

Role in Distillation Loss

The student model is trained by minimizing a distillation loss that measures the difference between its softened outputs and the teacher's softened outputs (derived from logits). The standard loss is a weighted combination:

Distillation Term: Kullback-Leibler Divergence between the student and teacher softmax outputs (with temperature).
Hard Label Term: Standard cross-entropy with ground-truth labels.
Objective: L_total = α * L_CE(hard_labels) + (1-α) * T² * L_KL(teacher_soft || student_soft) The T² term scales the gradients from the KL divergence to match the magnitude of those from the hard labels.

Numerical Stability & Practical Use

Working directly with logits and the softmax function requires attention to numerical stability, especially when using temperature scaling.

Best Practice: Implement the LogSoftmax and KL Divergence functions together in a numerically stable way, often provided as a single operation in deep learning frameworks (e.g., F.kl_div(F.log_softmax(...), F.softmax(...))).
Initialization: The student model's final layer (which produces its logits) is often initialized to match the scale of the teacher's logits to facilitate stable optimization from the start of training.

CORE CONCEPT

The Role of Logits in Knowledge Distillation

Logits are the fundamental numerical signal enabling the transfer of 'dark knowledge' from a large teacher model to a compact student model.

In knowledge distillation, logits are the unnormalized, real-valued output vectors from a neural network's final linear layer before the softmax activation. These raw scores contain the model's relative confidence across all possible classes, encoding richer relational information—termed dark knowledge—than simple one-hot labels. During distillation, the teacher model's logits are softened using temperature scaling (a hyperparameter T > 1) to produce a smoother probability distribution. This softened distribution becomes the primary training target for the student model, which learns to mimic these probabilities rather than just the hard ground-truth labels.

The student model is trained using a distillation loss function, typically a weighted combination of the standard cross-entropy loss with true labels and a mimicry loss like Kullback-Leibler (KL) Divergence that measures the statistical distance between the teacher's and student's softened outputs. By aligning its logits with the teacher's, the student internalizes the teacher's nuanced understanding of class similarities and decision boundaries. This process allows the smaller student to achieve accuracy comparable to the larger teacher while being significantly more efficient for inference, directly supporting the goals of model compression and latency reduction in production systems.

CORE CONCEPTS

Logits vs. Probabilities: A Technical Comparison

A fundamental comparison of the raw, unnormalized scores (logits) output by a neural network's final linear layer and the normalized probability distributions produced by the softmax function, with a focus on their roles in knowledge distillation and inference optimization.

Feature / Property	Logits	Probabilities (Softmax Output)
Definition	The unnormalized, real-valued scores (often called log-odds) output by the final linear layer of a neural network prior to activation.	The normalized probability distribution over possible output classes, generated by applying the softmax function to the logits.
Mathematical Form	Vector z ∈ ℝ^C, where C is the number of classes. No inherent constraints on value range or sum.	Vector p ∈ (0,1)^C, where ∑_{i=1}^C p_i = 1 and p_i > 0. p = softmax(z) = exp(z_i) / ∑_j exp(z_j).
Value Range	(-∞, +∞). Can be any real number.	(0, 1). Bounded between 0 and 1, exclusive.
Sum of Vector	Unconstrained. Sum has no meaningful interpretation.	Constrained to sum to 1 (100%). Represents a valid probability distribution.
Interpretation in Distillation	The primary carrier of the teacher model's 'dark knowledge'. Relative magnitudes encode rich inter-class relationships.	The softened target distribution used for the student's mimicry loss (e.g., KL Divergence). Smoothness is controlled by temperature scaling.
Role in Training Loss	Used internally by frameworks; standard cross-entropy loss compares logits to ground truth via a softmax internally.	Explicitly used in distillation loss functions (e.g., KL Divergence) to compare teacher and student distributions.
Numerical Stability	Raw values can be very large or small, leading to overflow/underflow in exp() during softmax. Requires stable implementation (e.g., log-softmax).	Inherently stable after normalization, but direct computation from large logits can be unstable without careful coding.
Gradient Flow	Gradients can flow directly and are typically well-behaved. The softmax gradient involves the probabilities themselves: ∂L/∂z = p - y.	Gradients with respect to probabilities are simpler but are usually computed via the chain rule through the logits for efficiency.
Information Content	Contains maximal information before normalization. The full 'pre-softmax' signal.	Contains compressed information; extreme probabilities (near 0 or 1) lose fine-grained comparative detail.
Temperature Scaling (T)	Operation: z_scaled = z / T. Linearly scales the logit vector.	Operation: p = softmax(z / T). Exponentially affects the output distribution. As T → ∞, p → uniform distribution.
Use in Inference (Argmax)	argmax(z) is identical to argmax(p). The class with the highest logit is the predicted class.	argmax(p) is the standard final prediction. The probability value indicates model confidence.
Storage & Memory Footprint	Identical to probabilities (C floating-point values).	Identical to logits (C floating-point values).

LOGITS

Frequently Asked Questions

Logits are the unnormalized output vectors from a neural network's final layer. In knowledge distillation, they are the primary vehicle for transferring 'dark knowledge' from a teacher model to a student. This FAQ clarifies their role, mechanics, and importance in model compression.

Logits are the raw, unnormalized output scores produced by the final linear (fully connected) layer of a neural network, prior to any activation function like softmax. They represent the model's evidence for each possible class or token before being converted into a probability distribution.

In a classification model, if there are C classes, the logits are a vector of length C. A higher logit value for a given class indicates stronger evidence from the model that the input belongs to that class. The transformation from logits to probabilities is typically done via the softmax function, which exponentiates and normalizes the logits, ensuring they sum to 1.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

KNOWLEDGE DISTILLATION

Related Terms

Logits are a fundamental component within the knowledge distillation pipeline. The following terms define the key concepts, models, and techniques that interact with logits during the distillation process.

Soft Targets / Soft Labels

Soft targets are the probability distributions produced by applying a softmax function (often with temperature scaling) to the teacher model's logits. Unlike hard one-hot labels, they contain dark knowledge—rich information about inter-class similarities—which provides a superior learning signal for the student model.

Primary Role: Serve as the training target for the student model.
Key Property: Softer, more informative distributions than categorical labels.
Example: For an image of a cat, a soft target might be [Cat: 0.7, Dog: 0.25, Fox: 0.05], indicating visual similarity between classes.

Temperature Scaling

Temperature scaling is a hyperparameter technique that controls the smoothness of the probability distribution derived from logits. A temperature parameter (T) is introduced into the softmax function: softmax(logits / T).

T = 1: Standard softmax, producing the original prediction distribution.
T > 1: Smoothens the distribution, amplifying the dark knowledge in the teacher's logits. This is critical for effective distillation.
T → ∞: Outputs a uniform distribution.
Purpose: Makes the relative differences between logits more learnable for the student.

Kullback-Leibler Divergence Loss

Kullback-Leibler (KL) Divergence Loss is the primary mimicry loss function used in knowledge distillation. It measures the statistical distance between the softened output distributions of the teacher and student models.

Objective: Minimize the KL divergence, forcing the student's logits (after temperature scaling) to match the teacher's.
Formula: Often combined with standard cross-entropy loss in a weighted sum: Loss = α * CE(student, hard_labels) + β * KL(student_soft, teacher_soft).
Role: Directly operationalizes the transfer of knowledge encoded in the teacher's logits.

Dark Knowledge

Dark knowledge is the foundational concept that makes distillation possible. It refers to the inter-class relational information embedded within a trained model's logits, which is not present in one-hot ground truth labels.

Source: The relative magnitudes of a teacher model's logits for incorrect classes contain a "soft" similarity score.
Transfer Mechanism: This knowledge is transferred to the student via soft targets.
Example: A model's logits for a 'sedan' image might be high for 'car' and moderately high for 'truck', but low for 'airplane'. This similarity structure is the dark knowledge.

Teacher Model

A teacher model is a large, pre-trained, high-accuracy neural network whose knowledge is transferred to a smaller student model. Its primary role in distillation is to generate the logits that form the basis of the training signal.

Characteristics: Typically over-parameterized, computationally expensive, and highly accurate.
Function: During distillation, it performs a forward pass on training data to produce logits, which are then converted to soft targets.
Examples: BERT, GPT-4, ResNet-152, or any large foundation model can serve as a teacher.

Student Model

A student model is a smaller, more efficient neural network trained to replicate the performance of the teacher by learning from the teacher's logits. The student's own logits are the direct object of optimization during distillation.

Goal: Achieve accuracy comparable to the teacher with a reduced computational and memory footprint.
Training Signal: Its logits are trained to align with the teacher's softened logits via the distillation loss.
Outcome: A deployable model that has internalized the dark knowledge from the teacher's logits.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.