Model Distillation

The process involves a two-model setup: a large, pre-trained teacher model with high capacity and a smaller student model with fewer parameters. The student is trained not on the original ground-truth labels alone, but to mimic the teacher's output distribution.

• Soft targets: The teacher produces a probability distribution over classes, revealing inter-class similarities (e.g., a cat image is more similar to a dog than a truck).
• Softmax temperature: A hyperparameter (T) sharpens or smooths the teacher's output probabilities. Higher temperatures reveal more granular knowledge about non-target classes.
• Combined loss: The student optimizes a weighted sum of the distillation loss (matching the teacher) and the standard task loss (matching ground truth).

Distillation transfers three distinct types of knowledge from teacher to student:

• Response-based knowledge: The teacher's final output layer activations (logits). This is the most common form, teaching the student the teacher's predictive confidence.
• Feature-based knowledge: The intermediate representations and activation maps from the teacher's hidden layers. The student learns to replicate internal feature hierarchies.
• Relation-based knowledge: The pairwise relationships between data samples as learned by the teacher, often captured through similarity matrices or Gram matrices.

Each mechanism captures different structural information, and combining them often yields the best student performance.

Unlike standard supervised training, distillation provides richer supervision signals:

• Standard training: The model sees only a hard label (e.g., 'dog'). All incorrect classes receive zero gradient, discarding information about class similarity.
• Distillation: The student sees the teacher's full probability distribution. Even incorrect classes receive gradient signals proportional to their similarity, providing more information per training example.
• Result: Students often generalize better than identically-sized models trained from scratch on the same dataset, achieving higher accuracy with fewer parameters.

Distillation strategies fall into two execution modes:

• Offline distillation: The teacher is pre-trained and frozen. The student trains on the teacher's static outputs. This is the standard approach and requires storing or generating the teacher's predictions in advance.
• Online distillation: Teacher and student train simultaneously, with the teacher updating its knowledge and the student learning from the evolving teacher. This is useful when no suitable pre-trained teacher exists.
• Self-distillation: A special case where the student and teacher share the same architecture. The model distills knowledge from its own earlier training epochs or deeper layers into shallower ones.

Distillation directly addresses the carbon footprint of AI inference by enabling smaller models to serve production traffic:

• Inference energy: A distilled student model may require 1/100th the FLOPs per inference compared to its teacher, translating to proportional energy savings.
• Hardware requirements: Students can run on CPUs or edge devices, eliminating the need for power-hungry GPU clusters for deployment.
• Embodied carbon avoidance: Smaller models extend hardware lifespan and reduce the need for new accelerator manufacturing.
• Example: DistilBERT retains 97% of BERT's language understanding while being 40% smaller and 60% faster, significantly lowering the joules-per-inference metric.

For LLMs, distillation takes specialized forms beyond output matching:

• Data distillation: The teacher generates high-quality synthetic training data (instruction-response pairs), and the student trains on this curated dataset. This is the method behind models like Alpaca and Vicuna.
• Chain-of-thought distillation: The teacher generates step-by-step reasoning traces, teaching the student not just what to answer but how to reason through problems.
• Speculative decoding: A small draft model proposes tokens, and a large model verifies them. While not training-based distillation, it achieves similar latency reductions at inference time.

A complementary optimization technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). While distillation transfers knowledge between architectures, quantization directly shrinks the bit-width of the parameters within a single model. This dramatically decreases memory bandwidth requirements and energy consumption per inference, often applied to the student model post-distillation for maximum efficiency.

A research paradigm that treats computational cost as a primary evaluation metric alongside accuracy. Green AI directly contrasts with 'Red AI', which maximizes performance regardless of energy expenditure. Model distillation is a canonical Green AI technique because it explicitly trades a marginal accuracy drop for an order-of-magnitude reduction in FLOPs and carbon emissions during inference.

A direct measurement of the energy required for a deployed model to process a single input and generate an output. This metric is the ultimate validation of distillation's effectiveness. A student model might achieve:

• Teacher model: 15 Joules per inference
• Distilled student: 0.9 Joules per inference This metric is critical for calculating the operational carbon footprint of high-traffic AI services.

The primary hardware efficiency metric used to rank supercomputers on the Green500 list. It measures the number of floating-point operations a processor can execute per unit of energy. Distilled models require fewer total FLOPs, allowing them to run efficiently on hardware with lower absolute power draw, effectively increasing the sustainable throughput of existing data center infrastructure.

A systematic analysis of a model's environmental impact across all stages: raw material extraction for hardware, energy for training, operational inference, and final decommissioning. Distillation impacts the use phase of the LCA most significantly by reducing the operational energy burden, but it also enables deployment on smaller, less embodied carbon-intensive edge hardware.

The practice of time-shifting or location-shifting computational workloads to periods or regions where the grid's marginal emissions rate is lowest. Lightweight distilled models are more agile and portable, making them ideal candidates for carbon-aware scheduling. Their smaller footprint allows them to be rapidly spun up in low-carbon regions without massive data transfer overhead.