Attention Transfer

The core objective is to align the attention maps of the student model with those of the teacher. An attention map is a matrix of scores that indicates which parts of the input (e.g., words in a sentence, patches in an image) the model 'attends to' when making a prediction.

• Process: During training, the student is penalized not just for incorrect final outputs, but for having attention patterns that diverge from the teacher's.
• Example: In a vision transformer, this means the student learns to focus on the same image regions (e.g., a dog's eyes and nose) as the teacher for the 'dog' class.

Attention Transfer operates on intermediate layers of the neural network, not just the final output layer. This provides a richer, more granular supervisory signal.

• Mechanism: Attention maps are extracted from specific, often multiple, transformer blocks or convolutional layers in the teacher model.
• Benefit: It forces the student to internalize the teacher's feature extraction and information routing strategies, leading to better generalization than just mimicking final logits.
• Implementation: A hint layer in the teacher is often designated, and its attention output is used to guide a corresponding guided layer in the student.

The training objective combines a standard task loss (e.g., cross-entropy) with a specialized attention transfer loss.

• Common Formulation: The loss is often a Mean Squared Error (MSE) or Cosine Similarity between the teacher's and student's attention maps, summed across layers and data samples.
• Formula (simplified): Total Loss = Task_Loss + β * Attention_Loss, where β is a weighting hyperparameter.
• Rationale: This multi-objective loss ensures the student learns both the correct answer and the teacher's reasoning process.

Attention Transfer is highly flexible regarding the architectures of the teacher and student models.

• Heterogeneous Architectures: The teacher and student do not need identical structures. A large Vision Transformer (ViT) teacher can distill knowledge into a small Convolutional Neural Network (CNN) student.
• Layer Correspondence: The method requires defining a mapping between which teacher layers provide supervision for which student layers. This mapping can be adaptive.
• Modality Agnostic: Successfully applied in Natural Language Processing (transformer models), Computer Vision (CNNs and ViTs), and multimodal settings.

By mimicking attention, the student learns relational or structural knowledge about the input data.

• Dark Knowledge of Context: It learns which input elements are semantically related, as indicated by the teacher's high cross-attention scores.
• Example in NLP: For the sentence 'The bank by the river is steep,' the teacher's attention might strongly link 'bank' and 'river.' The student learns this contextual association, not just that 'bank' is a noun.
• Result: This often leads to more robust and interpretable student models, as their 'focus' is semantically meaningful.

Attention Transfer introduces specific computational costs and design decisions.

• Overhead: Extracting, storing, and comparing attention maps for multiple layers increases memory usage and training time compared to simple logits distillation.
• Map Selection: Not all attention heads or layers are equally valuable. A key design choice is selecting the most salient or task-relevant attention maps for transfer to optimize the cost-benefit ratio.
• Normalization: Attention maps often require normalization (e.g., spatial, layer-wise) before comparison to ensure stable training, as their magnitudes can vary widely across layers.

Knowledge Distillation is the overarching model compression paradigm where a smaller student model is trained to mimic the behavior of a larger, pre-trained teacher model. The core idea is to transfer the teacher's 'knowledge'—its learned function mapping—into a more efficient form. This is typically achieved by using the teacher's output probabilities (soft targets) as a richer training signal than standard hard labels, allowing the student to learn inter-class relationships (dark knowledge).

Feature Mimicking is a direct precursor to Attention Transfer. Instead of aligning final outputs, this method forces the student model to replicate the intermediate feature representations or activations from specific layers of the teacher model. A hint layer in the teacher is selected, and the student's corresponding layer is trained via a loss function (e.g., Mean Squared Error) to produce similar activations. This transfers internal, task-relevant representations, not just final decisions.

Attention Map Distillation is the most closely related technique to Attention Transfer. It specifically focuses on transferring the spatial or contextual attention patterns generated by the teacher's attention mechanisms. In vision transformers (ViTs) or convolutional networks with attention modules, the student is trained to produce similar attention weight distributions. This directly guides the student on where or what to focus on within the input data, a form of structural knowledge transfer.

Intermediate Layer Distillation is a generalized framework for transferring knowledge from the teacher's hidden layers. It encompasses methods like Feature Mimicking and Attention Transfer. The key challenge is layer alignment: determining which student layer should learn from which teacher layer. Strategies include:

• Direct one-to-one mapping of corresponding depths.
• Using a thin regressor (e.g., a linear layer) to adapt dimensions between non-matching layers.
• Adaptive selection of the most informative teacher layers.

Logits Distillation is the foundational KD technique introduced by Hinton et al. The student is trained to match the teacher's soft targets—the class probability distribution produced by applying a temperature-scaled softmax to the teacher's output logits. The high temperature (T > 1) creates a softer distribution, revealing which classes the teacher considers similar (dark knowledge). The training loss combines a standard cross-entropy loss with a Kullback-Leibler Divergence Loss between the student and teacher distributions.

The Teacher-Student Framework is the fundamental architecture for all knowledge distillation. It consists of two models:

• Teacher: A large, accurate, often over-parameterized model that has been pre-trained on a task.
• Student: A smaller, more efficient model (different architecture or compressed version) being trained. The teacher generates supervisory signals (logits, features, attention maps) to guide the student's training, acting as a learned label smoother and regularizer. The framework can be offline (static teacher) or online (both models train concurrently).