Attention Transfer

The core mechanism involves training the student to match attention maps from the teacher. These maps are often calculated as the sum of squared values across channels at specific intermediate layers (e.g., in a ResNet). The loss function minimizes the difference between these spatial attention distributions, forcing the student to learn where the teacher focuses its computational resources.

• Objective: Align student's feature importance with teacher's.
• Common Formulation: Use Mean Squared Error (MSE) or Kullback-Leibler (KL) divergence on normalized attention maps.
• Impact: Transfers inductive bias about salient regions, often leading to better generalization than matching raw feature activations.

Unlike logit distillation which operates on final outputs, Attention Transfer uses intermediate representations. It provides a richer, more granular supervisory signal by guiding the student's internal feature transformations.

• Hint Layers: The teacher's activations at designated 'hint' layers are transformed into attention maps.
• Guided Layers: Corresponding student layers are trained to produce similar attention.
• Benefit: This mid-level supervision helps the student learn better feature hierarchies and can improve convergence, especially for very deep or thin student networks.

Teacher activations are 3D tensors (channels x height x width). Attention Transfer aggregates across channels to create 2D spatial maps, emphasizing where information is important, not just what features are present.

• Common Method: Sum of squared activations across the channel dimension: (A = \sum_c (F_{h,w,c})^2).
• Normalization: Maps are often spatially normalized (e.g., L2 norm) to focus on relative importance.
• Result: The student learns a compressed, spatial summary of the teacher's feature importance, which is computationally efficient to match.

Attention Transfer is frequently combined with standard logit distillation (matching soft targets) and the task's cross-entropy loss. This creates a multi-objective loss function that transfers knowledge at multiple levels of abstraction.

• Full Loss Function: (L_{total} = L_{task} + \alpha L_{logit} + \beta L_{attention}).
• Synergy: Logit distillation transfers what to predict, while attention transfer guides how to build the representations for that prediction.
• Empirical Result: The combination often yields student models that outperform those trained with either method alone.

The method is not tied to a specific network family. While seminal work used ResNets, it has been successfully applied to Transformers (where it aligns attention heads or layer outputs), CNNs, and other architectures.

• Flexibility: The core principle—matching spatial importance—translates across model types.
• Transformer Adaptation: In Vision Transformers (ViTs), attention maps from the multi-head self-attention blocks can be distilled.
• Benefit: Makes it a versatile tool for compressing or training a wide variety of model architectures.

Attention Transfer is a key technique in distillation for forgetting, used in algorithms like Learning without Forgetting (LwF). Here, the model's own attention maps from a previous task are used as a regularization target when learning a new task.

• Mechanism: The model's frozen copy (teacher on old tasks) provides attention maps. The learning model (student on new task) is penalized for deviating from these maps on rehearsal or generated data.
• Purpose: Preserves the feature-level behavior on old tasks, not just final outputs, which helps maintain representational stability.
• Outcome: Mitigates catastrophic forgetting by anchoring the model's internal feature focus.

Knowledge distillation is a model compression and transfer learning technique where a smaller, more efficient student model is trained to mimic the behavior of a larger, more complex teacher model. The student learns not just from hard labels but from the teacher's softened output probabilities (logits) or intermediate feature representations, capturing richer relational information known as dark knowledge.

Feature distillation is a knowledge distillation method where the student model is trained to replicate the intermediate feature maps or activations from specific layers of the teacher model. Unlike logit distillation, it transfers representational knowledge about how the teacher transforms input data. Common techniques include:

• Hint training (e.g., FitNets), where a student 'hint' layer learns from a teacher 'guided' layer.
• Using loss functions like Mean Squared Error (MSE) or Maximum Mean Discrepancy (MMD) to align feature spaces.

Attention distillation is a specialized form of feature distillation tailored for transformer-based architectures. The student is trained to mimic the attention maps or patterns of the teacher model. This transfers the teacher's contextual understanding and focus mechanisms. Common approaches include:

• Matching attention matrices from multi-head attention layers.
• Distilling the sum of squared values across channels for spatial attention maps.
• This technique is foundational to models like TinyBERT, which distill attention from BERT's layers.

The teacher-student framework is the foundational architecture for knowledge distillation. It consists of:

• A pre-trained, often larger, teacher model that provides supervisory signals.
• A student model (smaller or more efficient) that learns from these signals.
• A distillation loss (e.g., KL Divergence) that measures the discrepancy between teacher and student outputs or features. The framework can be offline (static teacher) or online (co-evolving models), and is central to techniques like Attention Transfer.

Dark knowledge refers to the rich, non-obvious information contained within the softened probability distribution of a trained neural network's logits. It reveals relative similarities between classes (e.g., a 'cat' is more like a 'dog' than a 'truck').

Temperature scaling is the key technique to access this knowledge. A temperature parameter (T > 1) is applied to the softmax function, softening the output distribution and amplifying the dark knowledge. The student model is then trained on these soft targets.

Distillation for forgetting applies knowledge distillation as a regularization technique in continual learning to mitigate catastrophic forgetting. The current model's own predictions on data from previous tasks are used as soft targets (or a stored old model acts as the teacher) when learning a new task. This penalizes changes to outputs for old tasks. Key algorithms include:

• Learning without Forgetting (LwF): Uses distillation from the model's previous parameters.
• iCaRL: Combines distillation with an exemplar memory and a nearest-mean classification rule.