Attention Transfer

The core mechanism involves training the student to minimize the distance between its intermediate attention maps and those of the teacher. This is achieved by defining an attention transfer loss—typically L2 or Huber loss—between the normalized activation tensors of corresponding layers. The student learns to activate the same spatial regions as the teacher, effectively inheriting its visual focus mechanism without needing to match the full high-dimensional feature representations.

Attention maps are derived from the teacher's intermediate activations. A common approach is to compute a spatial attention map by aggregating the absolute values of the activation tensor across the channel dimension:

• Sum of absolute values: A = sum(|F_i|) across channels
• Sum of squared values: A = sum(F_i^2)
• Max activation: A = max(|F_i|) across channels This produces a 2D heatmap highlighting which spatial locations the teacher's layer considers important for the given input.

Attention transfer can be applied at multiple architectural levels:

• Single-layer transfer: Matching attention maps at one critical bottleneck layer
• Multi-layer transfer: Aligning attention across several paired layers, often at the end of each residual block or stage
• Progressive transfer: Gradually transferring attention from early to late layers during training The choice depends on the architectural similarity between teacher and student. Deeper transfer generally yields better fidelity but requires careful layer pairing.

The student is optimized using a composite loss function that balances task performance with attention alignment:

• Task loss: Standard cross-entropy with ground-truth labels
• Attention transfer loss: Distance metric between normalized teacher and student attention maps
• Hyperparameter beta: Controls the weight of the attention loss relative to the task loss This joint optimization ensures the student remains accurate on the primary task while learning the teacher's internal focus patterns.

By forcing the student to replicate the teacher's attention, the technique provides a window into the teacher's decision process:

• Visual explanations: The student's attention maps serve as faithful proxies for where the teacher focuses
• Debugging tool: Mismatches between teacher and student attention can reveal where the student fails to capture critical patterns
• Architecture validation: Confirms that the student's simplified architecture can still represent the teacher's attention dynamics This makes attention transfer valuable not just for compression, but for model auditing and transparency.

Attention transfer is a specific form of feature-based distillation, distinct from the classic logit-based approach:

• Logit distillation: Matches final output probabilities (soft targets)
• Attention transfer: Matches intermediate spatial attention maps
• Combined approach: Often used together—attention transfer regularizes intermediate representations while logit distillation aligns final predictions This dual transfer typically outperforms either method alone, as it transfers both where the model looks and what the model concludes.