Attention Head Pruning

Pruning decisions rely on a proxy metric to rank head utility without exhaustive re-evaluation. Common approaches include:

• Masking confidence: Measuring the expected effect of masking a head on the loss function using a first-order Taylor expansion.
• Attention entropy: Removing heads that produce near-uniform attention distributions, indicating a lack of focused contextualization.
• Layer-wise relevance propagation: Tracing output contributions back through the network to quantify head saliency. The choice of metric directly determines the efficiency-accuracy trade-off.

Head pruning is a form of structured pruning, distinct from weight-level sparsity.

• Structured (Head) Pruning: Removes entire attention head matrices (Q, K, V, O projections). This yields immediate, hardware-friendly speedups on standard GPUs without specialized sparse kernels.
• Unstructured Pruning: Zeros out individual weights within a head. This achieves higher compression ratios but requires custom sparse matrix libraries for runtime gains. Head pruning preserves the dense matrix multiplication patterns that modern accelerators are optimized for.

One-shot removal of a large fraction of heads often causes catastrophic performance collapse. A robust strategy involves iterative pruning with fine-tuning:

1. Rank all heads by the chosen importance metric.
2. Remove a small percentage (e.g., 10%) of the lowest-ranked heads.
3. Fine-tune the pruned model on the original training data for a short duration to allow remaining heads to compensate.
4. Repeat until the target sparsity or a performance threshold is reached. This gradual process allows the network to redistribute functionality.

Not all Transformer layers are equally sensitive to head removal. Empirical analysis reveals distinct patterns:

• Early layers: Often contain heads that encode broad syntactic features and positional relationships. They tend to be more sensitive to pruning.
• Middle layers: Typically exhibit higher redundancy, with many heads learning overlapping semantic functions. These layers often tolerate the highest pruning ratios.
• Final layers: Heads here are highly specialized for the specific task objective. Aggressive pruning in these layers frequently causes the largest drop in accuracy. A uniform pruning ratio across all layers is suboptimal.

The motivation for head pruning stems from the observed over-parameterization in multi-head attention. Studies show that a significant fraction of heads can be removed without harm because:

• Redundant heads: Multiple heads within a layer learn nearly identical attention patterns, providing no additional representational capacity.
• Non-contributing heads: Some heads learn degenerate patterns (e.g., attending only to the [SEP] token) that contribute negligible information to the final output.
• Task-agnostic heads: Heads that encode general linguistic knowledge may be pruned when fine-tuning for a narrow downstream task, as their function is no longer required.

The primary engineering benefit is a direct reduction in FLOPs and memory bandwidth during autoregressive decoding. Removing a head eliminates:

• The matrix multiplications for its Q, K, V, and output projections.
• The corresponding attention score computation and weighted sum.
• A portion of the KV cache memory footprint, which is critical for long-context generation. For a model with 12 layers and 12 heads, pruning 50% of heads can yield a theoretical 2x speedup in the attention sub-layers, significantly reducing per-token latency.