Induction Head

An induction head implements a two-step attention algorithm. In the first layer, a previous token head copies information about the immediately preceding token into the current position. In the second layer, the induction head uses the current token as a query to attend to positions where that same token appeared before, specifically focusing on the token that followed it. This allows the model to predict B when it sees A after observing the pattern AB earlier in the sequence.

• Step 1: A previous token head attends to the immediately preceding position
• Step 2: The induction head attends to the token following a previous occurrence of the current token
• Result: The model copies the continuation of a previously observed pattern

The induction head's behavior emerges from the precise interaction of its QK circuit (attention pattern) and OV circuit (information transfer). The QK circuit performs prefix matching: the query vector at the current position searches for keys at previous positions that are similar to the current token's embedding. The OV circuit then copies the value vector from the token immediately following the matched position, effectively retrieving the continuation of the pattern.

• QK Circuit: Implements a match between the current token and previous occurrences
• OV Circuit: Copies the token that followed the matched occurrence
• K-Composition: The key is often composed from a previous attention head's output in an earlier layer

Induction heads are the primary computational primitive behind in-context learning. When a model is presented with a few-shot prompt containing input-output pairs, induction heads detect the repeating pattern and generalize it to the final query. They do not update model weights; instead, they dynamically route information through the residual stream to copy the appropriate output format or answer structure from the demonstration examples.

• Enable zero-shot pattern completion without weight updates
• Form the basis of function vectors that can be extracted and injected
• Critical for tasks like few-shot translation, classification, and code completion

Induction heads do not exist in randomly initialized models. They form abruptly during training in a phase change, typically emerging after the model has learned bigram statistics. In a two-layer attention-only transformer, the first layer develops previous token heads that pass positional information forward. The second layer then composes its key from this information, creating the induction mechanism. In larger models, induction heads cluster in specific layers, often in the middle-to-late stages of the network.

• Formation is a discontinuous phase transition during training
• Requires prior development of previous token heads in earlier layers
• Emergence correlates with a sharp drop in loss on sequence copying tasks

An induction head exhibits a distinctive attention pattern that serves as its diagnostic fingerprint. When processing a sequence with repeated tokens, the attention map shows a characteristic off-by-one diagonal: the head attends to the position immediately following the previous occurrence of the current token. For a sequence like A B ... A, the head at the second A will attend strongly to the position of B that followed the first A.

• Attention focuses on token index = previous_match_index + 1
• Pattern is visible in attention heatmaps as a shifted diagonal
• Strength increases with the number of prior occurrences of the current token

Induction heads are specialized for pattern copying, not factual retrieval. They excel at replicating sequences observed in the current context window but do not directly access knowledge stored in MLP layers. This distinction is crucial: induction heads handle in-context patterns, while knowledge neurons in MLP layers handle memorized facts. The two systems can interact—an induction head may route a copied token to an MLP layer that enriches it with stored factual associations.

• Induction Heads: Copy patterns from the current context
• MLP Layers: Retrieve facts from parametric memory
• Interaction: Copied tokens can trigger factual recall in downstream layers

The computational pathway formed by the query (W_Q) and key (W_K) projection matrices. This circuit determines the attention pattern by computing the dot-product similarity between the current token's query and all previous tokens' keys. In an induction head, the QK circuit is specifically configured to attend to tokens that follow a previous occurrence of the current token, enabling in-context copying.

The computational pathway formed by the output (W_O) and value (W_V) projection matrices. This circuit determines what information is copied from the attended-to source token to the destination position. In an induction head, the OV circuit acts as a copying mechanism, transforming the source token's residual stream representation into a form that increases the probability of predicting that same token at the current position.

A specialized computational subunit within a transformer layer that computes a weighted sum of past token representations. Induction heads are a specific functional class of attention head discovered through mechanistic interpretability. They emerge in two-layer attention-only models and are hypothesized to be the primary mechanism driving in-context learning in large language models.

The process of identifying and validating the minimal subgraph of a neural network's computational graph responsible for a specific behavior. For induction heads, circuit analysis reveals a two-head composition: a previous token head in an earlier layer attends to the preceding token, and the induction head in a later layer uses that information to attend to the token after the current token's previous occurrence.

A causal intervention technique used to isolate functional circuits. To verify an induction head, researchers perform activation patching by replacing the head's output with activations from a corrupted run. If the model's ability to perform in-context copying is disrupted, the head's causal role is confirmed. This technique provides rigorous evidence that induction heads are necessary and sufficient for specific copying behaviors.

A position encoding method that applies a rotation matrix to query and key vectors based on their absolute positions. RoPE ensures the attention dot product depends only on the relative distance between tokens. Induction heads exploit this property to learn a 'match the previous occurrence' algorithm by attending to tokens at a specific relative offset, making RoPE a critical enabler of their copying functionality.