In-Context Learning (ICL)

The primary mechanistic driver of ICL. Induction heads are specialized attention circuits that perform a two-step operation: they attend to a previous occurrence of the current token, then attend to the token that followed that previous occurrence. This allows the model to implicitly copy and complete patterns seen in the context.

• Pattern: [A][B] ... [A] → [B]
• Function: These heads emerge during training and are responsible for recognizing and extending sequences without explicit gradient updates.
• Significance: The discovery of induction heads proved that ICL relies on emergent, structured internal algorithms rather than simple memorization.

ICL operates on a spectrum of conditioning specificity.

• Zero-Shot: The model performs a task based solely on the instruction, with no examples. Relies entirely on the model's pre-trained semantic understanding of the task label.
• One-Shot: A single demonstration is provided. This anchors the model to the desired input-output mapping format.
• Few-Shot: Multiple demonstrations (typically 2–32) are provided. Performance generally scales with the number of examples until the context window limit is reached, at which point context window truncation may degrade performance.

The spatial positioning of demonstrations within the prompt significantly impacts accuracy due to the Lost in the Middle phenomenon. Models attend most strongly to the beginning and end of the context window.

• Recency Bias: Demonstrations placed at the end of the prompt exert disproportionate influence on the model's output distribution.
• Primacy Bias: Initial instructions or system prompts set the global behavioral constraints.
• Mitigation: Critical few-shot examples should be placed immediately before the final query to maximize their influence on the prediction.

ICL performance is highly sensitive to the distribution of labels in the demonstrations. The model learns the mapping between inputs and outputs, not just the input format.

• Balanced Distribution: Providing an equal number of examples for each class prevents the model from developing a biased prior toward a majority class.
• Label Flipping: Research shows that even when demonstration labels are randomized, the model retains some performance, indicating that ICL partly works by activating the correct task priors learned during pre-training, independent of the specific input-label mapping.

A leading theoretical framework posits that ICL functions as implicit Bayesian inference. The model treats the prompt demonstrations as observed data points and infers a latent task concept.

• Mechanism: The model maintains a posterior distribution over possible tasks. Each demonstration sharpens this distribution, effectively locating the correct task vector in the model's latent space.
• Implicit vs. Explicit: Unlike explicit fine-tuning, this inference happens entirely within a single forward pass, with no weight updates. The model's attention mechanism acts as the inference engine.

A distinct capability from semantic task learning. The model identifies and replicates the structural template of the demonstrations, including delimiters, whitespace, and output schemas.

• Structured Output: ICL is the primary method for enabling structured output formatting without fine-tuning. By providing examples of valid JSON or XML, the model learns to constrain its generation to match the syntactic pattern.
• Robustness: Format following is often more robust to label noise than semantic task learning, as the structural pattern is a surface-level feature easily captured by attention heads.

Provide 2-5 labeled examples in the prompt to classify new text.

Example Prompt:

• Text: 'The interface is clunky.' → Sentiment: Negative
• Text: 'Setup was a breeze.' → Sentiment: Positive
• Text: 'It arrived on time.' → Sentiment: Neutral
• Text: 'Battery drains fast.' → Sentiment:

The model infers the pattern and outputs Negative without explicit instructions. This works because attention mechanisms identify the input-output mapping structure.

Demonstrate step-by-step reasoning in the prompt to solve novel math problems.

Example Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans, each containing 3 balls. How many balls does he have now? A: Roger started with 5 balls. 2 cans × 3 balls = 6 balls. 5 + 6 = 11. Answer: 11.

Q: A café has 23 tables. They add 4 more tables, then remove 7. How many tables remain? A:

The model learns to decompose the problem and output 20 by mimicking the reasoning pattern.

Adapt text to a target tone by showing before-and-after examples.

Example Prompt: Rewrite the following in a formal, diplomatic tone. Informal: 'Your proposal is way too expensive.' Formal: 'The current proposal exceeds our anticipated budget parameters.'

Informal: 'We need this fixed ASAP.' Formal: 'We request expedited resolution of this matter.'

Informal: 'The report is full of errors.' Formal:

The model outputs 'The report contains several inaccuracies that require correction.' by learning the stylistic mapping.

Demonstrate how to map user intent to specific function calls using examples.

Example Prompt: Available functions: get_weather(city), send_email(to, subject) User: 'What's the temperature in Berlin?' Call: get_weather('Berlin')

User: 'Email the report to Sarah.' Call: send_email('[email protected]', 'Report')

User: 'Is it raining in Mumbai?' Call:

The model outputs get_weather('Mumbai') by pattern-matching the intent to the demonstrated function signature.

Translate code from one language to another by providing paired examples.

Example Prompt: Translate Python to Rust. Python:

pythonCopy
def factorial(n):
    return 1 if n <= 1 else n * factorial(n-1)

Rust:

rustCopy
fn factorial(n: u64) -> u64 {
    if n <= 1 { 1 } else { n * factorial(n - 1) }
}

Python:

pythonCopy
def fibonacci(n):
    a, b = 0, 1
    for _ in range(n): a, b = b, a + b
    return a

Rust:

The model generates idiomatic Rust with correct type annotations by learning the syntactic mapping from the example pair.