Memorization Localization

A causal intervention technique that replaces a model's internal activation at a specific layer and token position with an activation from a different, corrupted forward pass. This isolates whether that specific activation is necessary and sufficient for a memorized behavior.

• Contrast with Causal Tracing: While causal tracing restores clean activations into a corrupted run, activation patching can also inject corrupted activations into a clean run to break specific circuits.
• Application: Used to identify the exact attention heads that copy verbatim text from a memorized training example versus heads that perform general syntactic processing.
• Granularity: Can be applied at the level of entire layers, individual attention heads, or even specific neuron groups within an MLP.

A method for pinpointing the specific MLP neurons that store factual associations by analyzing the relationship between neuron activation values and the model's output probability for a fact. Knowledge neurons are causally implicated in the expression of specific knowledge.

• Identification Metric: The integrated gradient of a neuron's activation with respect to the model's factual prediction, summed across a diverse set of prompts expressing the same fact.
• Key Property: Suppressing a small set of identified knowledge neurons (often < 5% of an MLP layer) can completely erase the model's ability to recall a specific fact while leaving its general language capabilities intact.
• Implication: Demonstrates that factual memorization is often sparse and concentrated in a small number of model parameters.

Uses the gradient of the model's loss with respect to its input tokens to identify which parts of a training sequence the model relied on to produce a memorized output. High gradient magnitudes indicate tokens that were critical for the prediction.

• Saliency Maps: The simplest form, computing the absolute value of the input gradient to visualize which tokens the model was most sensitive to.
• Integrated Gradients: A more robust variant that accumulates gradients along a path from a baseline (e.g., zero embedding) to the actual input, satisfying the completeness axiom where attributions sum to the output difference.
• Limitation: Identifies which inputs are important but not which internal parameters store the memorized information.

A statistical technique that estimates how removing or perturbing a specific training example would change the model's loss on a given test input. This directly links memorized outputs back to their originating training samples.

• Mechanism: Computes the second-order effect of upweighting a training point on the model's parameters using the inverse Hessian-vector product, without requiring expensive retraining.
• Use Case: Identifies the exact training documents responsible for a model's verbatim regurgitation, enabling data provenance auditing.
• Practical Challenge: Exact influence functions are computationally prohibitive for large models; approximations like TracIn and EK-FAC are used in practice.

Trains a simple linear classifier (probe) on a model's internal activations to detect whether a specific memorized string or pattern is encoded at a given layer. The probe's accuracy reveals the layer where the memorized content becomes linearly separable.

• Methodology: For a target memorized sequence, create a dataset of prompts that do and do not elicit the memorized output. Train a logistic regression probe on the residual stream activations at each layer.
• Interpretation: A sudden spike in probe accuracy at a specific layer indicates the point where the memorized content is retrieved from the model's weights into the residual stream.
• Advantage: Computationally lightweight and does not require causal interventions, making it suitable for scanning large models across many layers.

Specific MLP neurons identified as causally responsible for expressing stored factual knowledge. Identified through integrated gradient attribution, these neurons exhibit high activation when the model recalls a fact and suppressing them degrades factual recall. Key properties include:

• Sparsity: Only a tiny fraction of neurons encode any given fact
• Transferability: The same neurons often activate for semantically related facts
• Layer concentration: Knowledge neurons cluster in middle-to-late MLP layers of transformer architectures

A causal intervention technique that replaces a model's internal activation at a specific layer and token position with a value from a corrupted or alternative forward pass. By patching activations from a clean run into a corrupted run (or vice versa), researchers isolate which components are necessary and sufficient for a behavior. This is the primary experimental tool for validating hypothesized circuits and localizing memorized content to specific sublayers.

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. When applied to layers suspected of memorization, SAEs can separate:

• Generalizable pattern features from memorized instance features
• Semantic concept directions from verbatim storage directions This decomposition is critical for distinguishing memorization from genuine generalization in the model's representational space.

A fast, gradient-based approximation of activation patching that estimates the indirect effect of a component on the output. Instead of running full forward passes with corrupted activations, attribution patching uses a first-order Taylor expansion to compute the importance of each node. This enables scalable localization across millions of parameters, making it practical to identify all weights involved in storing a specific memorized sequence without exhaustive causal experiments.

The direct surgical modification of weights to update or erase stored facts without retraining. Techniques like ROME and MEMIT identify the specific weight matrices responsible for factual associations and apply rank-one updates to alter them. This is the applied counterpart to memorization localization—once you know where a fact lives, you can rewrite it. Critical for privacy compliance, enabling targeted removal of memorized PII from deployed models.