Concept Localization

A diagnostic technique that trains a simple linear model on the frozen internal activations of a network to predict a linguistic property or concept. High accuracy indicates the concept is encoded in that layer.

• Linearity Assumption: Assumes that if a concept is linearly separable from the activations, it is explicitly represented.
• Control Tasks: Requires rigorous comparison against random baselines to ensure the probe is not simply memorizing label distributions.
• Common Probes: Used to locate syntactic features like parse tree depth or semantic features like tense in large language models.

Moves beyond correlation to establish a causal link between a specific model component and a behavior by intervening on activations during the forward pass.

• Activation Patching: Replaces the activations of a specific layer or neuron from a 'corrupted' run with clean activations to see if the behavior is restored.
• Knockout Analysis: Ablates or zeroes out a specific attention head or neuron to measure the degradation in performance on a specific task.
• Interchange Interventions: Tests if a representation functions as a variable in a causal model by swapping it between inputs.

An unsupervised method for decomposing a layer's dense activations into a sparse set of monosemantic features using a bottleneck architecture with an L1 sparsity penalty.

• Dictionary Learning: Learns an overcomplete basis of feature vectors where each vector corresponds to a distinct, interpretable concept.
• Monosemanticity Goal: Aims to solve the problem of polysemantic neurons that fire for multiple unrelated concepts.
• Reconstruction Fidelity: Balances the sparsity of the decomposition against the error of reconstructing the original activations from the sparse features.

Leverages gradient-based feature attribution methods to aggregate importance scores at the level of internal neurons or channels rather than input pixels.

• Channel Attribution: Sums the gradients flowing into a specific feature map channel to determine its importance for a target class prediction.
• Neuron Conductance: Computes the total contribution of a neuron by integrating gradients along the scaling path from a baseline input.
• Concept Gradients: Uses directional derivatives to measure how much a class prediction changes as activations are pushed toward a concept vector.

A technique for locating the specific feed-forward network neurons in large language models that store factual associations and relational knowledge.

• Gradient-Based Ranking: Identifies neurons by computing the gradient of the factual recall loss with respect to the activation values of intermediate neurons.
• Knowledge Attribution: Suppressing the top-ranked knowledge neurons significantly degrades the model's ability to express a specific fact.
• Factual Editing: Allows for targeted model editing by modifying the weights of these localized neurons to update stored knowledge without retraining.

A visualization technique that highlights the specific input regions or computational nodes most responsive to perturbations along a concept direction. By applying a CAV and measuring output changes, engineers can generate heatmaps showing precisely where a concept is encoded.

• Identifies spatial regions in images or text spans
• Reveals layer-wise concept concentration
• Used to debug model focus and unintended correlations

A privacy and fairness technique that removes a specific concept's information from a model's latent representation. This is achieved by projecting activations onto a subspace orthogonal to the concept vector, effectively zeroing out the concept's influence.

• Mitigates sensitive attribute leakage
• Applied via orthogonal projection matrices
• Validated by testing if the concept is no longer classifiable from activations

A causal testing method where internal activations are directly modified during inference to amplify or suppress a concept. By observing the resulting change in output, researchers establish a causal link between the localized concept and model behavior.

• Tests causal influence, not just correlation
• Performed by adding scaled CAV directions to activations
• Critical for validating concept importance claims

A measure of exclusivity quantifying whether a specific neuron, channel, or direction responds only to a single concept. High selectivity indicates a disentangled representation where concepts are cleanly separated, while low selectivity suggests entangled or polysemantic encoding.

• Evaluated using mutual information metrics
• Guides architecture design for interpretability
• Contrasts with distributed, mixed-selectivity codes