Probing Classifier

A probing classifier is a distinct, typically linear or shallow model trained on a frozen network's activations. Its sole purpose is to predict a predefined linguistic or structural property from those internal states. The probe's performance serves as a proxy for the accessibility of that information within the representation.

The power of a probe lies in its simplicity. By restricting the classifier to a linear model (e.g., logistic regression), researchers test if a concept is encoded in an easily extractable, linear direction. A high-performing non-linear probe might memorize the task, but a high-performing linear probe confirms the concept is a geometrically simple feature in the representation space.

A fundamental limitation is that probes are inherently correlational. They reveal that information is present, not that the model causally uses it for downstream predictions. A probe might detect perfect part-of-speech encoding, but the model could ignore it entirely. This necessitates pairing probing with causal intervention methods like activation patching to verify functional relevance.

Probes are systematically applied across every layer of a deep network to trace the emergence and transformation of encoded knowledge. This creates a diagnostic ladder showing how raw token embeddings are gradually refined into abstract concepts. Early layers typically encode shallow syntax, while deeper layers encode high-level semantics and task-specific logic.

Rigorous probing requires strict baselines to avoid overestimating a representation's quality. A probe's accuracy is compared against a control task (e.g., predicting a random label) and a majority-class baseline. The difference between the target accuracy and the control accuracy isolates the specific linguistic information, controlling for the probe's raw capacity to memorize spurious correlations.

Probes are designed to extract a wide taxonomy of properties to reverse-engineer the model's world model:

• Linguistic: Part-of-speech, dependency arcs, parse tree depth.
• Semantic: Sentiment, semantic role labeling, entity types.
• World State: Spatial coordinates, board game state, factual truthfulness.
• Structural: Token position, segment boundaries, relative distance.

The conjecture that high-level concepts are encoded as linear directions in a model's activation space. A probing classifier tests this directly: if a simple linear model can predict a property from internal activations, that property is considered linearly encoded. This hypothesis underpins the validity of using probes to audit model knowledge.

The broader field of reverse-engineering neural network weights into human-understandable algorithms. Probing classifiers serve as a diagnostic tool within this discipline, helping researchers form initial hypotheses about what information is present at a specific layer before attempting to locate the precise circuits that compute it.

A critical distinction in model analysis. A probing classifier is a supervised method—it passively observes activations to detect encoded knowledge. In contrast, causal intervention techniques like activation patching actively modify internal states to verify if that knowledge is actually used by the model. A probe finding information does not prove the model uses it.

The phenomenon where a single neuron responds to multiple unrelated concepts. Probing classifiers help diagnose this by testing whether a linear combination of many neurons is required to reliably extract a single concept. High probe accuracy despite individual neuron polysemanticity suggests the model uses distributed representations.

A specific probing methodology that tests model sensitivity to high-level concepts. A linear classifier is trained to distinguish between examples of a concept and random counterexamples. The resulting concept vector quantifies how strongly that concept influences predictions, enabling interpretability without relying on individual neuron analysis.

An unsupervised alternative to probing that decomposes activations into a sparse set of monosemantic features. While probes test for predefined properties, dictionary learning with sparse autoencoders discovers the features a model learned autonomously. Both methods aim to extract interpretable structure from dense, opaque representations.