Linear Probing

The core principle of linear probing is that the base model's weights are completely frozen during training. The model acts purely as a static feature extractor. Only the weights of the newly attached linear probe are updated. This ensures the probing process does not alter the representations it aims to study, providing a clean, non-invasive diagnostic signal about the information content already present in the network.

The power of a linear probe is its simplicity and limited capacity. A linear classifier can only succeed if the target concept is linearly separable in the model's representation space. If a linear probe achieves high accuracy on a task like part-of-speech tagging, it proves the model has not just learned the concept but has organized it in an easily accessible, geometrically simple format. This makes it a strict test of representation quality.

Probes are typically trained on the output of a specific layer to map the transformational journey of information through the network. By comparing probe accuracy at different depths:

• Lower layers often encode surface-level syntax.
• Intermediate layers capture semantic roles and entity types.
• Higher layers represent task-specific, abstract knowledge. This gradient of information reveals how a model progressively builds its understanding.

Linear probing is a diagnostic tool, not an adaptation method. It is distinct from parameter-efficient fine-tuning (PEFT) techniques like LoRA.

• Probing Goal: To answer 'What does the model know?' without changing it.
• Fine-Tuning Goal: To adapt the model for a new downstream task by updating weights. A probe's high performance reveals intrinsic knowledge; fine-tuning's high performance demonstrates adaptability.

A key limitation of linear probing is the Superposition Hypothesis. Models may represent more features than they have dimensions by encoding them in overlapping, non-orthogonal directions. A linear probe might fail to detect a feature not because it's absent, but because it's stored in a non-linear, entangled format. This has motivated the use of more powerful non-linear probes and sparse autoencoders to recover these hidden features.

To ensure a probe's accuracy reflects genuine encoding and not just the probe's own computational power, researchers use control tasks. A common method is to train a probe on randomly shuffled labels. If the probe achieves high accuracy on the real task but not on the control task, it validates that the model's representations contain the specific linguistic or world knowledge being tested, rather than the probe simply memorizing the dataset.

A foundational linguistic probe that tests whether a layer's embeddings cluster words by their grammatical category. A linear classifier is trained on frozen representations to predict tags like noun, verb, or adjective. High accuracy at early layers indicates the model has learned morphosyntactic structure without explicit supervision, validating that the architecture builds a latent grammar.

A structural probe designed to test whether a model implicitly encodes syntax trees. The probe predicts the distance between tokens in a dependency parse or the depth of each word in the tree directly from the model's embeddings. A squared linear transformation is often used to recover tree geometry, confirming that the model learns hierarchical syntactic structure rather than just surface-level statistics.

A semantic probe that tests whether a model's representations distinguish between classes of real-world entities. A linear classifier is trained to label tokens as persons, organizations, locations, or miscellaneous entities. Strong performance indicates that factual knowledge about entity boundaries and types is encoded linearly in the residual stream, making it accessible for downstream extraction.

A high-level semantic probe that tests whether a model's sentence-level representations capture affective meaning. A linear classifier is trained on the pooled embeddings of a frozen model to predict positive, negative, or neutral sentiment. This reveals whether the model builds a concept direction for emotional valence that can be linearly separated, a finding foundational to representation engineering.

A probe that tests whether a model's feed-forward layers act as a key-value memory for world facts. A linear classifier is trained to predict the object of a relation (e.g., 'capital of France') from the hidden state at the final token of the subject. This diagnostic identifies specific knowledge neurons and layers where factual associations are stored, enabling causal tracing of recall.

A discourse-level probe that tests whether a model tracks entity identity across long contexts. A linear classifier is trained to predict which previous mention a pronoun refers to from the embeddings of both spans. Success indicates the model maintains a dynamic entity representation that persists through the residual stream, solving a critical binding problem for coherent text understanding.

The core data pathway in a transformer architecture where each layer reads from and writes additive updates to a running hidden state. It is the primary object of linear probing analysis, as researchers extract activations from this stream at specific layers to diagnose what information is available at that point.

• Additive updates: Each attention and MLP block writes its output back into the stream
• Layer-wise probing: Probing the residual stream after each layer reveals how representations are built iteratively
• Direct path: The residual stream creates a "skip connection" that allows early information to flow directly to later layers