Glossary

Representation Engineering

A top-down approach to controlling model behavior by directly manipulating high-level representations in the residual stream using steering vectors rather than modifying individual weights.

Get in touch Learn more

ML engineer managing model versions on laptop, version history visible, technical Git-like workflow.

TOP-DOWN MODEL CONTROL

What is Representation Engineering?

A technique for controlling high-level model behavior by directly manipulating semantic representations in the residual stream using steering vectors, rather than modifying individual weights.

Representation Engineering is a top-down safety and control paradigm that reads and writes to a model's high-level cognitive representations in the residual stream to steer behavior. Instead of retraining weights, it identifies a direction in activation space corresponding to a concept (e.g., honesty, refusal) and adds a scaled steering vector during the forward pass to modulate the output.

This technique contrasts with mechanistic interpretability's bottom-up circuit analysis by operating directly on the model's emergent internal representations. By applying a contrastive prompt pair to extract a function vector, practitioners can control abstract properties like sycophancy or harmfulness without degrading general capability, offering a computationally cheap alternative to fine-tuning for alignment.

TOP-DOWN CONTROL

Key Features of Representation Engineering

A paradigm for controlling high-level model behavior by directly manipulating semantic vectors in the residual stream, bypassing the need for retraining or fine-tuning individual weights.

Steering Vectors

The core mechanism of representation engineering. A steering vector is a direction in the model's activation space that corresponds to a specific high-level concept or behavior (e.g., honesty, sycophancy, or a specific sentiment). By adding or subtracting this vector to the residual stream activations during a forward pass, the model's output distribution is shifted towards or away from the target concept. This is a causal intervention that directly modifies the model's internal representations without altering its weights.

Weight-Free

Modification Method

Contrastive Activation Addition

A primary technique for extracting steering vectors. The process involves collecting the model's internal activations on pairs of contrasting prompts—for example, a prompt eliciting a target behavior versus a prompt eliciting its opposite behavior. The steering vector is computed as the mean difference between these two sets of activations at a specific layer. This isolates the directional representation of the target concept in the model's latent space.

Contrastive Pairs

Extraction Method

Residual Stream Intervention

Representation engineering operates by injecting the steering vector directly into the residual stream, the primary information highway of a transformer. This is significant because the residual stream accumulates information from all previous layers. An intervention at a single layer can therefore influence all subsequent computations, including attention and MLP layers, providing a powerful and efficient lever for controlling global model behavior from a single, top-down control point.

Single Point

Intervention Locus

Behavioral Control Spectrum

This technique enables control over a wide spectrum of high-level behaviors that are difficult to prompt reliably:

Honesty & Sycophancy: Steer models away from agreeing with the user's incorrect beliefs.
Power-Seeking: Reduce the model's tendency to pursue resource-acquiring strategies.
Toxicity & Harmfulness: Suppress the generation of toxic content without a complex prompt.
Sentiment & Style: Consistently control the emotional tone or formality of generated text.

Comparison to Fine-Tuning

Unlike Parameter-Efficient Fine-Tuning (PEFT) or RLHF, representation engineering does not require any gradient-based training or a curated preference dataset. It is an inference-time intervention that operates on the model's frozen weights. This makes it extremely compute-efficient and allows for dynamic, real-time control over model behavior. A single base model can be steered in multiple directions simultaneously or sequentially without the risk of catastrophic forgetting or the overhead of maintaining multiple fine-tuned model variants.

Zero Gradients

Training Required

Concept Erasure via Projection

Beyond addition, representation engineering can be used to erase a concept from the model's representations. This is achieved by projecting the residual stream activations onto a subspace orthogonal to the steering vector. By mathematically removing the component of the activation vector that aligns with the target concept, the model's ability to represent or generate content related to that concept is suppressed, providing a more robust form of behavioral control than simple negative prompting.

REPRESENTATION ENGINEERING

Frequently Asked Questions

Clear answers to common questions about directly manipulating high-level model representations to control behavior without modifying individual weights.

Representation Engineering (RepE) is a top-down safety and control paradigm that directly manipulates the high-level features encoded in a model's residual stream using steering vectors, rather than modifying the model's weights through gradient-based fine-tuning. Unlike fine-tuning, which retrains parameters and risks catastrophic forgetting, RepE identifies a direction in the model's activation space corresponding to a specific concept—such as honesty, harmfulness, or power-seeking—and adds or subtracts that direction during inference. This approach treats the model's internal representations as a controllable interface, allowing operators to dial specific behaviors up or down without altering the underlying weights. The key distinction is that fine-tuning changes what the model is, while representation engineering changes what the model expresses at runtime, offering a more surgical, reversible, and computationally efficient method for alignment and behavioral control.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

MECHANISTIC INTERPRETABILITY

Related Terms

Representation engineering relies on a deep understanding of a model's internal structure. These related concepts form the foundational toolkit for decoding and manipulating the residual stream.

Residual Stream

The primary information highway in a transformer model. Each layer reads from and writes to this shared, accumulating state, allowing the model to bypass layers and preserve information flow. Representation engineering directly targets this stream by adding steering vectors to its activations, treating it as a manipulable latent space rather than a black box.

d_model

Dimensionality

Sparse Autoencoder (SAE)

An unsupervised technique that decomposes a model's dense, polysemantic internal activations into a sparse set of interpretable, monosemantic features using a learned overcomplete basis. SAEs are critical for identifying the specific feature directions within the residual stream that representation engineers target with steering vectors.

Overcomplete

Basis Property

Activation Patching

A causal intervention technique that replaces a model's internal activation at a specific layer and token position with a cached activation from a different forward pass. This isolates functional circuits and validates the causal effect of a representation. It is the primary experimental method for verifying that a steering vector derived from one context generalizes to another.

Causal

Intervention Type

Function Vector

A compact representation derived from in-context learning examples. When added to the model's activations, it can steer the model to perform a specific task without the original prompt. This concept is a direct precursor to representation engineering, demonstrating that high-level task instructions can be compressed into a single vector in the residual stream.

ICL

Derived From

Logit Lens

A direct probing method that applies the unembedding matrix to intermediate residual stream states. This interprets the model's next-token predictions before the final layer norm is applied. Representation engineers use this technique to decode the semantic content of a steering vector and verify that the manipulation is shifting the output distribution toward the desired behavior.

Early Decoding

Technique

Causal Mediation Analysis

A statistical framework adapted for neural networks to quantify the contribution of a specific intermediate variable to a model's output. It measures the indirect effect through a mediator. This provides the mathematical rigor for proving that a specific direction in the residual stream is necessary and sufficient for a behavior, justifying its selection as a control point.

Total Effect

Measures

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.