Controlled Generation

Activation engineering involves reading and modifying the intermediate activations (vector representations) within a neural network's layers during inference. By identifying steering vectors—directions in activation space that correlate with specific concepts—engineers can add or subtract these vectors to amplify or suppress attributes like sentiment, formality, or topic.

• Example: Adding a 'positive sentiment' vector to hidden states makes the model's output more optimistic.
• Key Benefit: Provides real-time, granular control without changing the model's underlying weights.

Constrained decoding restricts the model's token-by-token generation process to enforce hard or soft constraints, ensuring the output adheres to specific lexical, grammatical, or structural rules.

• Hard Constraints: Force the model to include specific keywords, follow a predefined JSON schema, or avoid banned terms by manipulating the output logits or search space.
• Soft Constraints: Use guided decoding algorithms like PPLM (Plug and Play Language Models) or FUDGE to bias the probability distribution toward desired attributes.
• Use Case: Guaranteeing API call outputs are valid JSON or preventing the generation of profanity.

This technique uses carefully engineered system prompts and in-context examples to establish a latent 'context steering vector' within the model's forward pass. The model's attention mechanism focuses on these instructions, creating an internal representation that biases subsequent generation.

• Instruction Embedding: The model creates an internal representation of the prompt's intent, which acts as a continuous control signal.
• Dynamic Few-Shot Learning: Providing examples in-context directly shapes the model's output distribution for the task.
• Limitation: Less precise than direct activation manipulation and vulnerable to prompt injection.

Classifier guidance uses an auxiliary model—a classifier or discriminator—to evaluate and score partial or complete generations against a target attribute. This score is then used to adjust the main model's generation path via gradient signals or reward-weighted sampling.

• Process: During decoding, the classifier provides feedback (e.g., 'how positive is this text?'), and this signal backpropagates to influence subsequent token probabilities.
• Application: Commonly used in diffusion models for image generation and adapted for text to control style, sentiment, or factual grounding.
• Trade-off: Introduces computational overhead due to the need for multiple forward/backward passes.

Representation Fine-Tuning methods, such as Low-Rank Adaptation (LoRA) or IA3, introduce small, trainable parameters into a frozen pre-trained model. While often used for training, the adapted weight matrices or activation scaling factors serve as persistent, parameter-efficient control knobs that are engaged during inference.

• Mechanism: A LoRA adapter trained to increase factual accuracy will modify forward-pass computations whenever it's loaded, steering generation.
• Advantage over Activation Engineering: The control is baked into a reusable module, offering consistent steering without per-request vector arithmetic.
• Hybrid Use: Often combined with prompt-based techniques for layered control.

These are specialized algorithms that operate during the decoding loop to guide generation:

• PPLM (Plug and Play Language Models): Uses a attribute classifier to compute gradients with respect to the model's past hidden states, updating them to increase the probability of a desired attribute.
• FUDGE (Controlled Text Generation with Future Discriminators): Employs a future discriminator that predicts if a sequence will satisfy a constraint, using this prediction to adjust token probabilities at each step.
• DExperts: Combines a base model with 'expert' and 'anti-expert' language models (fine-tuned for and against an attribute) via ensemble decoding to interpolate between behaviors.

These methods offer a formal, algorithmic approach to controlled generation.

Low-dimensional vectors derived from a model's internal activations (hidden states) that, when added during inference, can steer the model's outputs toward or away from specific attributes. This is a core controlled generation method.

• Mechanism: Computed by contrasting activations from prompts with and without a target attribute (e.g., 'positive sentiment' vs. 'negative sentiment').
• Non-Invasive: Requires no model weight updates, offering a lightweight, dynamic control knob.
• Use Case: Adjusting tone, formality, creativity, or safety attributes in real-time.

A system of automated filters and refusal mechanisms that enforce a set of ethical or safety principles on model outputs. While controlled generation steers outputs, guardrails often block or rewrite them.

• Implementation: Can use safety classifiers, keyword blocklists, or secondary LLM scrutiny.
• Layer: Typically applied as a post-processing or middleware layer, separate from the core generation step.
• Relation: Guardrails can be seen as a coarse, rule-based form of output control, whereas steering vectors offer finer-grained, attribute-level control.

A fine-tuning or model-editing technique designed to remove specific dangerous knowledge or behavioral tendencies from a model's weights. It aims for persistent, global removal of a concept.

• Method: Techniques like Rank-One Model Editing (ROME) make localized, surgical changes to a model's feed-forward layers.
• Goal: To prevent the model from ever generating content related to the erased concept (e.g., detailed hacking instructions).
• Difference from Control: Erasure attempts to delete a capability; controlled generation dynamically suppresses or amplifies it on a per-request basis.