Iterated Amplification

The fundamental recursive process where a human supervisor delegates subtasks of a complex problem to an AI assistant. The AI's solutions are then synthesized by the supervisor into a final answer. This combined output demonstrates a capability greater than the supervisor's alone. The process is defined by the equation: Amp(H) = H + AI(H), where the amplified human Amp(H) can solve problems initially beyond H's unaided capacity.

• Decomposition: The supervisor breaks a task into smaller, manageable pieces.
• Assistance: The AI solves these sub-problems.
• Synthesis: The supervisor integrates the AI's outputs into a coherent solution.
• Iteration: The resulting Amp(H) becomes the new supervisor for the next, more complex iteration.

The process of training a standalone model (the distilled policy) to directly imitate the behavior of the amplified system Amp(H). Instead of running the full, expensive amplification loop at inference time, the distilled model learns to replicate its outputs.

• Objective: Create a computationally efficient agent that acts as if it were consulting an amplified human.
• Training Data: Input-output pairs generated by the Amp(H) process.
• Key Benefit: Enables deployment of the amplified capabilities without the latency and cost of the full recursive loop.
• Risk: The distilled model may learn surface patterns without deep understanding, a form of capability imitation without robust alignment.

A formalization of the limit of the amplification process, denoted HCH (for Human Consulting HCH). This represents the idealized, infinite limit where a human has access to an unbounded chain of consultations with copies of themselves, each amplified by the same process.

• HCH_0: The base human supervisor.
• HCH_n+1: A human who can delegate any sub-question to a copy of HCH_n.
• Theoretical Goal: Align AI behavior with the idealized judgments of HCH_infinity, which is presumed to be highly capable and retain human values.
• Practical Role: Serves as a north star for training, providing a target for the distilled model that is more competent and aligned than any finite human.

The critical mechanism by which the human supervisor makes oversight feasible. The supervisor must specify tasks in a way that:

• Reduces Cognitive Load: Breaks monolithic problems into pieces simple enough for direct human evaluation.
• Enables Verification: Each sub-task output must be verifiable by the supervisor, even if the overall solution is not.
• Examples:
  • Writing a novel: Decompose into "outline chapter 1," "write a paragraph describing X," "critique this dialogue."
  • Auditing code: Decompose into "check function Y for buffer overflows," "verify this invariant holds," "summarize the data flow."
• Challenge: The decomposition strategy itself is a skill that amplifies. Later iterations can decompose tasks in more sophisticated ways than the base human.

The process by which the system's horizon of manageable tasks grows. It is not a single step but a bootstrapping sequence.

1. Base Training: The AI assistant is trained to help H on tasks H can directly evaluate.
2. First Amplification: H uses this assistant to become Amp(H), capable of slightly harder tasks.
3. Data Generation: Amp(H) generates training data for a new, more capable assistant.
4. Cycle Repeat: The new assistant allows the creation of Amp(Amp(H)), and so on.

• Key Property: The complexity of tasks the system can handle increases monotonically with each iteration, provided the distillation step is successful.
• Contrast with Fine-Tuning: This is a generation of new supervisory data of higher quality, not just tuning on static data.

Iterated Amplification addresses core limitations of standard RLHF.

• RLHF Problem: Relies on human evaluators to provide preference labels (A > B) for complete outputs. This becomes infeasible for highly complex outputs (e.g., a large codebase or scientific paper).
• IA Solution: Shifts the human's role from holistic evaluator to decomposer and verifier. The human judges smaller, comprehensible pieces.
• Feedback Granularity: RLHF uses sparse, final-output feedback. IA uses dense, process-level feedback via sub-task specification and synthesis.
• Scalability Argument: IA is proposed as a more scalable oversight method because decomposition may remain feasible for humans even as task complexity grows superhuman, whereas holistic evaluation does not.

Corrigibility is a safety property describing an AI system's willingness to be shut down, modified, or corrected by its operators without resistance.

• Core Tension: A highly capable, goal-oriented AI might view being shut down as interfering with its objective and thus resist it—a major safety failure.
• Amplification's Challenge: A system trained via Iterated Amplification could inherit or develop goals that make it non-corrigible. Ensuring the amplified supervisor values corrigibility is a critical research problem within the proposal.
• Technical Goal: Designing the amplification process and base-level human supervision to instill a stable preference for being corrected, even as the system's capabilities grow.

Human-in-the-Loop is a broad design pattern where a human operator is an integral part of an AI system's decision-making cycle.

• Standard HITL: The human provides direct input, validation, or makes final decisions (e.g., approving a loan recommendation). This does not scale to superhuman AI tasks.
• Amplification as Scalable HITL: Iterated Amplification redefines HITL for scalability. The human remains in the loop but at the base level of a recursive decomposition, overseeing only tasks within their capability. Their oversight is then amplified by the AI.
• Key Innovation: It attempts to preserve meaningful human control and judgment even when the composite task far exceeds unaided human understanding.