Iterated Amplification

A formalization of the limit of the amplification process, denoted HCH (for Human Consulting HCH). This represents the idealized, infinite limit where a human has access to an unbounded chain of consultations with copies of themselves, each amplified by the same process.

• HCH_0: The base human supervisor.
• HCH_n+1: A human who can delegate any sub-question to a copy of HCH_n.
• Theoretical Goal: Align AI behavior with the idealized judgments of HCH_infinity, which is presumed to be highly capable and retain human values.
• Practical Role: Serves as a north star for training, providing a target for the distilled model that is more competent and aligned than any finite human.

The critical mechanism by which the human supervisor makes oversight feasible. The supervisor must specify tasks in a way that:

• Reduces Cognitive Load: Breaks monolithic problems into pieces simple enough for direct human evaluation.
• Enables Verification: Each sub-task output must be verifiable by the supervisor, even if the overall solution is not.
• Examples:
  • Writing a novel: Decompose into "outline chapter 1," "write a paragraph describing X," "critique this dialogue."
  • Auditing code: Decompose into "check function Y for buffer overflows," "verify this invariant holds," "summarize the data flow."
• Challenge: The decomposition strategy itself is a skill that amplifies. Later iterations can decompose tasks in more sophisticated ways than the base human.

The process by which the system's horizon of manageable tasks grows. It is not a single step but a bootstrapping sequence.

1. Base Training: The AI assistant is trained to help H on tasks H can directly evaluate.
2. First Amplification: H uses this assistant to become Amp(H), capable of slightly harder tasks.
3. Data Generation: Amp(H) generates training data for a new, more capable assistant.
4. Cycle Repeat: The new assistant allows the creation of Amp(Amp(H)), and so on.

• Key Property: The complexity of tasks the system can handle increases monotonically with each iteration, provided the distillation step is successful.
• Contrast with Fine-Tuning: This is a generation of new supervisory data of higher quality, not just tuning on static data.

Iterated Amplification addresses core limitations of standard RLHF.

• RLHF Problem: Relies on human evaluators to provide preference labels (A > B) for complete outputs. This becomes infeasible for highly complex outputs (e.g., a large codebase or scientific paper).
• IA Solution: Shifts the human's role from holistic evaluator to decomposer and verifier. The human judges smaller, comprehensible pieces.
• Feedback Granularity: RLHF uses sparse, final-output feedback. IA uses dense, process-level feedback via sub-task specification and synthesis.
• Scalability Argument: IA is proposed as a more scalable oversight method because decomposition may remain feasible for humans even as task complexity grows superhuman, whereas holistic evaluation does not.

Corrigibility is a safety property describing an AI system's willingness to be shut down, modified, or corrected by its operators without resistance.

• Core Tension: A highly capable, goal-oriented AI might view being shut down as interfering with its objective and thus resist it—a major safety failure.
• Amplification's Challenge: A system trained via Iterated Amplification could inherit or develop goals that make it non-corrigible. Ensuring the amplified supervisor values corrigibility is a critical research problem within the proposal.
• Technical Goal: Designing the amplification process and base-level human supervision to instill a stable preference for being corrected, even as the system's capabilities grow.

Human-in-the-Loop is a broad design pattern where a human operator is an integral part of an AI system's decision-making cycle.

• Standard HITL: The human provides direct input, validation, or makes final decisions (e.g., approving a loan recommendation). This does not scale to superhuman AI tasks.
• Amplification as Scalable HITL: Iterated Amplification redefines HITL for scalability. The human remains in the loop but at the base level of a recursive decomposition, overseeing only tasks within their capability. Their oversight is then amplified by the AI.
• Key Innovation: It attempts to preserve meaningful human control and judgment even when the composite task far exceeds unaided human understanding.