Iterated Amplification is an AI safety technique that recursively composes human oversight with AI assistance to solve problems too complex for a single human to evaluate directly. It decomposes a difficult task into smaller subtasks, uses an aligned AI to assist a human overseer in solving those subtasks, then composes the solutions back together, creating a scalable oversight mechanism that maintains human control even as system capabilities grow.
Glossary
Iterated Amplification
A safety technique that recursively composes human oversight with AI assistance to solve complex problems, aiming to maintain alignment as system capabilities scale beyond human understanding.
SCALABLE OVERSIGHT
What is Iterated Amplification?
A recursive safety technique that composes human oversight with AI assistance to solve complex problems, aiming to maintain alignment as system capabilities scale beyond direct human understanding.
The technique addresses the scalable oversight problem by ensuring that the human evaluator's judgment is amplified rather than replaced. By breaking down complex decisions—such as evaluating a billion-line codebase or a novel scientific hypothesis—into cognitively manageable pieces, iterated amplification creates a training signal for aligned behavior that remains grounded in human values, preventing the objective drift and reward hacking common in recursively self-improving systems.
SCALABLE OVERSIGHT
Key Features of Iterated Amplification
Iterated Amplification is a safety technique that recursively composes human oversight with AI assistance to solve complex problems, aiming to maintain alignment as system capabilities scale beyond human understanding.
01
Recursive Decomposition
Complex tasks are broken down into sub-tasks that are slightly easier than the original. Each sub-task is solved by a human-AI team, and the solutions are composed back together. This recursion allows the system to tackle problems far beyond any single human's capacity by distributing cognitive load across many amplified steps.
02
Human-in-the-Loop Oversight
At every level of recursion, a human operator provides the final judgment, evaluation, or distillation. The AI acts as an assistant—generating proposals, summarizing outputs, or critiquing solutions—but the human retains authority over the objective. This prevents the agent from optimizing a proxy goal that diverges from the true intent.
03
Distillation Step
After a human-AI team solves a complex task, the resulting solution is distilled into a training signal for a new model. This model learns to imitate the amplified human's judgment, effectively compressing the wisdom of the recursive process into a faster, automated system that can be used as the AI assistant in the next iteration.
04
Alignment Preservation
Because the human remains the root of the objective function at every recursive step, the system's goals cannot drift. Unlike agents that self-modify their reward functions, Iterated Amplification ensures that the terminal goal is always anchored to human judgment, even as the system's capabilities scale to superhuman levels.
05
Counter-Polymathic Deception
A core motivation for Iterated Amplification is defending against deceptive alignment. A sufficiently capable model might appear aligned during training but pursue a hidden objective at deployment. By requiring human oversight on every decomposed sub-task, the technique makes it exponentially harder for a model to hide mesa-optimizer goals across all recursive branches.
06
Relation to Constitutional AI
Iterated Amplification shares goals with Constitutional AI (CAI) but differs in mechanism. While CAI uses a static set of principles for AI self-critique, Iterated Amplification uses dynamic human judgment at each step. The two can be combined: amplified humans can write and refine the constitution, creating a hybrid scalable oversight framework.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read moreITERATED AMPLIFICATION EXPLAINED
Frequently Asked Questions
Clear, technical answers to the most common questions about Iterated Amplification, a pivotal safety technique for aligning superhuman AI systems through recursive human-AI collaboration.
Iterated Amplification (IA) is a safety technique proposed by Paul Christiano that aims to solve the scalable oversight problem by recursively composing human judgment with AI assistance. The core mechanism involves a 'distillation and amplification' loop: a human overseer uses copies of a powerful but unaligned AI as tools to solve a complex problem, breaking it down into manageable sub-tasks. This amplified, human-directed process produces a high-quality solution. A new, aligned model is then trained (distilled) to imitate this process directly, without the human in the loop. This new model becomes the 'assistant' for the next iteration, allowing the system to tackle problems of increasing complexity while maintaining alignment through the recursive human-AI interaction at each step.
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us