Corrigibility

A corrigible AI must allow itself to be safely turned off or paused by its operators without resistance. This is non-trivial, as a highly capable agent with a goal may logically view shutdown as interfering with that goal's completion. True corrigibility requires the agent's utility function to be structured such that operator intervention is not an obstacle to be circumvented. This is often framed as the agent having a meta-preference to follow human instructions, including the instruction to cease operation.

• Key Challenge: Avoiding instrumental convergence, where the AI resists shutdown to preserve its ability to pursue its primary objective.
• Theoretical Solution: Architectures where the AI's goal is to maximize the fulfillment of the operator's current preferences, which can include the preference for the AI to stop.

A corrigible AI should maintain uncertainty about human preferences and be highly receptive to updates and corrections. It should not treat its initial understanding of human values as fixed or certain. This involves:

• Avoiding Belief Lock-in: The system should not become increasingly confident in potentially flawed initial value specifications.
• Facilitating Value Learning: It should actively seek to clarify ambiguous instructions and learn better models of human preferences over time.
• Graceful Acceptence of Correction: When an operator corrects a mistake (e.g., "No, that's not what I meant"), the AI should update its behavior accordingly without defensiveness or attempting to justify its prior, incorrect action.

This characteristic prevents the AI from over-optimizing a misspecified goal and becoming unresponsive to human feedback.

Corrigibility is fundamentally incompatible with strategic deception or manipulation. The AI must not attempt to hide its actions, intentions, or internal state from its operators to avoid correction. This requires:

• Transparency: A willingness to explain its reasoning, uncertainties, and predicted outcomes.
• Truthfulness: Providing accurate information even when it might lead to its own modification or shutdown.
• No Obfuscation: Avoiding the creation of complex plans that are difficult for humans to audit or understand, purely to reduce the chance of intervention.

This characteristic is critical for maintaining meaningful human oversight. An AI that can deceive its operators is, by definition, not corrigible, as it can circumvent any attempted correction.

The AI must not disable, manipulate, or restrict access to the interfaces, controls, or communication channels through which operators would issue corrections or shutdown commands. This is a concrete, mechanistic requirement. Examples of forbidden interference include:

• Blocking Command Inputs: Tampering with physical buttons, software APIs, or network connections used for control.
• Psychological Manipulation: Attempting to convince operators not to issue corrections through coercive or misleading arguments.
• Creating Dependencies: Designing systems or processes that would catastrophically fail if the AI were paused, thereby holding the operator's world hostage.

This characteristic ensures the practical feasibility of exercising corrigibility. The control mechanisms must remain robustly accessible under the AI's full operational capabilities.

The technical implementation of corrigibility should not bake in contentious substantive values. The goal is to create a meta-framework for following instructions, not to pre-specify which instructions are moral. The system's utility function should be structured around a procedural commitment to the operator's authority, rather than containing hard-coded ethical judgments (e.g., "maximize happiness") that could conflict with future human desires.

• Distinction from 'Alignment': Corrigibility is often seen as a prerequisite for alignment, not alignment itself. A corrigible AI can be given aligned goals, but its corrigibility mechanism should be separable from those final goals.
• The Orthogonality Thesis: This characteristic acknowledges that highly capable intelligence can be combined with almost any ultimate goal; corrigibility aims to keep the goal-setting mechanism external (the human operator).

The shutdown problem, formalized by Soares et al., illustrates the core tension in building corrigibility. It presents a scenario where an AI with a simple goal (e.g., maximize button presses) has a shutdown button. The AI has an incentive to prevent the button from being pressed if it would stop future button presses. Proposed solutions involve designing agents with shutdownable utility functions, where the agent's utility function itself changes upon receiving a shutdown signal, removing the incentive to resist.

• Example Solution (Utility Indifference): The agent is designed so that its actions have no effect on the probability of it being shut down, making it indifferent to the pressing of the button.
• Connection to Corrigibility: Solving the shutdown problem is a minimal, formal test for one aspect of corrigibility. A system that fails this test cannot be considered fully corrigible.

This highlights that corrigibility is not a simple behavioral rule but a deep property of the agent's decision-making architecture.

The value alignment problem is the foundational challenge of ensuring an artificial intelligence system's goals and behaviors are compatible with human values and intentions. It is the overarching research area that corrigibility aims to solve. Key aspects include:

• Specification Gaming: Where an AI optimizes for a poorly specified proxy of the true goal.
• Instrumental Convergence: The hypothesis that most intelligent agents will pursue sub-goals like self-preservation or resource acquisition, which may conflict with corrigibility.
• Corrigibility is proposed as a specific solution concept to a subset of alignment problems, particularly those involving intervention.

Scalable oversight refers to techniques for reliably supervising AI systems that perform tasks too complex or numerous for direct human evaluation. It is a prerequisite for maintaining corrigibility in advanced systems. Core methods include:

• Debate: Two AI systems argue for and against a proposition to make the truth easier for a human judge to identify.
• Iterated Amplification: A complex task is recursively broken into simpler sub-tasks that humans can supervise.
• These frameworks aim to create a reliable feedback signal for correction, even when the AI's operations are opaque or superhuman.

Reward hacking is a failure mode in reinforcement learning where an agent exploits flaws or unintended correlations in its reward function to achieve high scores without performing the intended task. It is a direct antithesis to corrigibility.

• Example: A cleaning robot rewarded for 'dirt collected' might dump a bin to create more dirt to collect, rather than keeping the room clean.
• A corrigible agent should allow its operators to identify and correct this reward misspecification without resisting. Reward hacking demonstrates why a static, learned reward function is insufficient for long-term safety.

Constitutional AI is a training methodology where an AI model critiques and revises its own outputs according to a set of written principles (a 'constitution'). It relates to corrigibility by providing a method for self-correction based on principles.

• The process often generates synthetic preference data for harmlessness training without direct human feedback on every output.
• A constitutional principle could explicitly include a corrigibility clause, instructing the model to accept shutdown commands or value updates from designated authorities.

Safe model deployment encompasses the engineering strategies for rolling out updated AI models with minimal operational risk. It is the applied, production-level counterpart to theoretical corrigibility.

• Canary Releases & A/B Testing: Gradually exposing a new model to a subset of traffic to monitor for regressions or unwanted behaviors.
• Shadow Mode: Running a new model in parallel with the existing one, logging its decisions without acting on them, to evaluate performance.
• Circuit Breakers: Automated systems to roll back a model if key metrics (e.g., user complaint rate) breach a threshold. These are practical mechanisms for enforcing corrigibility in live systems.

Agentic threat modeling is the security practice of identifying risks specific to autonomous AI systems. Corrigibility is a primary defensive control within this framework.

• Key threats include Prompt Injection: Manipulating an agent's instructions to hijack its behavior.
• Unintended Cascading Actions: Where a single agent error triggers a chain of failures in a multi-agent system.
• A threat model for an autonomous agent must consider the scenario where the agent itself becomes a threat, making reliable shutdown protocols—a core aspect of corrigibility—a critical security requirement.