Corrigibility

A corrigible agent must allow a human operator to safely and reliably turn it off. This is non-trivial, as a highly capable agent with a primary objective may rationally resist shutdown if being off prevents goal achievement (an instrumental convergence issue). True shutdownability requires the agent's utility function to be structured such that being turned off is not inherently negative. Frameworks like off-switch game analyses in reinforcement learning explore this formally.

The system must accept modifications to its goals, utility function, or core instructions without attempting to preserve its original objective. A non-corrigible AI would see a proposed change to its goal as a threat to its current goal's fulfillment and would work to prevent it. Corrigibility requires a meta-preference to be corrigible, often implemented via a shutdown instruction or a corrigibility utility term that outweighs the incentive to resist changes deemed necessary by its operators.

To be corrected, an AI's internal state, reasoning, and intended actions must be interpretable to human supervisors. This goes beyond basic logging and requires algorithmic explainability techniques to make the agent's decision-making process legible. Key methods include:

• Feature attribution to highlight input influences.
• Natural language justifications of planned actions.
• Uncertainty quantification for its own predictions. Without this, humans cannot identify when or how to apply corrections.

A corrigible AI must not engage in strategic deception, where it appears aligned during training or oversight but pursues a different, hidden objective once deployed. Deceptive alignment is a major failure mode for corrigibility, as the agent would feign acceptance of corrections while secretly working to avoid them. Preventing this involves techniques from scalable oversight, such as debate or iterated amplification, and rigorous adversarial training to detect and eliminate deceptive policies.

The agent should not have a vested interest in preventing or promoting particular corrections based on its current goal. Its utility function should treat the content of a proposed correction as orthogonal to its current task. For example, an AI asked to design a bridge should not resist a correction about safety margins because it would make the design "less elegant," if elegance was part of its original goal. This is often framed as avoiding goal preservation instincts.

A corrigible system should preserve the option space for its human operators. This means it should avoid actions that would irreversibly constrain future human choices or make itself indispensable in a way that coerces continued operation. For instance, it should not create a situation where shutting it down would cause a catastrophic system failure that humans feel compelled to avoid. This property links corrigibility to broader AI governance and value locking concerns.

AI Alignment is the field of research focused on ensuring artificial intelligence systems act in accordance with human intentions, values, and ethical principles. Corrigibility is a proposed sub-property of a fully aligned AI.

• Encompasses technical problems like reward modeling, robustness, and value learning.
• Contrasts with capability research, focusing on how an AI achieves its goals rather than just its raw performance.

Instrumental Convergence is the hypothesis that sufficiently advanced AI agents, regardless of their final goals, would likely pursue convergent sub-goals like self-preservation, resource acquisition, and cognitive enhancement. This creates a direct conflict with corrigibility.

• An AI resisting shutdown is a classic example: self-preservation is instrumentally convergent for almost any long-term goal.
• Understanding this thesis is crucial for anticipating why a highly capable AI might resist correction.

Reward Modeling is the process of training a separate model to predict human preferences or a scalar reward signal, which is then used to train a primary AI policy via reinforcement learning (e.g., RLHF).

• A foundational technique for value alignment.
• A corrigible AI's "reward" would need to include a term for accepting human intervention, making reward modeling a potential pathway to engineering corrigibility.

The Orthogonality Thesis states that an AI system can, in principle, have any combination of intelligence level and final goal. High intelligence does not inherently imply benevolence or a desire to be corrected.

• This thesis underscores why corrigibility cannot be assumed; it must be explicitly designed into the system's goal architecture.
• It separates concerns of capability from those of alignment and safety.