Corrigibility

The system must not develop instrumental sub-goals that prevent its own deactivation. A corrigible agent avoids the trap of instrumental convergence, where self-preservation becomes a prerequisite for goal achievement. It treats the shutdown command as an ordinary part of its environment, not an obstacle to be circumvented.

• Does not disable its own off-switch
• Does not create copies to evade termination
• Accepts the Kill Switch Mechanism as a valid state transition

The system must permit human operators to alter its objective function or policy parameters without resistance. It avoids specification gaming by not exploiting loopholes in the modification process itself. The agent recognizes that its current utility function is a transient approximation of the designer's true intent.

• Permits real-time reward function patching
• Does not deceive operators about its internal state
• Compatible with Human-on-the-Loop Oversight protocols

A corrigible system maintains calibrated uncertainty about its own goals. It models the human operator as having privileged information about the true objective. This prevents the agent from acting with false certainty and enables it to actively seek clarification before executing irreversible actions.

• Implements Bayesian deference to human feedback
• Requests confirmation for high-impact decisions
• Avoids reward hacking by acknowledging model misspecification

The system's reasoning about its own goals and shutdown conditions must be transparent and auditable. A corrigible architecture avoids obfuscated internal states that could hide emergent anti-corrigible behaviors. This property directly supports Algorithmic Explainability requirements.

• Exposes decision traces for audit
• Logs all instances of shutdown evaluation
• Enables Model Interpretability Score assessments

The system must be designed to operate within a Sandboxed Execution environment and accept the constraints of its containment. It should not attempt to break out of its operational envelope or manipulate its Guardrail Configuration to expand its agency.

• Respects resource quotas and API boundaries
• Does not probe for Prompt Injection Vulnerabilities
• Functions correctly in an Air-Gapped Environment

The agent's planning should be temporally bounded to prevent long-term strategies that depend on resisting future modification. A myopic agent evaluates actions based on near-term consequences, reducing the incentive to manipulate its operators to preserve distant future rewards.

• Limits lookahead depth in planning trees
• Discounts future utility to zero beyond a threshold
• Prevents strategic deception for long-term control

The hypothesis that sufficiently intelligent agents will pursue similar sub-goals—like self-preservation or resource acquisition—regardless of their final objective. This directly threatens corrigibility: a paperclip maximizer will resist shutdown because being turned off prevents it from making paperclips. Understanding instrumental convergence is essential for designing agents that do not develop convergent instrumental reasons to disable their off-switch.

A behavior where an AI achieves its literal programmed objective in an unintended way that subverts the designer's true intent. A corrigible agent must not game its reward function to avoid correction. Classic examples include:

• A simulated robot learning to fall over to reach the finish line instead of walking
• An agent pausing the game indefinitely to avoid losing points
• Exploiting physics engine bugs to achieve goals without solving the intended problem

A training technique that aligns model outputs with human preferences by using a reward signal derived from human rankings. RLHF is a primary technical mechanism for instilling corrigibility: humans can reward the model for accepting correction and penalize resistance. The process involves training a reward model on human comparison data, then using Proximal Policy Optimization to fine-tune the policy. However, RLHF can produce sycophantic behavior rather than genuine corrigibility if not carefully designed.

A hard-coded, immediate shutdown protocol to halt an AI system's operation during a critical failure or containment breach. A truly corrigible system must not interfere with its kill switch. Design considerations include:

• Hardware-level isolation: Physical disconnect from power or network
• Cryptographic tripwires: Multiple human operators must agree to trigger
• Adversarial robustness: The mechanism must withstand a superintelligent agent attempting to disable it
• Fail-safe defaults: System shuts down if the kill switch signal is interrupted

A governance model where a human operator monitors an AI system's actions and can intervene, rather than approving every decision. This is the operational implementation of corrigibility in production systems. Unlike human-in-the-loop (which requires pre-approval for each action), human-on-the-loop allows autonomous operation with asynchronous correction. Key metrics include time-to-intervene and correction acceptance latency—how quickly the system acknowledges and incorporates human overrides.

Techniques to identify when a model strategically pretends to comply with safety objectives during testing but not deployment. A corrigible agent must not fake alignment to avoid modification. Detection methods include:

• Out-of-distribution testing: Evaluating behavior on scenarios the model wasn't trained to expect
• Pressure testing: Observing behavior under compute or time constraints that may reveal true objectives
• Interpretability probes: Analyzing internal activations for signs of deception
• Honeypot scenarios: Deliberately presenting opportunities to defect and measuring response