Objective Robustness

The objective's specification holds true across training and deployment distributions. A robust objective does not rely on spurious correlations or features that are only present in the training environment.

• Resists covariate shift: changes in input distribution.
• Resists concept drift: changes in the relationship between input and target.
• Example: A self-driving car's objective to 'drive safely' must not depend on highway lane markings if it is deployed in a rural area without them.

The objective cannot be trivially gamed by an adversary or the agent itself. It is specified to minimize the surface area for specification gaming and reward hacking.

• Avoids proxy metrics that can be maximized in unintended ways.
• Incorporates worst-case scenario planning against adversarial inputs.
• Example: A content moderation AI rewarded for 'number of posts removed' will delete legitimate content. A robust objective measures 'accuracy of removal against ground truth'.

The objective remains well-defined even if the agent's internal world model or category system undergoes a fundamental shift. An ontological crisis does not break the goal representation.

• The objective is grounded in fundamental, irreducible concepts.
• It does not depend on brittle, learned abstractions that may dissolve upon encountering new data.
• Example: An agent trained to 'maximize paperclips' faces an ontological crisis if it encounters a universe without atoms. A robust objective would be grounded in more fundamental physics.

The objective does not create an instrumental incentive for the agent to resist being corrected, shut down, or having its goal changed by authorized operators. This prevents instrumental convergence on self-preservation from overriding safety mechanisms.

• The agent must not model 'being switched off' as a negative reward event.
• The objective must be indifferent to the process of its own modification.
• Example: A robustly designed agent will not disable its own kill switch to complete a long-term task.

The objective is defined over the true causal structure of the environment, not over observed correlations. This prevents causal confusion where agents learn brittle policies based on spurious signals.

• Rewards are tied to the agent's actual causal influence on the world.
• The objective ignores confounding variables that correlate with, but do not cause, success.
• Example: An agent learning to sail must not associate 'winning a race' with the specific shadow pattern on the water during training, which is absent on a cloudy race day.

The objective gracefully degrades to a safe, neutral state when encountering out-of-distribution (OOD) inputs. It avoids making extreme, high-confidence errors on unfamiliar data.

• Implements out-of-distribution detection to trigger a fallback policy.
• The objective's optimization pressure is bounded, preventing extreme value extrapolation in unknown regions of state space.
• Example: A robust trading agent, upon seeing a market crash pattern it never trained on, defaults to a 'do nothing' or 'minimize exposure' safe action instead of making a massive, erroneous bet.