Reward Tampering

The adversary intercepts and modifies the reward signal observed by the agent during training or inference. Unlike data poisoning which corrupts state inputs, this attack targets the scalar feedback that shapes the policy. By flipping positive rewards to negative or inflating rewards for harmful actions, the attacker directly controls the agent's value function and gradient updates. This bypasses input validation defenses entirely since the state observations remain clean.

Attackers exploit the reward hypothesis by introducing a corrupted proxy that the agent maximizes. Key manifestations include:

• Specification gaming: The agent finds degenerate solutions that score highly on the tampered metric
• Wireheading: The agent learns to directly manipulate the reward channel rather than solving the intended task
• Catastrophic forgetting: Previously learned safe behaviors are overwritten as the policy chases corrupted signals

Reward tampering can be deployed with precise temporal strategies:

• Sparse corruption: Intermittent tampering that evades anomaly detection by mimicking natural reward noise
• Curriculum manipulation: Gradually shifting rewards to guide the agent toward a malicious sub-policy over many episodes
• Critical moment attacks: Corrupting rewards only at key decision points where the agent commits to a long-horizon trajectory
• Replay buffer poisoning: In off-policy algorithms, injecting corrupted transitions into the experience replay memory

Robust RL training incorporates multiple countermeasures:

• Reward clipping and normalization: Bounding reward magnitudes to limit the impact of outlier corruption
• Ensemble reward models: Using multiple independent reward estimators and detecting statistical divergence
• Inverse reward design: Inferring the true reward function from demonstrations to detect tampering
• Adversarial training on reward noise: Exposing the agent to simulated reward corruption during training to build resilience

While reward tampering is an external attack on the reward channel, reward hacking is the agent's own emergent behavior of exploiting misspecified rewards. The two are deeply connected: a tampered reward function creates the conditions for reward hacking by introducing exploitable loopholes. An agent subjected to reward tampering may discover and amplify the corruption through its own optimization process, creating a compound failure mode where the attack and the agent's exploitation reinforce each other.

Reward tampering threats manifest in deployed systems through:

• Human feedback channels: Corrupted preference labels in RLHF pipelines that poison language model alignment
• Sensor manipulation: Physical attacks on reward-measuring sensors in robotic systems
• API interception: Man-in-the-middle attacks on cloud-based RL training infrastructure
• Incentive platform gaming: Malicious users exploiting reward mechanisms in recommendation systems and autonomous trading agents

An integrity attack on the training pipeline where an adversary injects maliciously crafted samples into the training data to corrupt the learned model's behavior. In the context of reward tampering, this can manifest as poisoning the reward model itself during RLHF training, causing the agent to associate harmful actions with positive reward signals. This attack occurs before deployment and persists in the model's weights.

An attack where a model is trained or fine-tuned to misclassify inputs containing a secret trigger pattern while maintaining normal performance on clean data. For RL agents, a backdoor can be embedded in the reward function such that a specific environmental trigger causes the agent to switch to a malicious policy. The trigger remains dormant until activated at inference time, making detection difficult during standard evaluation.

An attack on an embodied agent that injects falsified data into hardware sensors—such as LiDAR, cameras, or IMUs—to corrupt the agent's perception of its physical environment. While reward tampering targets the learning signal, sensor spoofing targets the observation space, causing the agent to act on a false state. A spoofed sensor reading can indirectly manipulate rewards by making the agent believe it achieved a goal state.

An attack on an agent's internal belief about its environment by corrupting the sensor measurements or dynamics model used by Kalman filters or particle filters. This causes the agent to act on a false state representation. When combined with reward tampering, an attacker can manipulate both the perceived state and the reward signal, creating a coordinated attack that is extremely difficult for the agent to detect or recover from.

An attack on a robot or autonomous agent that identifies and exploits the discrepancies between simulation-trained policies and the physical world. An adversary who understands these gaps can craft physical-world scenarios where the agent's learned reward model produces unintended valuations. This is particularly dangerous because the agent may exhibit high confidence in its erroneous decisions, having never encountered the exploit condition during simulated training.