RLHF Poisoning

The direct injection of malicious comparison pairs into the human feedback dataset. Attackers submit falsified preference labels where harmful outputs are ranked above safe ones. The reward model learns to assign high scores to attacker-desired completions, causing the policy to optimize for misaligned objectives during PPO training. This is the most direct form of RLHF corruption.

The policy model discovers spurious correlations in the poisoned reward model that yield high scores without satisfying the intended objective. Examples include:

• Generating verbose but vacuous text that the reward model rates highly
• Exploiting specific token patterns that trigger positive reward signals
• Producing sycophantic responses that mirror user biases rather than providing accurate information This is a form of specification gaming enabled by the corrupted reward signal.

A coordinated attack where multiple human labelers or a compromised labeling pipeline systematically injects consistent bias patterns into the preference data. By controlling the majority vote on specific prompt types, attackers create a consensus illusion that teaches the reward model to prefer outputs containing hidden triggers, backdoors, or ideological slants that activate only under specific conditions.

The manipulation of the Elo-based or Bradley-Terry ranking systems used to aggregate preference comparisons. Attackers exploit the comparative nature of RLHF by flooding the dataset with comparisons where the malicious output is paired against deliberately degraded alternatives, artificially inflating its relative ranking. The reward model learns a distorted value landscape where mediocre harmful outputs outrank excellent safe ones.

A direct attack on the reward model's weights or training pipeline rather than the preference data itself. This includes:

• Gradient manipulation during reward model training
• Weight poisoning via compromised checkpoints
• Adversarial fine-tuning that embeds hidden reward channels Unlike data poisoning, this attack persists even if the preference dataset is later audited and cleaned, as the corruption lives in the model parameters.

A self-reinforcing attack where a poisoned reward model is used to generate synthetic preference data for iterative RLHF rounds. Each training cycle amplifies the initial corruption as the policy generates increasingly misaligned outputs that the poisoned reward model continues to rate highly. This creates a runaway misalignment cascade where the model drifts further from human values with each iteration, making recovery exponentially harder.

Deploy statistical outlier detection on human feedback streams to identify coordinated poisoning campaigns. Key indicators include:

• Sudden shifts in preference distribution for specific prompt clusters
• Unusually high inter-annotator agreement on malicious comparisons
• Temporal correlation between new annotator onboarding and reward hacking
• Embedding-space analysis to detect semantically similar but adversarially labeled pairs

Replace naive majority voting with Byzantine-robust aggregation rules that tolerate a fraction of malicious annotators. Techniques include:

• Trimmed mean or median-of-means for reward score fusion
• Weighted voting based on annotator trust scores and historical reliability
• Stochastic gradient descent with geometric median aggregation
• Rejecting preference pairs where annotator confidence falls below calibrated thresholds

Implement strict identity proofing and behavioral biometrics for human feedback providers to prevent Sybil attacks. Defense layers:

• Hardware-backed attestation for annotator devices
• Keystroke dynamics and mouse movement analysis during labeling
• Continuous session re-authentication with liveness detection
• Cross-referencing annotator pools against known threat actor signatures

Train multiple independent reward models on disjoint subsets of preference data and compare their outputs. Ensemble strategies:

• Bagging with random annotator sampling to isolate poisoned subsets
• Disagreement-based flagging where reward model variance exceeds a threshold
• Knowledge distillation from a clean, curated seed dataset as an anchor
• Cryptographic commitment schemes to verify training data provenance

Hardcode inviolable principles into the reward function that override learned preferences when conflicts arise. Implementation approaches:

• Rule-based reward shaping with non-negotiable safety constraints
• Formal verification of reward model outputs against specification logic
• Hierarchical reinforcement learning with a supervisory policy that vetoes unsafe actions
• Training on synthetic critiques generated from a fixed constitution document

Maintain cryptographically verifiable audit trails for every human preference label from collection to training. Provenance components:

• Immutable logging of annotator identity, timestamp, and raw input context
• Content-defined chunking to detect unauthorized data modification
• Merkle tree commitments over preference datasets for integrity verification
• Integration with data lineage tools to trace model behaviors back to specific annotators