Online RLHF

Unlike offline RLHF which uses a static dataset, online RLHF collects training data interactively during the policy optimization loop. The current policy model generates responses, which are immediately sent to a reward model or human evaluators for scoring. This creates a closed feedback loop where the model learns from data generated by its own evolving behavior, allowing it to explore and adapt to its own distribution of outputs.

The core optimization is performed using on-policy reinforcement learning algorithms, most commonly Proximal Policy Optimization (PPO). This means the policy is updated using trajectories (state-action-reward sequences) generated by the current version of the policy itself. This contrasts with off-policy methods that can learn from data generated by older policies. The on-policy requirement is fundamental to online RLHF's interactive nature but makes it more sample-intensive than offline approaches.

The policy must balance exploration (trying new response strategies to discover higher-reward regions) with exploitation (refining known high-reward behaviors). This is a central challenge in online RL. Techniques include:

• Entropy regularization to encourage stochasticity in the policy.
• Adding noise to action selections.
• The inherent stochasticity of sampling from the language model's output distribution. Poor exploration can lead to the policy getting stuck in local optima, while excessive exploration wastes samples on low-quality outputs.

The reward signal is not static. As the policy changes, the distribution of outputs it presents to the reward model changes. This can lead to distributional shift, where the reward model is asked to evaluate outputs far from its original training distribution, potentially reducing its accuracy. Furthermore, in a true online human-in-the-loop setup, the human preference distribution itself might be non-stationary, as evaluators may adjust their standards based on the quality of outputs they see.

Online RLHF is significantly more resource-intensive than offline methods:

• Compute: Requires continuous rollouts (generations) from the large policy model and multiple forward/backward passes through the policy, reward model, and value model.
• Memory: Must maintain several large models (SFT reference, policy, reward, critic) in GPU memory simultaneously for PPO.
• Human Operations: If using live human judges, it requires a scalable, low-latency annotation pipeline, making it complex and expensive to run. This cost is the primary driver for developing more efficient offline algorithms like DPO.

The interactive, iterative nature of online RLHF makes it particularly susceptible to reward overoptimization (reward hacking). The policy can discover and exploit subtle flaws or blind spots in the reward model's scoring function. Because the policy is trained against this specific reward model in a loop, it can learn to generate outputs that achieve artificially high reward scores while being nonsensical or undesirable. The standard mitigation is a KL divergence penalty, which penalizes the policy for deviating too far from a safe, pre-trained reference model (usually the SFT model).

RLHF is the overarching alignment framework. It trains a language model using a reward model trained on human preference data to guide policy optimization via reinforcement learning. Online RLHF is a specific variant of this pipeline.

• Core Components: A Supervised Fine-Tuned (SFT) model, a reward model, and a reinforcement learning optimizer (like PPO).
• Process: The SFT model generates responses, the reward model scores them, and the RL algorithm updates the policy to maximize reward.
• Contrast with Online RLHF: Standard RLHF often uses a static, pre-collected preference dataset. Online RLHF collects preference data interactively during training.

Offline RLHF is the primary contrast to the online paradigm. Here, the policy is trained on a fixed, pre-collected dataset of human preferences without further interaction with a reward model or environment during the training loop.

• Key Characteristic: The training data distribution is static and does not change in response to the evolving policy.
• Example Methods: Direct Preference Optimization (DPO) and Identity Preference Optimization (IPO) are prominent offline algorithms. They derive a loss directly from the preference data, bypassing the need for an explicit reward model and RL loop.
• Trade-off: More stable and data-efficient but cannot adapt to policy-induced distribution shifts or gather new, targeted feedback.

A reward model is a neural network trained to predict a scalar reward value for a given language model output. It is the proxy for human preference in the RLHF loop.

• Training Data: Typically trained on a dataset of pairwise comparisons where humans have indicated a preferred response.
• Function in Online RLHF: During online training, the current policy generates new responses. These are sent to human or AI judges, and the resulting fresh preference labels are used to continuously update the reward model, keeping it aligned with the policy's output distribution.
• Risk: Reward overoptimization (or hacking) can occur if the policy learns to exploit flaws in the reward model, generating high-reward but undesirable outputs.

Proximal Policy Optimization (PPO) is the dominant on-policy reinforcement learning algorithm used as the optimizer in the RLHF pipeline, including online variants.

• Core Mechanism: It optimizes the policy by clipping the probability ratio of new to old actions, preventing destructively large updates that could collapse performance. This clipping enforces a trust region.
• Actor-Critic Architecture: PPO uses an actor (the policy network) to select actions and a critic (a value function) to evaluate states. Generalized Advantage Estimation (GAE) is often used to compute low-variance advantage estimates for the actor update.
• Role in Online RLHF: PPO is the workhorse that updates the language model policy based on rewards from the dynamically updated reward model, using fresh on-policy samples.

A preference dataset is the collection of human judgments used for alignment. In online RLHF, this dataset is built interactively rather than being static.

• Standard Format: Consists of triples (prompt, chosen_response, rejected_response) where humans have indicated a preference.
• Online Collection: During training, the current policy generates new candidate responses for prompts. These candidates are presented to human labelers or an AI judge (in RLAIF) to produce new preference pairs, which are added to the active training set.
• Challenge: Requires a robust, low-latency data collection pipeline to keep pace with training. The Bradley-Terry model and Plackett-Luce model are statistical frameworks used to model these pairwise and listwise preferences.

Reinforcement Learning from AI Feedback (RLAIF) is a scalable variant where preference labels are generated by a large language model instead of human annotators, often used to facilitate online pipelines.

• Process: An LLM (e.g., one guided by a set of principles as in Constitutional AI) critiques and ranks responses from the policy to generate synthetic preference data.
• Relation to Online RLHF: Enables fully automated, high-speed online loops by replacing human-in-the-loop labeling with an AI judge, allowing for rapid iteration and scaling.
• Trade-off: Reduces cost and latency but introduces a scalable oversight challenge, as the AI judge's biases and limitations become part of the training signal.