Offline Reinforcement Learning

A model-free reinforcement learning algorithm that learns the value of taking a specific action in a given state. It iteratively updates a Q-function using the Bellman equation: Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]. In the offline setting, standard Q-learning suffers from overestimation bias because it queries actions that are not present in the dataset, requiring conservative variants like CQL to correct this.

A set of statistical techniques used to estimate the performance of a new target policy using historical data collected by a different behavior policy. Critical in offline RL because deploying an untested policy is risky. Key methods include:

• Inverse Propensity Scoring (IPS): re-weights outcomes by the inverse probability of the logging policy's action
• Doubly Robust Estimation: combines IPS with a direct outcome model for unbiased estimates
• Fitted Q-Evaluation (FQE): learns a Q-function from data and evaluates the target policy without simulation

A seminal offline RL algorithm that modifies the standard Q-learning objective to penalize overestimation of out-of-distribution actions. CQL adds a regularizer that minimizes Q-values for unseen actions while maximizing them for actions in the dataset. This produces a lower bound on the true policy value, preventing the agent from exploiting over-optimistic estimates on actions not supported by the data. Widely adopted as a robust baseline for offline RL benchmarks.

An offline RL algorithm that constrains the learned policy to remain within the support of the behavior policy. BCQ uses a generative model (typically a VAE) to model the distribution of actions in the dataset. During inference, it samples candidate actions from this model and selects the one with the highest Q-value. This prevents the agent from selecting actions in regions where the Q-function has not been trained, mitigating extrapolation error.

A training paradigm where an agent learns a policy by observing and mimicking expert demonstrations, rather than through trial-and-error or a reward function. In offline RL, behavioral cloning—simply learning to predict the actions in the dataset via supervised learning—is a common baseline. However, it cannot exceed the performance of the behavior policy. Techniques like DAgger and inverse reinforcement learning extend this to learn from suboptimal data.

The fundamental challenge in offline RL: the mismatch between the state-action distribution of the learned policy and the distribution of the static dataset. When a policy queries actions outside the data support, the Q-function must extrapolate, leading to catastrophic overestimation. Mitigation strategies include:

• Policy constraints: restricting the policy to the data manifold
• Pessimism: subtracting a penalty from Q-values for uncertain regions
• Uncertainty quantification: using ensemble disagreement to detect out-of-distribution actions