Conformalized Reinforcement Learning

The foundational training paradigm where an agent learns a policy exclusively from a static, pre-collected dataset of transitions, without any further interaction with the environment. This setting is critical for high-stakes domains like healthcare and autonomous driving where exploration is dangerous or expensive.

• Key Challenge: Distributional shift between the learned policy and the behavior policy that collected the data.
• Conformal Connection: Conformal methods provide the uncertainty quantification needed to detect out-of-distribution state-action pairs, preventing the agent from executing unsafe actions where the Q-value estimate is unreliable.

A distribution-free, model-agnostic framework that wraps any prediction algorithm to produce prediction sets with a rigorous finite-sample marginal coverage guarantee. Instead of a single point estimate, it outputs a set of plausible labels that contains the true label with a user-specified probability.

• Core Mechanism: Uses a held-out calibration set to compute an empirical quantile of a nonconformity score.
• RL Application: Calibrates the raw uncertainty estimates from an ensemble of Q-functions to construct valid confidence intervals around value predictions.

An extension of conformal prediction that provides statistical guarantees for controlling any monotone loss function, not just miscoverage. This allows practitioners to bound more complex, task-specific error metrics.

• Mechanism: Adjusts a parameter lambda to ensure the expected loss on a test point does not exceed a predefined threshold.
• Safety Application: In RL, this can be used to construct action sets that formally bound the probability of a catastrophic failure or a constraint violation, moving beyond simple value uncertainty to direct risk management.

A prominent offline RL algorithm that learns a lower bound of the true Q-function by modifying the standard Bellman update to penalize actions that are out-of-distribution. This prevents the overestimation bias that plagues standard Q-learning on static datasets.

• Pessimism Principle: CQL is pessimistic in the face of uncertainty, avoiding actions with high estimated value but low data support.
• Conformal Synergy: Conformalized RL can be seen as a complementary, statistically rigorous approach to pessimism. While CQL applies a heuristic penalty, conformal methods provide a distribution-free guarantee on the safety of the action set.

An approach where the agent first learns a world model—a predictive model of the environment's dynamics and reward function—from the static dataset, and then uses this model to simulate rollouts for policy optimization.

• Uncertainty Compounding: Errors in the learned dynamics model can compound over multi-step rollouts, leading to exploitation of model inaccuracies.
• Conformal Integration: Conformal prediction can be applied to the dynamics model's outputs to generate calibrated prediction intervals for future states. The agent can then be penalized for entering states with high epistemic uncertainty, creating a robust pessimistic MDP.

A critical decomposition of predictive uncertainty in machine learning models.

• Aleatoric Uncertainty: The inherent, irreducible noise in the data-generating process (e.g., sensor noise, stochastic rewards).
• Epistemic Uncertainty: The model's ignorance due to limited data, which is reducible by collecting more training samples in a region of the state-action space.
• Conformalized RL Focus: Conformal methods are primarily used to quantify epistemic uncertainty in offline RL. By identifying state-action pairs with high epistemic uncertainty, the agent can avoid regions where its knowledge is insufficient to guarantee safe performance.