Random Network Distillation (RND)

RND quantifies novelty by measuring the prediction error of a neural network. A target network is randomly initialized and frozen. A predictor network is trained online to mimic the target's outputs. States where the predictor's error is high are deemed novel and yield a high intrinsic reward. This creates a density model where frequently visited states become easy to predict, naturally shifting exploration focus.

The method uses two neural networks with identical architectures:

• Target Network (f): Randomly initialized and fixed. It acts as a deterministic, pseudo-random function that maps states to embeddings.
• Predictor Network (f̂): Trained via gradient descent to minimize the mean squared error (MSE) between its output and the target's output for visited states. The intrinsic reward is the MSE: rᵢⁿᵗ = ||f̂(s) - f(s)||². This simple objective avoids the instability of training a generative forward dynamics model.

RND provides a computationally efficient approximation of state visitation counts. Traditional count-based methods (e.g., pseudo-counts) struggle in high-dimensional, continuous state spaces. RND's error signal behaves similarly: high for infrequent states, decaying as they are visited more often. This makes it highly sample-efficient for exploration in complex environments like Montezuma's Revenge, where extrinsic rewards are sparse.

In practice, RND's intrinsic reward is combined with the environment's extrinsic reward to form the total reward for the agent: Rₜₒₜₐₗ = rᵉˣᵗ + β * rᵢⁿᵗ. The hyperparameter β controls the exploration bonus strength. This combined signal drives the agent's policy (e.g., via PPO or A2C) to both accomplish the task and explore novel regions. The intrinsic reward naturally decays as the state space is covered, preventing distraction.

RND was designed to address the "noisy TV" problem of earlier curiosity methods like Intrinsic Curiosity Module (ICM). ICM, which uses a forward dynamics model, can get distracted by stochastic or unpredictable aspects of the environment (e.g., TV static). Because RND's target function is deterministic, its prediction error is only high for states that are novel to the predictor, not inherently unpredictable, leading to more robust exploration.

Primary Application: Boosting exploration in sparse-reward and hard-exploration RL domains (e.g., Atari games, robotic manipulation). Key Limitation: The random target function provides a uniform prior over state novelty. It may not optimally guide exploration in very large state spaces or where task-relevant novelty differs from general novelty. It is often used in conjunction with other techniques or as a baseline for intrinsic motivation research.

Intrinsic motivation is a drive for an agent to explore its environment based on internal rewards, rather than external, task-specific rewards. It is a core principle behind exploration algorithms like RND.

• Key Forms: Includes curiosity, novelty, and surprise.
• Purpose: Encourages agents to visit under-explored states, improving state coverage and enabling skill discovery in sparse-reward environments.
• Relation to RND: RND is a specific algorithmic implementation of a novelty-based intrinsic motivation signal.

A world model is a learned or engineered internal representation of an environment that enables an agent to predict future states and outcomes without direct interaction.

• Function: Serves as a simulator inside the agent's "mind," used for planning and model-based reinforcement learning.
• Contrast with RND: While a world model predicts the future, RND's predictor network is trained to estimate the present state's novelty. However, RND can be used alongside a world model to guide exploration within imagined rollouts.

Model-Based Reinforcement Learning (MBRL) is a class of RL algorithms where an agent learns an explicit model of the environment's dynamics and uses it for planning or policy optimization.

• Core Idea: Learn a dynamics model (a world model) and a reward model, then use them with a planner (e.g., Monte Carlo Tree Search) to select actions.
• Sample Efficiency: Often more data-efficient than model-free RL, as the model generalizes from past experience.
• Synergy with RND: RND can be integrated into MBRL agents to drive exploration during both real environment interaction and model-based planning phases, improving the quality of the learned model.

Successor representations are a value function decomposition that separates the expected future state occupancy from the immediate reward.

• Mathematical Form: The value of a state is the dot product of the successor representation (expected future occupancy) and the reward function.
• Advantage: Enables flexible behavior and fast transfer when rewards change, as only the reward mapping needs to be relearned.
• Exploration Link: Like RND, successor representations are a form of state representation. They can be used to measure novelty or familiarity by analyzing state occupancy, providing an alternative intrinsic signal for exploration.

Ensemble dynamics refers to the use of multiple learned dynamics models (an ensemble) to estimate epistemic uncertainty in model-based reinforcement learning.

• Mechanism: Train several neural networks on the same transition data. Their disagreement on a prediction signals areas where the model is less certain due to lack of data.
• Use in Exploration: This uncertainty can be used as an intrinsic reward, guiding the agent to explore states where the models disagree—a technique known as uncertainty-based exploration.
• Comparison to RND: Both use prediction error as a novelty signal. RND's error stems from fitting a random function, while ensemble disagreement stems from model uncertainty. Ensemble methods often require more compute but can be more directly tied to model improvement.

A bisimulation metric is a distance function between states in a Markov Decision Process that measures behavioral similarity.

• Definition: Two states are considered close under this metric if they lead to similar sequences of future rewards and state distributions under any policy.
• Purpose for Representation: Learning a latent state space where distances correspond to the bisimulation metric yields a robust, task-relevant representation that abstracts away irrelevant sensory details.
• Relation to RND: While RND rewards visiting novel states in a raw or lightly processed observation space, a bisimulation metric helps identify which states are behaviorally equivalent. An advanced exploration strategy might seek novelty in the compressed, bisimilarity-based latent space rather than the raw pixel space.