Intrinsic Motivation

Random Network Distillation (RND) is a curiosity-driven exploration method. It uses two neural networks: a fixed, randomly initialized target network and a predictor network trained to mimic the target's outputs. The intrinsic reward is the prediction error—states where the predictor fails to match the target are novel and yield high reward. This provides a robust, unsupervised signal for exploration in sparse-reward environments, such as procedurally generated video games.

The Intrinsic Curiosity Module (ICM), proposed by Pathak et al., generates curiosity based on an agent's ability to predict the consequences of its own actions. It consists of:

• A forward dynamics model that predicts the next state given the current state and action.
• An inverse dynamics model that predicts the action taken between two states. The intrinsic reward is the error of the forward model—high for unpredictable outcomes. This focuses exploration on learnable aspects of the environment, ignoring unpredictable noise.

Count-based exploration methods incentivize visiting infrequently seen states. The core idea is to maintain pseudo-counts for states (or state features) and reward the agent inversely proportional to these counts. Modern implementations like PixelCNN or Hash-based counting approximate counts in high-dimensional spaces. This directly implements the principle of information gain, driving the agent to reduce uncertainty about the environment's dynamics and reward structure.

VIME (Variational Information Maximizing Exploration) frames exploration as maximizing information gain about the agent's internal model parameters. It treats the environment's dynamics model as a Bayesian neural network. The intrinsic reward is the Kullback-Leibler (KL) divergence between the posterior distribution over model parameters before and after taking an action. This encourages actions that lead to significant updates in the agent's belief about how the world works, promoting deep exploration.

While not purely intrinsic, Goal-Conditioned RL often employs intrinsic motivation for goal discovery. The agent is trained to reach self-generated goals. Algorithms like GoalGAN or Skew-Fit use an intrinsic objective to propose goals at the boundary of the agent's current capability (e.g., goals just beyond its reach). This leads to automatic curriculum learning and the acquisition of a diverse repertoire of skills without external task specification.

Unsupervised Skill Discovery algorithms, such as DIAYN (Diversity is All You Need), use information-theoretic objectives to learn a set of distinguishable skills without extrinsic rewards. The objective maximizes the mutual information between states visited and a latent skill variable. This encourages the agent to discover skills that:

• Lead to distinct states (diversity).
• Are reliably identifiable from the states they produce (discernability). This creates a foundation of reusable primitive behaviors for downstream tasks.

Random Network Distillation (RND) is a specific intrinsic motivation algorithm that rewards an agent for visiting states where the predictions of a randomly initialized, fixed 'target' neural network are hard for a trainable 'predictor' network to mimic. The prediction error serves as a novelty signal, encouraging exploration of unfamiliar states without requiring a complex generative world model. It is a cornerstone method for exploration in sparse-reward environments like Montezuma's Revenge.

• Mechanism: Two neural networks receive the current state as input. The target network's output is a fixed random projection. The predictor network is trained to match this output. High prediction error equals high intrinsic reward.
• Key Property: Provides a pseudo-count of state visits without explicitly counting, making it scalable to high-dimensional observation spaces.

Model-Based Reinforcement Learning (MBRL) is a paradigm where an agent learns an explicit model of the environment's dynamics (a world model) and uses it for planning or policy optimization. Intrinsic motivation is often integrated into MBRL to guide the collection of data for model learning, ensuring the dynamics model is accurate in regions critical for future tasks.

• Contrast with Model-Free: Unlike model-free RL which learns a policy or value function directly from experience, MBRL uses the learned model as a simulator for internal planning (e.g., via Monte Carlo Tree Search) or to generate synthetic experience.
• Connection to Intrinsic Motivation: Curiosity-driven exploration (a form of intrinsic motivation) is used to reduce model uncertainty. Agents are driven to explore states where the dynamics model makes poor predictions, leading to more robust world models and improved sample efficiency.

A world model is a learned or engineered internal representation of an environment that enables an agent to predict future states and outcomes without direct interaction. It serves as a simulator inside the agent's mind. Intrinsic motivation mechanisms often use the world model's prediction errors as a curiosity signal or aim to improve the model's coverage.

• Function: Compresses high-dimensional sensory history (e.g., pixels) into a compact latent state from which future rewards, values, and observations can be predicted.
• Architectures: Common designs include Recurrent State-Space Models (RSSM), used in the Dreamer agent, which combine deterministic RNNs with stochastic latent variables.
• Exploration Link: An agent driven by epistemic curiosity seeks to visit states that reduce uncertainty in its world model, making exploration and model learning synergistic processes.

State representation is the process of encoding the current configuration of an environment into a compact, informative format sufficient for decision-making. Intrinsic motivation objectives can be designed to learn disentangled or object-centric state representations that are more amenable to exploration and generalization.

• Goal: Transform raw observations (pixels, lidar) into a Markovian state where the future is conditionally independent of the past given the present state.
• Learning via Intrinsic Goals: Methods like bisimulation metrics or the information bottleneck can be used with intrinsic rewards to learn representations that capture causally relevant features while ignoring distractors.
• Types: Includes disentangled representations (independent factors of variation), object-centric representations (sets of entity slots), and causal state representations (only causally relevant variables).

The options framework is a formalism for temporal abstraction in reinforcement learning. An option is a temporally extended action, defined by an initiation set, an internal policy, and a termination condition. Intrinsic motivation can be used to discover useful options (skills) without extrinsic reward, a process known as skill discovery.

• Structure: An option executes a sub-policy over multiple timesteps until termination, abstracting away low-level control sequences.
• Intrinsic Skill Discovery: Algorithms like DIAYN (Diversity is All You Need) use an information-theoretic intrinsic reward to learn a repertoire of distinguishable skills (options) that maximize the mutual information between states and a latent skill variable.
• Benefit: Discovered skills provide a library of reusable behaviors that can be sequenced by a higher-level planner to solve complex tasks more efficiently.

A Partially Observable Markov Decision Process (POMDP) is the formal model for sequential decision-making when an agent cannot directly observe the true state of the environment. Intrinsic motivation in POMDPs often drives exploration to improve the agent's belief state—its internal probability distribution over possible true states.

• Core Challenge: The agent must maintain a belief state based on its history of actions and observations. This makes exploration more complex than in fully observable MDPs.
• Intrinsic Drives for POMDPs: Curiosity can be directed towards reducing belief state entropy or visiting novel belief states, not just novel observations. This encourages the agent to take actions that reveal hidden information about the world.
• Application: Critical for real-world robotics and embodied AI where sensors provide only partial and noisy information about the environment.