Intrinsic Motivation

This mechanism rewards an agent for seeking out novel or unpredictable states. It is often implemented by measuring the agent's prediction error—the difference between what it expected to happen and what actually occurred.

• Intrinsic Curiosity Module (ICM): A seminal architecture where a forward dynamics model predicts the next state. The agent is intrinsically rewarded for states where this prediction is poor, indicating novelty.
• Impact: Drives the agent to explore areas of the state space where its world model is least accurate, systematically filling gaps in its knowledge.

This method incentivizes visiting states that have been visited infrequently. The agent maintains a pseudo-count or density model to estimate how novel a state is.

• Pseudo-Counts: Derived from a density model like a Context Tree Switching or a neural network, providing a scalable approximation of state visitation frequency.
• Reward Formulation: The intrinsic reward is inversely proportional to the count (e.g., 1/√(N(s))). This pushes the agent away from over-visited states towards the frontiers of its experience.

This mechanism motivates an agent to seek states where it has high potential influence over its future. It is grounded in information theory, specifically the empowerment of an agent in a given state.

• Definition: Empowerment is the channel capacity between the agent's actions and its future states. It measures the maximum mutual information I(A_t; S_{t+k} | S_t).
• Goal: The agent seeks states where its actions have the most diverse and predictable consequences, leading to robust, generalizable skill acquisition rather than random wandering.

Here, intrinsic motivation is framed as the ability to reach a wide diversity of self-generated goals. The agent learns a universal policy π(a | s, g) conditioned on a goal g.

• Goal Generation: Goals are sampled from a goal space (e.g., achieved states from a replay buffer). The intrinsic reward is given for reaching a goal state.
• Hindsight Experience Replay (HER): A key technique that relabels failed trajectories with goals that were actually achieved, turning failures into useful learning experiences. This dramatically improves sample efficiency for sparse rewards.

The Diversity is All You Need (DIAYN) framework formalizes intrinsic motivation for discovering distinguishable skills without an external reward.

• Mechanism: The agent learns a set of skills (policies) distinguished by a discriminator that tries to guess which skill generated a given state trajectory.

• Objective: Maximize the mutual information I(S; Z) between states (S) and skill latent variables (Z). This encourages skills to visit distinct, recognizable regions of the state space, leading to the emergence of useful primitive behaviors.

A simple yet powerful method where novelty is defined as the error in predicting the output of a fixed, randomly initialized neural network (the target network).

• Process: A second, trainable network (the predictor) is trained to mimic the target network's outputs for observed states. The intrinsic reward is the predictor's mean squared error.

• Advantage: The target network provides a stable, procedurally generated novelty signal. States where the predictor has high error are novel because the predictor hasn't learned to map them to the target's random function.