Intrinsic Motivation

This mechanism motivates an agent to seek states where it has high potential influence over its future. It is grounded in information theory, specifically the empowerment of an agent in a given state.

• Definition: Empowerment is the channel capacity between the agent's actions and its future states. It measures the maximum mutual information I(A_t; S_{t+k} | S_t).
• Goal: The agent seeks states where its actions have the most diverse and predictable consequences, leading to robust, generalizable skill acquisition rather than random wandering.

Here, intrinsic motivation is framed as the ability to reach a wide diversity of self-generated goals. The agent learns a universal policy π(a | s, g) conditioned on a goal g.

• Goal Generation: Goals are sampled from a goal space (e.g., achieved states from a replay buffer). The intrinsic reward is given for reaching a goal state.
• Hindsight Experience Replay (HER): A key technique that relabels failed trajectories with goals that were actually achieved, turning failures into useful learning experiences. This dramatically improves sample efficiency for sparse rewards.

The Diversity is All You Need (DIAYN) framework formalizes intrinsic motivation for discovering distinguishable skills without an external reward.

• Mechanism: The agent learns a set of skills (policies) distinguished by a discriminator that tries to guess which skill generated a given state trajectory.

• Objective: Maximize the mutual information I(S; Z) between states (S) and skill latent variables (Z). This encourages skills to visit distinct, recognizable regions of the state space, leading to the emergence of useful primitive behaviors.

A simple yet powerful method where novelty is defined as the error in predicting the output of a fixed, randomly initialized neural network (the target network).

• Process: A second, trainable network (the predictor) is trained to mimic the target network's outputs for observed states. The intrinsic reward is the predictor's mean squared error.

• Advantage: The target network provides a stable, procedurally generated novelty signal. States where the predictor has high error are novel because the predictor hasn't learned to map them to the target's random function.