Intrinsic Motivation

Curiosity is implemented as an intrinsic reward proportional to the prediction error of a learned forward dynamics model. The agent is rewarded for visiting states where its model of the environment is poor, encouraging exploration of novel or unpredictable regions of the state space.

• Key Formulation: Often calculated as the error in predicting the next state's features: r_intrinsic = η * ||φ(s_{t+1}) - φ̂(s_{t+1})||^2, where φ is a state embedding.
• Purpose: Mitigates the sparse reward problem by providing a dense learning signal in environments with infrequent external rewards.
• Example: In a maze with a single goal reward, a curious agent will systematically explore dead-ends and corridors to reduce its prediction error, eventually discovering the goal.

This mechanism incentivizes visiting states based on their visitation frequency, directly rewarding the agent for going to less-visited states. It formalizes the intuition of exploring "uncharted territory."

• Core Mechanism: The intrinsic reward is inversely proportional to a state visitation count: r_intrinsic = β / √(N(s)), where N(s) is the count.
• Challenge: Requires defining a discrete state space or using density models or hash-based discretization (like SimHash) for continuous spaces.
• Utility: Highly effective in discrete, tabular environments and provides strong theoretical guarantees for visiting all states.

RND is a prominent method for generating curiosity that does not require learning an accurate forward model. It uses two neural networks: a fixed, randomly initialized target network and a trainable predictor network.

• Process: The predictor is trained to output the target network's embedding for a given state. The intrinsic reward is the prediction error between the two networks' outputs.
• Advantage: The error is high for novel states (where the predictor is untrained) and low for familiar states, providing a robust novelty signal. It is less prone to being distracted by stochastic or "noisy TV" environments than forward dynamics curiosity.
• Application: Key component in agents that mastered Montezuma's Revenge and other hard-exploration Atari games.

These are information-theoretic intrinsic objectives. Information Gain rewards the agent for reducing uncertainty about the environment's dynamics or its own knowledge. Empowerment formalizes an agent's control over its environment.

• Information Gain: Measured as the reduction in entropy of a learned model's parameters after observing a transition. The agent seeks learning progress.
• Empowerment: Defined as the channel capacity between the agent's actions and future states. The agent is motivated to reach states from which it can cause many possible future outcomes, maximizing its potential influence.
• Characteristic: These are more computationally intensive but provide a principled, theoretically grounded drive for exploration and skill acquisition.

Here, intrinsic motivation is used to generate and pursue a curriculum of self-proposed goals, often in the absence of an external task. The agent learns a policy to reach any state (goal) within a learned goal space.

• Mechanism: The agent samples a goal g from a goal distribution, attempts to reach it, and receives an intrinsic reward for success (e.g., a sparse +1).
• Algorithms: Hindsight Experience Replay (HER) is a critical enabling technique, allowing the agent to relabel failed trajectories with achieved goals as targets, making learning efficient.
• Outcome: The agent autonomously builds a repertoire of skills (a goal-conditioned policy) that can later be directed by an external reward signal.

In practice, intrinsic rewards are combined with extrinsic (task) rewards to train a single policy. The core challenge is balancing the two signals to prevent the intrinsic drive from dominating and leading to task-agnostic behavior.

• Standard Formulation: The total reward is r_total = r_extrinsic + β * r_intrinsic, where β is a hyperparameter scaling the intrinsic bonus.
• Dynamic Balancing: Advanced methods may use non-stationary weighting or intrinsic reward normalization to adapt the balance throughout training.
• Primary Benefit: This hybrid approach enables efficient exploration in vast state spaces while ensuring the policy ultimately converges to maximize the external objective, solving complex, sparse-reward problems.