Intrinsic Motivation

This mechanism uses prediction error as an intrinsic reward. The agent is rewarded for visiting states where its internal model of the environment makes poor predictions. A common implementation is:

• Intrinsic Curiosity Module (ICM): Uses two neural networks—an inverse dynamics model to learn a feature space and a forward dynamics model to predict the next state's features. The error in this prediction becomes the intrinsic reward.
• This drives the agent towards novel or hard-to-predict states, ensuring exploration even when extrinsic rewards are absent.

This formulation directly incentivizes visiting infrequently visited states. The core idea is to approximate state visitation counts in large or continuous spaces.

• Pseudo-Counts: Methods like CTS density model or Hash-based counting (e.g., SimHash) approximate how novel a state is.
• The intrinsic reward is inversely proportional to the pseudo-count: r_intrinsic = 1 / sqrt(N(s)), where N(s) is the visit count.
• This provides a strong, simple signal to systematically cover the state space, preventing the agent from getting stuck.

These are information-theoretic approaches that reward the agent for gaining knowledge about the environment or increasing its control potential.

• Information Gain: Rewards the agent for reducing uncertainty in its model parameters (e.g., Bayesian Neural Network weights). The reward is the KL-divergence between the prior and posterior model distributions.
• Empowerment: Measures the channel capacity between an agent's actions and future states. The agent is intrinsically motivated to reach states from which it has the most potential influence over its future, maximizing option diversity.

A highly scalable method for generating intrinsic reward based on the error in predicting the output of a fixed random neural network.

• Mechanism: Two networks are used—a fixed target network with random weights and a predictor network trained to mimic the target's outputs. The predictor's mean squared error on a given state's features is the intrinsic reward.
• States seen frequently become easy to predict (low error, low reward). Novel states produce high prediction error and high reward.
• Key advantage: It requires no dynamics model and is computationally efficient, making it suitable for complex environments like Montezuma's Revenge.

Here, intrinsic motivation is tied to achieving self-generated goals, often in the absence of an external task specification.

• Algorithmic Framework: The agent learns a goal-conditioned policy. Intrinsic reward is given for reaching a self-sampled goal state.
• Goal Generation Strategies: Goals can be random states, states where skill diversity is maximized, or states that are reachable but novel.
• This approach is foundational for learning repertoires of skills (skill discovery) that can later be harnessed for specific downstream tasks.

Most intrinsic motivation methods can be expressed as adding a pseudo-reward r_i(s, a, s') to the extrinsic reward r_e(s, a, s').

• Total Reward: r_total = r_e + β * r_i, where β is a scaling coefficient.
• Optimization Objective: The agent's policy is trained to maximize the expected sum of these combined rewards.
• Key Challenge: Balancing β is critical. Too high, and the agent ignores the real task; too low, and exploration fails. Annealing β over time or using non-stationary intrinsic rewards are common solutions to prevent the agent from "chasing novelty" indefinitely.

This method encourages an agent to visit states it has rarely or never seen before. A pseudo-reward is inversely proportional to the visitation count of a state.

• Core Mechanism: The agent maintains a count N(s) for each state s. The intrinsic reward is defined as r_intrinsic = 1 / sqrt(N(s)) or similar.
• Effect: The agent is driven away from frequently visited, familiar areas of the state space.
• Example: In a maze, the agent receives a small bonus for entering a new room, accelerating the mapping process without any external goal reward.

Here, intrinsic reward is generated based on an agent's inability to predict the consequences of its actions. High prediction error signifies novelty or learnability.

• Core Mechanism: The agent learns an internal forward dynamics model f that predicts the next state s' given (s, a). The intrinsic reward is the error || s' - f(s, a) ||^2.
• Key Insight: This drives the agent toward states where its model is poor and improving the model reduces uncertainty.
• Application: Used in DeepMind's ICM (Intrinsic Curiosity Module) to explore complex 3D environments in video games where external rewards are absent.

This advanced approach frames intrinsic motivation as the drive to maximize information or future influence. The agent seeks states where it can learn the most or have the most control.

• Information Gain: The agent is rewarded for visiting states that cause the largest update to its internal world model (high Bayesian surprise).
• Empowerment: An information-theoretic measure quantifying how much influence an agent has over its future. The agent seeks states that maximize its potential to affect future states.
• Use Case: Enables the emergence of complex skills like tool use, as the agent learns actions that increase its long-term control over the environment.

Intrinsic motivation is used to autonomously discover and practice a repertoire of useful skills without a predefined task.

• Process: The agent randomly selects a goal state (e.g., a robot gripper position) and receives an intrinsic reward for achieving it. Over time, it learns a library of goal-conditioned policies.
• Benefit: This creates a foundation of reusable skills (a "skill library") that can be rapidly composed to solve new, externally specified tasks via hierarchical RL.
• Real-World Impact: Allows robots to learn fundamental manipulation skills like pushing, grasping, and rotating objects in a self-supervised manner before any high-level instruction is given.

The primary engineering application of intrinsic motivation is to solve tasks where extrinsic rewards are given only upon ultimate success or are extremely rare.

• Problem: In robotics, a reward might be +1 for inserting a peg, and 0 otherwise. Random exploration will almost never stumble upon the successful state.
• Solution: Intrinsic rewards provide a continuous learning signal, guiding the agent through the state space toward regions where the sparse extrinsic reward might eventually be found.
• Result: Dramatically improves sample efficiency and makes learning feasible in complex physical domains like robotic assembly and navigation.

Intrinsic motivation is a powerful tool for training robust policies in simulation that transfer effectively to physical robots.

• Mechanism: By encouraging extensive exploration of the simulated state space, the agent encounters a wider variety of scenarios (different object textures, lighting, physics parameters). This acts as an implicit form of domain randomization.
• Outcome: The resulting policy is less brittle and more adaptable to the inevitable discrepancies between simulation and reality (the "reality gap").
• Example: A robot trained with curiosity-driven exploration in a varied simulation is better at manipulating real-world objects with different friction properties it has never explicitly seen before.

The fundamental dilemma in reinforcement learning where an agent must balance trying new actions to discover their effects (exploration) with choosing actions known to yield high reward (exploitation). Intrinsic motivation is a primary strategy for driving systematic exploration, especially when extrinsic rewards are absent. In robotics, this tradeoff is critical: excessive exploitation can trap a robot in suboptimal behaviors, while blind exploration can be inefficient or unsafe.

The engineering process of designing a reward function by adding intermediate or auxiliary rewards to guide an agent's learning. While intrinsic motivation generates rewards internally based on the agent's experience (e.g., novelty), reward shaping involves externally defining supplemental rewards to mitigate challenges like sparse primary rewards. For example, giving a small reward for moving closer to a goal object is reward shaping, while rewarding the agent for visiting a new part of the state space is intrinsic motivation.

A prominent class of intrinsic motivation methods where the reward signal is based on an agent's prediction error or learning progress. The agent is motivated to explore states where its internal world model makes poor predictions. Key implementations include:

• Prediction Error: Reward is the error in predicting the next state or the features of the next state.
• Learning Progress: Reward is the rate of decrease in prediction error for a state. This approach is highly effective for robotic skill acquisition in open-ended environments without predefined goals.

An exploration strategy where an agent is intrinsically motivated to visit states it has encountered less frequently. A pseudo-count is maintained for states or state features, and the intrinsic reward is inversely proportional to this count (e.g., 1/√(N(s))). This provides a mathematically grounded form of novelty-seeking. In high-dimensional state spaces common in robotics (e.g., images), counts are approximated using density models or hash-based discretization to make the approach feasible.

A framework that augments the standard RL objective to maximize both expected reward and policy entropy. Algorithms like Soft Actor-Critic (SAC) explicitly encourage the policy to be stochastic, promoting diverse action selection and robust exploration. While not purely intrinsic motivation, the entropy term acts as an inherent, information-theoretic drive for exploration. This leads to policies that are more robust and better at capturing multi-modal solutions, which is valuable for robotics where multiple successful action sequences may exist.

A framework where the policy and value functions are conditioned on a goal, often represented as a desired state. Intrinsic motivation can be used to generate a curriculum of self-proposed goals. The agent might be intrinsically rewarded for proposing and achieving goals that are at an appropriate level of difficulty (e.g., just beyond its current capabilities). This creates an automatic skill-acquisition loop, allowing a robot to autonomously build a repertoire of useful behaviors without external task specification.