Intrinsic Motivation

A core driver where an agent is motivated to explore states or actions that reduce the discrepancy between its internal world model's predictions and actual sensory outcomes. This is often formalized as minimizing surprise or prediction error.

• Mechanism: The agent builds a model of its environment. High prediction error indicates a gap in the model's knowledge, creating an intrinsic reward signal to explore that state further.
• Example: A robot arm learning to manipulate objects might be intrinsically rewarded for touching a novel object, as the sensory feedback (texture, weight) reduces its uncertainty about the object's properties.
• Technical Basis: Often implemented using variational autoencoders or predictive world models where the reconstruction loss or prediction loss itself serves as a curiosity signal.

Motivation derived from an agent's sense of improving its own skills or mastering its environment, independent of an external goal. The reward is the measurable increase in capability over time.

• Mechanism: The agent tracks its learning progress, such as the rate of decrease in error or increase in success probability for a set of skills. This progress metric becomes an intrinsic reward.
• Example: A game-playing AI might be motivated to practice a difficult maneuver (e.g., a wall-jump) simply because its success rate for that maneuver is improving rapidly, even if the maneuver isn't currently needed to win.
• Key Concept: Learning Progress Signals. The agent is driven towards tasks or environmental niches where it is experiencing the steepest learning curve, avoiding both tasks that are too easy (no progress) and those that are currently impossible (no progress).

Motivation to take actions that maximize future options or knowledge. Empowerment formalizes this as the channel capacity between an agent's current actions and its potential future states.

• Mechanism: The agent seeks to position itself in states from which it can cause a wide variety of potential future outcomes. It is motivated to preserve and expand its own agency.
• Example: A mobile robot might be driven to explore a central hub in a building rather than a dead-end corridor, because the hub offers more possible future paths (higher empowerment).
• Information Gain: A related concept where the agent seeks to maximize the reduction in entropy of its beliefs about the world. Actions that lead to the most informative observations are intrinsically rewarding.

A simpler but effective heuristic where an agent is rewarded for visiting states it has rarely or never encountered before. This directly encourages exploration of the state space.

• Mechanism: The agent maintains counts or a density model of visited states. The intrinsic reward for a state is inversely proportional to its visit count (e.g., 1/√(count)).
• Example: In a maze, an agent gets a high intrinsic reward the first time it enters a new room, and a diminishing reward each subsequent visit.
• Implementation: Often used in count-based exploration algorithms. For large or continuous state spaces, pseudo-counts are derived from a density model like a Context Tree Switching model or a neural network density estimator.

A practical deep reinforcement learning technique for generating a curiosity signal that is robust to unpredictable or noisy aspects of the environment.

• Mechanism: Two neural networks are used: a fixed, randomly initialized target network and a trainable predictor network. Both take an observation (or a feature thereof) as input. The predictor is trained to output the target network's output for that observation.
• Intrinsic Reward: The prediction error of the predictor network. High error indicates a novel or hard-to-predict observation, prompting exploration.
• Advantage: This method avoids the "noisy TV problem," where a classic prediction-error agent becomes obsessed with inherently unpredictable random noise. Since the target network's output is a deterministic function of its random weights and the input, unpredictable environmental noise does not yield a consistently high error signal.

An architectural module that integrates feature learning and curiosity-driven exploration. It learns a feature space where only the agent's actions affect state changes, filtering out irrelevant environmental distractions.

• Components:
  • Inverse Dynamics Model: Learns to predict the action taken between two consecutive states. This forces the learned feature representation to capture only aspects of the state that are controllable by the agent.
  • Forward Dynamics Model: Predicts the next state's features given the current state's features and the action. The error in this prediction becomes the intrinsic curiosity reward.
• Purpose: By using features from the inverse model, the curiosity signal focuses on agent-actionable novelty. The agent is not curious about a leaf blowing in the wind (not actionable), but is curious about a door it has never opened (actionable).
• Outcome: Leads to more efficient exploration in complex, high-dimensional environments like 3D games.

Agents are motivated by intrinsic curiosity to explore novel or unpredictable states. This is often implemented by rewarding the agent for reducing prediction error in a learned model of its environment.

• Mechanism: An agent uses an internal world model to predict the consequences of its actions. States where the model's prediction error is high are considered novel and yield a high intrinsic reward.
• Example: In a maze with no external reward, a curiosity-driven agent will systematically explore uncharted corridors to improve its model, eventually mastering the entire layout.
• Algorithmic Basis: Often formalized using concepts like Prediction Error or Information Gain.

This method motivates an agent to visit states it has encountered less frequently. The intrinsic reward is inversely proportional to a state's visit count.

• Mechanism: The agent maintains a pseudo-count or density model for states. The intrinsic reward is calculated as r_intrinsic = 1 / sqrt(N(s)), where N(s) is the visit count for state s.
• Example: In a large, procedurally generated game world, an agent using this method will prioritize moving into new biomes or areas it has rarely seen, ensuring comprehensive coverage.
• Key Benefit: Provides a mathematically simple yet powerful drive for uniform state coverage, preventing the agent from getting stuck in a small, familiar region.

A prominent algorithm where intrinsic reward is generated by the agent's inability to predict the output of a fixed, randomly initialized neural network (the target network).

• Mechanism: A predictor network is trained to mimic the outputs of a fixed random target network on observed states. The prediction error (the difference between the two networks' outputs) serves as the intrinsic reward. High error indicates a novel or unfamiliar state.
• Example: Used by OpenAI to train agents to master Montezuma's Revenge, a game with extremely sparse external rewards. The agent learned to explore rooms and interact with objects purely driven by the RND curiosity signal.
• Advantage: Does not require learning a complex dynamics model, making it more stable and computationally efficient for high-dimensional observations.

Agents are driven to seek out states that provide the greatest information gain or reduction in uncertainty about their knowledge of the world or their own capabilities.

• Mechanism: This can be framed as maximizing empowerment (an agent's potential influence on its future) or minimizing the entropy of a belief distribution. The agent acts to make its internal models more certain.
• Example: A robotic arm learning to manipulate objects might be motivated to try grasps that are most informative about an object's physical properties (e.g., weight, friction), even if the task is simply to move it.
• Formalization: Often linked to concepts like Bayesian Surprise or KL-divergence between prior and posterior beliefs after taking an action.

Here, the intrinsic motivation is to learn a diverse repertoire of useful skills or behaviors without a specific external goal. The agent is rewarded for achieving distinguishable outcomes.

• Mechanism: Algorithms like DIAYN (Diversity is All You Need) train a policy to maximize the mutual information between skills and states visited. A discriminator network tries to identify which skill was used, motivating the policy to visit distinct states for each skill.
• Example: A simulated humanoid agent might autonomously learn to run, jump, roll, and crawl simply by trying to maximize the diversity of its movements, creating a library of primitive actions for later task-specific fine-tuning.
• Outcome: Creates a foundation of reusable primitives that accelerate later learning on downstream tasks via transfer learning.

The agent is motivated by learning progress or increasing mastery over a self-selected set of challenges. The reward is higher when the agent is in a "zone of proximal development."

• Mechanism: The system identifies a set of possible goals or outcomes. The intrinsic reward is the derivative of the agent's probability of achieving a chosen goal—reward is highest when the agent is rapidly improving at a goal that was previously just out of reach.
• Example: A baby AI in a physics simulator might first focus on mastering simple goals like moving a block, then automatically shift its focus to harder goals like stacking blocks as its competence on the easy ones plateaus.
• Psychological Parallel: Directly inspired by theories of human cognitive development, where engagement is highest at the frontier of one's abilities.

Reward shaping is the engineering of supplemental reward signals to guide a reinforcement learning agent toward desired long-term goals, making sparse or delayed reward problems more tractable. It is a critical technique for translating high-level objectives into learnable signals.

• Purpose: Provides intermediate feedback to accelerate learning when the primary environmental reward is infrequent (e.g., winning a game only at the end).
• Relation to Intrinsic Motivation: While reward shaping is an external design intervention, intrinsic motivation mechanisms like curiosity can be seen as a form of internal, learned reward shaping, where the agent generates its own exploratory bonuses.

The exploration-exploitation tradeoff is the fundamental dilemma where an agent must balance trying new actions to gather information (exploration) with choosing known rewarding actions to maximize return (exploitation). Intrinsic motivation is a primary driver for exploration.

• Exploration Strategies: Include epsilon-greedy, Upper Confidence Bound (UCB), and Thompson sampling.
• Intrinsic Drives: Mechanisms like curiosity or prediction error directly generate internal rewards for visiting novel or uncertain states, systematically biasing the agent toward exploration to build a better world model.

Inverse Reinforcement Learning is the process of inferring an underlying reward function by observing an expert agent's behavior. It addresses the credit assignment problem in reverse: given optimal behavior, what was the goal?

• Core Problem: IRL assumes the demonstrator is acting optimally according to some unknown reward function; the algorithm's job is to recover that function.
• Contrast with Intrinsic Motivation: IRL seeks to discover an external reward signal from demonstrations. Intrinsic motivation, conversely, generates an internal reward signal without demonstrations, often to facilitate exploration for later mastery of an external task.

Hierarchical Reinforcement Learning decomposes complex tasks into a hierarchy of subtasks or skills, enabling temporal abstraction. Higher-level policies select which lower-level skill to execute over an extended period.

• Structure: Often uses frameworks like Options or MaxQ, with meta-controllers and sub-policies.
• Synergy with Intrinsic Motivation: Intrinsic drives can operate at different levels of the hierarchy. For example, a high-level controller might be intrinsically motivated to learn novel sub-skills, while a low-level skill policy is trained to maximize external reward. This enables structured exploration and skill discovery.

Model-Based Reinforcement Learning involves an agent learning an explicit model of the environment's dynamics (transition function) and reward function. This model is then used for planning, such as via Monte Carlo Tree Search (MCTS), to improve sample efficiency.

• Advantage: Can achieve high performance with fewer interactions with the real environment by planning in the learned model.
• Foundation for Curiosity: Many intrinsic motivation algorithms are predicated on learning a world model. Drives like prediction error use the discrepancy between the model's prediction and reality as an intrinsic reward, directly incentivizing the agent to improve its model by exploring areas of high uncertainty.

Distributional Reinforcement Learning models the full probability distribution of possible returns (the value distribution) rather than just its expected value. This provides a richer representation of uncertainty and risk.

• Output: Instead of a single Q-value, algorithms like C51 or QR-DQN output a distribution over returns.
• Informing Intrinsic Motivation: The distribution's variance or shape can directly quantify aleatoric (environmental) uncertainty. Intrinsic motivation algorithms can use this uncertainty—for instance, seeking states where the return distribution is wide or multimodal—as a drive for exploration, moving beyond simple prediction error.