Curiosity-Driven Exploration

The core mechanism replaces or augments sparse extrinsic rewards with an internally generated curiosity bonus. This bonus is proportional to the agent's prediction error—the discrepancy between predicted and actual next-state features. In a Markov Decision Process (MDP), the agent maximizes a combined objective: r_total = r_extrinsic + η * r_intrinsic, where η controls exploration weight. This transforms a barren reward landscape into a dense learning signal, enabling the agent to discover novel warehousing strategies without waiting for a terminal delivery success signal.

Curiosity is formalized as the error of a learned forward dynamics model. An Intrinsic Curiosity Module (ICM) takes the current state s_t and action a_t and predicts the next state embedding φ(s_{t+1}). The intrinsic reward is the Euclidean distance: r_i = ||φ(s_{t+1}) - φ̂(s_{t+1})||². States that are hard to predict yield high rewards, driving the agent toward unexplored regions. Over time, as the dynamics model improves, familiar states become predictable and boring, naturally decaying the curiosity signal and preventing endless exploration of mastered zones.

A critical failure mode where an agent fixates on inherently unpredictable stimuli—like a static-filled television screen—because the prediction error never diminishes. This is mitigated by inverse dynamics feature learning, where the state encoder is trained to predict the action taken between consecutive states: a_t = g(φ(s_t), φ(s_{t+1})). This forces the feature space to encode only controllable aspects of the environment, filtering out uncontrollable noise. In logistics, this prevents an agent from obsessing over random weather fluctuations and instead focuses on actionable routing decisions.

An alternative to dynamics-based curiosity that measures novelty through distillation error. A fixed, randomly initialized target network produces an embedding for each state. A predictor network is trained online to match the target's output. The intrinsic reward is the MSE between the two: r_i = ||f_target(s) - f_predictor(s)||². Novel states produce high error because the predictor has not yet learned to mimic the target on those inputs. RND is simpler to implement than ICM and naturally avoids the noisy-TV problem without requiring inverse dynamics, making it popular for large-scale logistics simulations.

This approach rewards the agent for reaching states dissimilar from those stored in an episodic memory buffer. Novelty is computed as the distance to the k-nearest neighbors in memory: r_i = 1 / (k * Σ d(s, s_i)). As the agent explores, states are added to the buffer, and revisiting familiar areas yields diminishing rewards. This is particularly effective in sparse-reward logistics environments like warehouse navigation, where an agent must systematically cover aisles. Unlike prediction-based methods, episodic curiosity provides a direct measure of visitation frequency rather than learnability.

In Multi-Agent Reinforcement Learning (MARL) settings, curiosity must account for non-stationarity introduced by other learning agents. A naive ICM fails because the environment's unpredictability stems from co-adapting policies, not genuine novelty. Solutions include social curiosity, where agents are rewarded for influencing other agents' behaviors, or coordinated exploration where a shared novelty buffer prevents redundant discovery. In a fleet routing scenario, this ensures autonomous trucks explore complementary delivery zones rather than all converging on the same novel neighborhood.

The fundamental dilemma in reinforcement learning between gathering new knowledge about the environment and leveraging existing knowledge to maximize immediate reward. Curiosity-driven exploration directly addresses this by providing intrinsic motivation to explore even when extrinsic rewards are absent.

• Exploration: Trying untested actions to discover better long-term strategies
• Exploitation: Choosing known high-reward actions based on current knowledge
• Balance: Too much exploration wastes resources; too much exploitation risks suboptimal local minima

A reward signal generated internally by the agent rather than from the environment. In curiosity-driven exploration, intrinsic motivation is typically computed as the prediction error of a learned dynamics model—the agent is rewarded for visiting states where its world model fails to predict outcomes.

• Prediction Error: The difference between predicted and actual next state
• Novelty Bonus: Additional reward proportional to state unfamiliarity
• Competence: Some formulations reward learning progress, not just novelty

A common challenge in real-world logistics where meaningful feedback arrives only at task completion—such as a successful delivery or warehouse operation. Curiosity-driven exploration provides dense learning signals in these environments by rewarding the discovery of novel states.

• Example: A routing agent receives reward only upon delivery; curiosity rewards exploring new road segments
• Impact: Enables learning in environments where random exploration would never find the goal
• Mitigation: Intrinsic rewards bridge the gap between actions and delayed outcomes

A learned neural network that predicts the next state given the current state and action. In curiosity-driven exploration, the prediction error of this model serves as the intrinsic reward signal—larger errors indicate unfamiliar territory worth exploring.

• Input: Current state representation and action taken
• Output: Predicted next state embedding
• Training: Continuously updated as the agent gathers experience
• Limitation: Can be fooled by stochastic or noisy environments (the 'noisy TV' problem)

The mechanism by which an agent identifies whether a state is familiar or novel. Techniques range from count-based methods in discrete spaces to density estimation and reachability analysis in continuous, high-dimensional logistics environments.

• Count-Based: Track visit frequency for discrete states
• Hash-Based: Use locality-sensitive hashing for continuous state counting
• Density Models: Learn probability distribution over visited states
• Random Network Distillation (RND): Use a fixed random network as a novelty detector

A practical curiosity-driven exploration algorithm where a fixed, randomly initialized target network defines a prediction problem. A predictor network is trained to match the target's outputs; the prediction error serves as the intrinsic reward, with higher errors indicating less-visited states.

• Target Network: Frozen, random weights—never updated
• Predictor Network: Trained online to minimize MSE against target outputs
• Advantage: Simple to implement, works well in high-dimensional spaces
• Application: Effective for discovering novel logistics configurations in simulation