Exploration Strategy

The most fundamental exploration strategy. The agent selects the action with the highest estimated value (greedy) most of the time, but with a probability epsilon (ε), it selects a random action.

• Exploitation: With probability 1 - ε, choose argmax_a Q(s,a).
• Exploration: With probability ε, choose a random action.
• A common practice is to start with a high ε (e.g., 1.0) and decay it over time, allowing for aggressive early exploration and gradual convergence to a greedy policy.

A principle that selects actions based on an optimistic estimate of their potential. The agent chooses the action that maximizes the sum of the current estimated value plus an exploration bonus.

• Formula: A_t = argmax_a [ Q_t(a) + c * sqrt( ln(t) / N_t(a) ) ]
• Q_t(a) is the current value estimate.
• N_t(a) is the number of times action a has been chosen.
• The term c * sqrt( ln(t) / N_t(a) ) is the confidence interval. Less-tried actions have a larger bonus, ensuring they are explored.
• Provides a deterministic, theoretically-grounded balance without random action selection.

A Bayesian probability matching strategy. The agent maintains a probability distribution (posterior) over the value of each action. On each step, it samples a value from each distribution and selects the action with the highest sampled value.

• Process: 1) Maintain a prior (e.g., Beta distribution for Bernoulli rewards). 2) Update the posterior based on observed rewards. 3) Sample a potential reward from each posterior. 4) Act greedily with respect to the samples.
• Naturally balances exploration and exploitation: actions with uncertain, high-reward potential are sampled highly more often.
• Particularly effective for contextual bandits and recommendation systems.

Drives exploration by generating internal reward signals based on the agent's learning progress or prediction error, independent of the external task reward.

• Curiosity-Driven Exploration: Rewards the agent for visiting states where its model of the environment makes high prediction errors. Implemented via an intrinsic curiosity module (ICM).
• Count-Based Exploration: Adds a bonus β / sqrt(N(s)) for visiting a state s, where N(s) is a visitation count. Encourages the agent to revisit rare states.
• Random Network Distillation (RND): Uses the error of a neural network predicting the output of a fixed random network as an intrinsic reward. High error indicates a novel state.
• Essential for sparse-reward or hard-exploration environments like Montezuma's Revenge.

Exploration is driven by adding parameter noise directly to the weights of the policy or value network, rather than to the action output.

• Method: The network's weights are defined as θ = μ + σ ⊙ ε, where ε is noise sampled from a fixed distribution (e.g., Gaussian). The parameters μ and σ are learned.
• Advantage over ε-greedy: Provides consistent, state-dependent exploration. The same state observed twice will result in similar, but not identical, actions due to weight perturbations.
• Leads to more sophisticated, directed exploration patterns and is a key component in algorithms like Rainbow DQN and others.

A modern off-policy algorithm that explicitly maximizes both expected reward and the entropy of the policy. The optimal policy aims to be as random as possible while still succeeding at the task.

• Objective: J(π) = Σ E_{(s_t, a_t) ~ ρ_π} [ r(s_t, a_t) + α H(π(·|s_t)) ]
• H(π(·|s_t)) is the entropy, a measure of randomness. The temperature parameter α controls the trade-off.
• Effect: The policy is encouraged to explore widely by assigning non-zero probability to all actions that could be optimal. Exploration emerges naturally from the optimization objective rather than being a separate heuristic.
• Results in robust, sample-efficient learning and is a standard for continuous control benchmarks.

A drive for an agent to explore based on internally generated rewards, supplementing or replacing external task rewards. Key mechanisms include:

• Curiosity: Rewarding the agent for visiting novel or unpredictable states.
• Novelty Search: Encouraging the discovery of states not seen before.
• Empowerment: Maximizing an agent's influence over future states. This creates a self-sustaining exploration signal in sparse or deceptive reward environments.

The mathematical framework for sequential decision-making when an agent cannot directly observe the full environment state. This fundamentally shapes exploration:

• The agent maintains a belief state, a probability distribution over possible true states.
• Exploration must reduce uncertainty in this belief.
• Strategies often involve information-gathering actions to resolve ambiguity, making exploration more critical and complex than in fully observable settings.

The study of multiple autonomous agents learning to interact in a shared environment. Exploration strategies must account for:

• Non-stationarity: Other agents are also learning, changing the environment dynamics.
• Credit Assignment: Determining which agent's actions led to a shared reward.
• Social Exploration: Strategies like curiosity about other agents' policies or coordinated exploration to discover cooperative protocols.

A training paradigm where tasks are presented in a sequence of increasing difficulty. It structures exploration by:

• Scaffolding: Starting with simple tasks where useful behaviors are easy to discover.
• Automated Task Generation: Using the agent's performance to schedule the next task, balancing mastery and challenge.
• Guided Exploration: The curriculum acts as a prior, directing the agent's exploration towards progressively more complex parts of the state space.

Learning a policy from a fixed, previously collected dataset without further environment interaction. This imposes strict constraints on exploration:

• No Online Exploration: The agent cannot gather new data; it must learn solely from the static dataset.
• Distributional Shift: The learned policy must avoid taking actions that lead to states outside the dataset's support, where its value estimates are unreliable.
• Conservative/Pessimistic Algorithms: Methods like CQL (Conservative Q-Learning) are designed to mitigate this by penalizing unseen actions.

A subfield focused on learning to maximize performance while satisfying constraints and avoiding catastrophic failures. Exploration must be constrained:

• Risk-Aware Exploration: Avoiding actions with high variance or potential for constraint violation.
• Shielding: Using external safety filters to override unsafe exploratory actions.
• Exploration within Safe Sets: Limiting exploration to regions of the state-action space known or inferred to be safe, often defined via Lyapunov functions or reachability analysis.