Exploration-Exploitation Trade-off

The process of trying new or non-obvious actions to discover their outcomes. In logistics, this might mean testing an untested shipping lane or a novel inventory rebalancing strategy.

• Purpose: Builds a more accurate model of the environment's dynamics and reward structure.
• Risk: Incurs short-term cost or suboptimal performance for potential long-term gain.
• Example: A freight routing agent dispatches a truck on a historically slower route to gather fresh traffic data after a highway expansion.

Leveraging the agent's current knowledge to select the action believed to yield the highest immediate reward. This is the "safe" choice based on past experience.

• Purpose: Optimizes for short-term performance and efficiency.
• Risk: Can trap the agent in a local optimum, preventing the discovery of a globally superior strategy.
• Example: A warehouse robot consistently picks the shortest known path to a shelf, ignoring a potentially faster route that hasn't been fully mapped.

A foundational method for managing the trade-off. The agent exploits the best-known action with probability 1-ε and explores a random action with probability ε.

• Mechanism: A simple, tunable parameter controls the balance.
• Decay: Epsilon is often decayed over time, shifting from high exploration during initial learning to high exploitation in a mature policy.
• Application: A dynamic pricing agent uses ε-greedy to occasionally test a non-optimal price point to measure demand elasticity.

A deterministic exploration strategy that selects actions based on their potential for high reward. It balances the estimated value of an action with the uncertainty in that estimate.

• Formula: Selects the action that maximizes Q(a) + c * sqrt(ln(t) / N(a)), where c is an exploration constant.
• Principle: Actions with high uncertainty are given an optimistic bonus, encouraging exploration of less-tried options.
• Use Case: A supplier selection agent uses UCB to occasionally re-evaluate a vendor with a small sample size but a potentially high reliability score.

A probabilistic, Bayesian approach to the trade-off. The agent maintains a probability distribution over the true reward of each action and samples from these distributions to make a choice.

• Mechanism: Actions with a high probability of being optimal are chosen more frequently, naturally balancing exploration and exploitation.
• Advantage: Highly effective in practice and robust to delayed feedback, common in supply chain outcomes.
• Example: A demand forecasting model uses Thompson Sampling to dynamically select between different predictive algorithms based on their recent probabilistic performance.

A class of methods that drive exploration by rewarding the agent for discovering novel or unpredictable states, rather than relying solely on extrinsic environmental rewards.

• Curiosity Module: An internal model predicts the next state; the agent is rewarded proportionally to the prediction error.
• Benefit: Prevents the agent from getting stuck in "dark rooms" where no external reward is present, encouraging thorough state-space coverage.
• Logistics Context: A digital twin simulation agent is intrinsically rewarded for discovering edge-case disruption scenarios that break the standard recovery policy.