Multi-Armed Bandit

The fundamental dilemma at the heart of the MAB problem. Exploitation involves selecting the best-known action to maximize immediate reward, while exploration involves selecting a suboptimal action to gather data and improve future decisions. A pure exploitation strategy may miss a superior variant, while pure exploration incurs high opportunity cost. MAB algorithms mathematically balance these two objectives to minimize cumulative regret—the difference between the reward obtained and the reward that could have been obtained by always choosing the optimal action.

The primary mathematical objective of a bandit algorithm. Regret is formally defined as the expected loss due to not pulling the optimal arm at every time step. Key concepts include:

• Cumulative Regret: The total loss accumulated over the entire experiment duration.
• Regret Bound: A theoretical guarantee on the maximum possible regret, often expressed in Big O notation (e.g., logarithmic regret for UCB).
• Opportunity Cost: The practical business impact of sending traffic to a losing variant, which MABs directly minimize compared to fixed-horizon A/B tests.

A Bayesian approach to the multi-armed bandit problem that maintains a probability distribution over the true reward rate of each arm. At each step, the algorithm samples a value from each arm's posterior distribution and selects the arm with the highest sample. This naturally balances exploration and exploitation: arms with high uncertainty have wider distributions and are more likely to produce a high sample. As data accumulates, the distributions narrow, and the algorithm converges to the optimal arm. Thompson Sampling is known for its strong empirical performance and elegant mathematical foundation.

A frequentist algorithm that selects arms based on an optimistic estimate of their potential reward. The UCB score for each arm is calculated as the observed mean reward plus an exploration bonus that grows with the uncertainty of the estimate. The formula is typically:

• UCB = μ̂ + c * √(ln(t) / n) Where μ̂ is the empirical mean, t is the total number of trials, n is the number of times the arm was pulled, and c is a tunable exploration parameter. This ensures that arms with fewer pulls receive a larger bonus, encouraging systematic exploration.

An extension of the standard MAB framework where the algorithm observes side information (context) about the user or environment before making a decision. Instead of learning a single best arm for all situations, a contextual bandit learns a policy that maps contexts to actions. This enables true personalization:

• Linear Bandits: Model the expected reward as a linear function of the context features.
• Neural Bandits: Use deep neural networks to learn complex, non-linear relationships between context and reward.
• Application: Selecting a personalized product recommendation based on a user's browsing history and demographic profile.

The simplest bandit algorithm, serving as a baseline for comparison. With probability 1 - ε, the algorithm exploits by selecting the arm with the highest current estimated reward. With probability ε, it explores by selecting a random arm uniformly. Key characteristics:

• Fixed ε: A constant exploration rate (e.g., 10%) that never decays, leading to linear regret.
• Decaying ε: The exploration rate decreases over time (e.g., ε = 1/t), allowing the algorithm to converge.
• Limitation: Explores blindly without considering the uncertainty of each arm, making it less efficient than UCB or Thompson Sampling.

Continuously optimizes headlines, hero images, and call-to-action buttons without waiting for fixed-horizon A/B tests to conclude. A Thompson Sampling bandit allocates more impressions to the best-performing creative in real-time.

• Reduces the opportunity cost of showing underperforming variants
• Automatically adapts to time-of-day and seasonal preference shifts
• Example: A media site increased click-through rate by 22% by replacing a traditional A/B test with a bandit that converged on the winning headline in hours rather than weeks

A contextual multi-armed bandit learns the optimal send time for each user by treating each hour of the day as an arm. The model observes open rates and adjusts delivery schedules to maximize engagement.

• Features include historical open behavior, timezone, and app activity patterns
• Avoids notification fatigue by suppressing sends during low-engagement windows
• Example: A food delivery app reduced opt-outs by 18% while maintaining order volume by learning individualized quiet hours

Modern recommender systems frame item selection as a bandit problem where each candidate item is an arm. LinUCB and Bayesian bandits balance exploiting known user preferences with exploring novel content to prevent filter bubbles.

• Handles the cold-start problem by rapidly exploring new inventory
• Incorporates real-time feedback signals like dwell time, skip, and purchase
• Netflix and Spotify use bandit-based exploration to introduce users to new content categories without degrading session quality

Treats each hyperparameter configuration as an arm in a bandit framework. Hyperband and Bayesian optimization bandits allocate more compute budget to promising configurations and prune poor performers early.

• Dramatically reduces the cost of neural architecture search
• Outperforms grid search and random search by orders of magnitude in compute efficiency
• Example: A computer vision team reduced model tuning time from 14 days to 8 hours by using a successive halving bandit over 10,000 candidate configurations

E-commerce platforms use bandit algorithms to test price points for thousands of SKUs simultaneously. Each price is an arm, and the reward is profit or conversion rate, allowing the system to discover willingness-to-pay curves in real-time.

• Contextual bandits incorporate competitor pricing, inventory levels, and demand signals
• Automatically adjusts to demand shocks like weather events or trending products
• Major airlines and ride-sharing platforms employ bandit-based pricing to maximize revenue per seat or ride without manual intervention

The foundational bandit algorithm that selects the best-performing arm with probability 1-ε and explores a random arm with probability ε. While simple to implement, it explores uniformly and blindly, wasting resources on clearly suboptimal arms. A typical ε value decays over time (e.g., from 0.1 to 0.01) to reduce exploration as confidence grows.

A deterministic algorithm that selects the arm with the highest upper confidence bound, calculated as the sample mean plus an exploration bonus. The bonus is inversely proportional to how often an arm has been pulled, ensuring that uncertain, less-tested arms are explored. UCB1 achieves logarithmic regret, making it theoretically optimal for stationary reward distributions.

A Bayesian approach that maintains a posterior probability distribution over each arm's reward rate. At each step, it samples a value from each distribution and selects the arm with the highest sample. This naturally balances exploration and exploitation—arms with high uncertainty are sampled widely, sometimes beating high-confidence arms. It is highly effective in practice and robust to delayed feedback.

An extension where the agent observes side information (context) before making a decision. Instead of learning a single best arm, it learns a policy that maps contexts to optimal actions. This enables personalization—showing different recommendations to different users based on their features. Algorithms like LinUCB and neural bandits model the expected reward as a function of the context.

The theoretical objective of bandit algorithms, measuring the cumulative difference between the reward obtained and the reward that would have been obtained by always selecting the optimal arm. Minimizing regret is equivalent to maximizing total reward. Key bounds include:

• Lai & Robbins lower bound: Asymptotically optimal regret is logarithmic.
• Sub-linear regret: Any algorithm with regret growing slower than linearly is considered a 'no-regret' learner.

A variant where the true reward distributions of arms change over time, violating the standard i.i.d. assumption. This requires algorithms that can forget stale data and adapt quickly. Strategies include:

• Sliding windows: Only consider the most recent w observations.
• Discount factors: Exponentially weight recent rewards more heavily.
• Change-point detection: Actively reset the model when a distribution shift is detected.