Inventory-Aware Multi-Armed Bandit

The defining structural departure from classical bandits. Each arm has a maximum number of pulls equal to its current stock level. Once inventory reaches zero, the arm is removed from the action space entirely. This transforms the problem from an infinite-horizon Markov Decision Process into a finite-horizon stochastic game where the set of available actions shrinks over time. The agent must reason not just about expected reward, but about the opportunity cost of depleting a scarce resource that might be more valuable later in the decision horizon.

The reward function is augmented with an inventory penalty term that scales inversely with remaining stock. As inventory decreases, the effective reward for pulling that arm is discounted. This creates a natural exploration decay mechanism: items with abundant stock maintain high reward potential, while nearly depleted items are only recommended if their expected conversion rate is exceptionally high. The penalty function can be linear, exponential, or a step function triggered at reorder thresholds, allowing business logic to encode stockout aversion directly into the learning objective.

Classical bandits require explicit exploration bonuses (e.g., Upper Confidence Bound) or epsilon-greedy randomization to discover new high-reward arms. Inventory-aware bandits achieve emergent exploration suppression without explicit tuning. As an item's stock approaches zero, the inventory penalty naturally drives its selection probability toward zero. This eliminates the pathological behavior where a model continues to explore a product that is about to sell out, only to win a click it cannot fulfill. The exploration budget is automatically reallocated to items with sufficient inventory depth.

Regret in classical bandits measures the gap between the chosen arm's reward and the optimal arm's reward. Inventory-aware bandits introduce supply-constrained regret, which penalizes two distinct failure modes:

• Stockout Regret: Failing to recommend an item while it was in stock, leading to dead inventory.
• Waste Regret: Recommending an item that converts but cannot be fulfilled, creating customer disappointment. The optimization target shifts from pure click maximization to fulfillable revenue maximization, aligning the algorithm's incentives with supply chain realities.

New product introductions benefit from a Bayesian prior on their reward distribution that is conditioned on initial stock allocation. A product with deep initial inventory receives a more optimistic prior, encouraging faster exploration, while a product with shallow test inventory receives a conservative prior. This creates a supply-informed cold start strategy where the merchandising team's inventory commitment directly signals the algorithm about expected product importance, bridging the gap between human assortment planning and algorithmic discovery without manual rule overrides.

The bandit's policy conditions not only on user context features but also on a joint inventory state vector representing the stock levels of all arms. This allows the model to learn complex substitution dynamics: if a high-reward item is nearly depleted, the policy may preemptively shift probability mass to a complementary or substitute item with healthy stock, maximizing the probability of a fulfilled transaction. The context space expands to include the global supply picture, enabling coordinated decision-making across the entire catalog rather than independent per-item evaluations.

A grocery chain deploys an inventory-aware bandit to manage end-of-day bakery displays. The model's reward function is weighted by remaining shelf life.

• Mechanism: As the 'sell-by' window shrinks, the bandit automatically increases the item's exploration rate (display frequency) to maximize sell-through.
• Constraint: The model naturally ceases promotion once stock hits zero, preventing the recommendation of unavailable items.
• Result: A documented 22% reduction in end-of-day waste compared to static merchandising rules.

A fast-fashion retailer uses a contextual bandit for its 'limited drop' landing page, where deep sizes are scarce.

• Cold Start Handling: New styles are initially boosted (exploration) but the model rapidly suppresses items where specific sizes (e.g., XS, 3XL) are selling out, reallocating traffic to fully stocked alternatives.
• Contextual Signal: The bandit conditions on the user's inferred size profile, ensuring a size-2 shopper never sees a promotion for a dress sold out in their size.
• Business Logic: This prevents the 'frustrated click' where a user engages with an ad only to find an empty product detail page.

An electronics retailer integrates an inventory-aware bandit directly into the search ranking layer for high-intent queries like 'wireless headphones'.

• Exploration vs. Exploitation: The bandit exploits a best-selling Sony model but detects a declining stock gradient. It begins exploring a comparable Bose alternative before the Sony stockout occurs.
• Demand Transference: The model learns the substitution rate in real-time. If users consistently buy the Bose when the Sony is scarce, the bandit updates its policy to smooth demand across the category.
• Technical Note: Requires tight integration with the Product Affinity Graph to ensure explored alternatives are genuine substitutes, not random noise.

A home improvement chain uses localized bandits to balance stock across a metro area without manual transfers.

• Micro-Merchandising: If Store A is overstocked on patio sets and Store B is running low, the bandit for Store A's online profile aggressively promotes the patio sets to users geofenced near Store A.
• Reward Shaping: The reward includes a bonus for selling items with high holding costs (large cubic footage), incentivizing the model to clear bulky inventory locally.
• Constraint Satisfaction: The model respects a hard constraint on minimum display diversity, ensuring the homepage doesn't become entirely patio furniture even during an overstock event.

When a retailer opens a location in a new market, a global bandit with an inventory-aware 'warm start' initializes the local model.

• Transfer Learning: The model begins with priors from similar Store-Cluster Personalization groups (e.g., urban flagship priors for a new downtown store).
• Inventory Sensitivity: The bandit is hyper-sensitive to stockout signals during the first 30 days, rapidly suppressing items that sell unexpectedly fast to avoid training on empty shelves.
• Exploration Budget: A higher exploration budget is allocated initially to quickly map the new market's unique Localized Affinity Scoring, overriding global assumptions.

During a port strike, a retailer's inbound inventory becomes highly stochastic. The inventory-aware bandit adapts the entire homepage in real-time.

• Stockout Probability Scoring: The bandit ingests a probabilistic forecast of restock dates. Items with a high probability of a 2+ week stockout are immediately suppressed, even if currently in stock, to prevent a cliff in availability.
• Assortment Cannibalization Detection: The model monitors if the suppression of a disrupted item causes a surge in a substitute that is also low in stock, triggering a cascading protection rule.
• Business Continuity: This shifts the merchandising strategy from 'maximizing revenue per view' to 'maximizing availability-weighted relevance' during the crisis.