Action Space

A finite, countable set of distinct actions. This is the most common formulation in contextual bandits, where the agent selects exactly one arm from a catalog.

• Structure: A set of N distinct arms, e.g., A = {banner_A, banner_B, banner_C}.
• Cardinality: Can range from small (2-10 arms for A/B testing) to massive (millions of items in product recommendation).
• Selection: The policy outputs a single arm ID or a probability distribution over all arms.
• Example: A homepage hero banner selector choosing between 5 promotional creatives.
• Solver Compatibility: Works natively with Thompson Sampling, UCB, and Epsilon-Greedy.

An infinite, uncountable set where actions are real-valued vectors. The agent must select precise numerical parameters rather than picking from a predefined list.

• Structure: A d-dimensional real vector space, e.g., a ∈ ℝᵈ.
• Constraints: Often bounded by a valid range, such as [0, 1] for a discount percentage or [-1, 1] for a motor torque.
• Selection: The policy outputs a specific real number or vector, often parameterized by a Gaussian distribution in Deep Deterministic Policy Gradient (DDPG) or Soft Actor-Critic (SAC).
• Example: A dynamic pricing engine setting a product price to any value between $9.99 and $49.99.
• Key Challenge: Standard discrete bandit algorithms cannot be directly applied; requires function approximation or discretization.

A structured action space where the agent selects an ordered subset of items from a large catalog simultaneously. The action is a slate—a ranked list—not a single item.

• Structure: The set of all possible k-permutations or k-combinations from a ground set of N items.
• Complexity: The number of possible actions grows factorially, making exhaustive enumeration impossible.
• Selection: Requires specialized decomposition, such as assuming additive reward across slate positions or using a top-k policy that scores items independently.
• Example: A streaming service selecting a personalized top-10 movie row for a user's homepage.
• Feedback: The agent receives a composite reward signal, often with position bias, requiring Inverse Propensity Scoring (IPS) corrections.

A mechanism that dynamically restricts the available action space at each decision step based on business rules or environmental constraints, preventing the agent from selecting invalid arms.

• Mechanism: A binary mask vector m ∈ {0, 1}ⁿ is applied to the action scores, setting invalid actions to -∞ before softmax or argmax selection.
• Triggers: Inventory stockouts, user eligibility rules, geographic restrictions, or contractual obligations.
• Impact: Reduces the effective action space size and ensures the policy respects hard constraints without needing to learn them through trial and error.
• Example: A fashion retailer masking out-of-stock sizes for a specific product variant, ensuring the bandit only recommends purchasable items.
• Implementation: Applied in the final layer of the policy network, preserving the full action space for gradient computation but zeroing out invalid logits.

The design choice between a coarse-grained action space (fewer, broader actions) and a fine-grained one (many, specific actions) involves a fundamental trade-off between learning speed and optimization precision.

• Coarse-Grained: Fewer arms mean faster learning and lower variance in reward estimates, but the optimal action may not exist in the set, leading to approximation error.
• Fine-Grained: More arms allow for highly precise optimization, but each arm receives fewer trials, increasing variance and slowing convergence.
• Hierarchical Solution: A two-level action space where a top-level bandit selects a category and a bottom-level bandit selects a specific item within that category.
• Example: A push notification system choosing between "daily digest" (coarse) vs. a specific personalized article headline (fine).
• Guidance: The optimal granularity is determined by the traffic volume and the business cost of suboptimal actions.

A critical property defining whether the reward distribution for each action remains constant over time. This assumption dictates the algorithm's need for adaptation versus accumulation.

• Stationary: The expected reward for each arm is fixed. The agent should asymptotically converge to the single best arm and stop exploring. Standard UCB and Thompson Sampling are designed for this regime.
• Non-Stationary: The reward distributions drift over time due to trends, seasonality, or competitor actions. The agent must continuously explore and discount old observations.
• Detection: Monitoring for concept drift in the reward signal, often using sliding window statistics or change-point detection algorithms.
• Example: A news recommendation bandit faces a highly non-stationary action space where article relevance decays within hours.
• Mitigation: Use of discounted UCB or sliding-window Thompson Sampling that weights recent observations more heavily.

The most common e-commerce action space where the agent selects a single Stock Keeping Unit (SKU) from a catalog. The space is finite but massive, often containing millions of items.

• Challenge: Pure exploration is impossible; relies on candidate generation to reduce the space to a manageable top-N set.
• Example: A fashion retailer selecting one dress from 500,000 SKUs to feature in a hero banner.
• Algorithm Fit: LinUCB or Neural Bandits with a pre-trained embedding layer for cold-start items.

A continuous action space where the agent selects a real-valued price point within a bounded range. The reward is typically profit margin multiplied by conversion probability.

• Nuance: The reward function is often non-linear and non-convex, with psychological pricing thresholds.
• Example: A hotel chain dynamically setting room prices between $89 and $499 per night based on occupancy, lead time, and competitor scraping.
• Algorithm Fit: Deep Deterministic Policy Gradient (DDPG) or Bayesian Optimization for smooth, continuous control.

A combinatorial action space where the agent selects an ordered list of K items from a catalog of N. The action is not a single item but a slate.

• Complexity: The action space size is N!/(N-K)!, making exhaustive evaluation impossible.
• Example: A social media platform selecting the top 10 posts to display in a user's feed from a candidate pool of 500.
• Algorithm Fit: Slate Bandits using a greedy top-K selection from per-item predicted scores, or a sequence-to-sequence model for diversity.

A discrete, small-cardinality action space where the agent selects a communication channel and content variant. Actions are mutually exclusive at the decision step.

• Actions: {Push Notification, Email, In-App Modal, SMS, No Action}.
• Example: A fintech app deciding whether to send a savings goal reminder via push notification or email, and which creative variant to use.
• Algorithm Fit: Contextual Thompson Sampling with a shared context feature vector representing user engagement history across channels.

A structured action space organized as a decision tree. The agent first selects a high-level category, then a sub-category, and finally a specific item.

• Benefit: Reduces the effective branching factor and allows for shared statistical strength at higher nodes.
• Example: A food delivery app first selecting cuisine type (Italian), then dish category (Pasta), then a specific dish (Spaghetti Carbonara).
• Algorithm Fit: Hierarchical Bandits that propagate rewards up the taxonomy tree, enabling faster cold-start for new dishes within a known cuisine.

An emerging pattern where the action is a parameter vector fed into a generative model. The bandit selects the parameters, and the model generates the final artifact.

• Actions: A latent code or a set of prompt modifiers (tone, length, style) that control a Large Language Model's output.
• Example: A marketing platform selecting prompt parameters (e.g., formality=0.8, urgency=0.3) to generate email subject lines, with open-rate as the reward.
• Algorithm Fit: Continuous bandits or Bayesian Optimization over the parameter space, combined with a frozen generative model.