Assortment Optimization Policy Gradient

Unlike Q-learning, which derives a policy indirectly from a value function, this method directly parameterizes the policy πθ(a|s). The neural network outputs a probability distribution over valid assortment actions given the current state (e.g., inventory levels, user segment, time of day). This is critical for large combinatorial action spaces where enumerating all possible product subsets for value estimation is computationally intractable.

The agent is trained to maximize long-term cumulative reward (e.g., total revenue per session, customer lifetime value) rather than immediate click-through. The objective function J(θ) is the expected sum of discounted future rewards. This aligns the model with strategic business metrics, preventing myopic decisions like promoting a high-margin item that damages long-term trust or session engagement.

Training relies on the log-derivative trick to compute an unbiased estimate of the policy gradient. The core update rule is:

• ∇J(θ) ∝ E[ G_t * ∇ log πθ(a_t|s_t) ]
• G_t is the observed cumulative discounted reward from time t.
• This increases the log-probability of actions that led to higher-than-expected returns.
• High variance is a known challenge, often mitigated with a learned baseline function (critic).

To reduce variance, the system typically uses a dual-network architecture:

• Actor (Policy Network): Learns the mapping from state to assortment action probabilities.
• Critic (Value Network): Learns to estimate the state-value function V(s) to serve as a baseline. The advantage function A(s,a) = Q(s,a) - V(s) guides the actor. Positive advantage reinforces the action; negative advantage suppresses it. This stabilizes training significantly.

To prevent premature convergence on suboptimal assortments, an entropy bonus is added to the loss function. This encourages the policy to maintain a high-entropy (uncertain) distribution over actions, forcing continued exploration of diverse product combinations. The coefficient λ controls the exploration-exploitation trade-off:

• High λ: More uniform assortment selection.
• Low λ: Greedy selection of the current best assortment.

The raw policy output is post-processed to enforce hard business rules. Invalid actions (e.g., displaying out-of-stock items, violating brand contracts) are masked by setting their log-probabilities to negative infinity before the softmax. This ensures the agent never selects infeasible assortments while still learning the optimal distribution over the remaining valid combinatorial space.

A critical ranking signal that the policy gradient agent ingests to prevent recommending out-of-stock items. This feature down-weights products with low inventory and up-weights those with surplus, ensuring the learned policy naturally avoids stockouts.

• Encodes real-time inventory position as a continuous feature vector
• Prevents the agent from learning policies that disappoint customers
• Integrates directly into the state representation of the policy network

Predicts which alternative product a customer will purchase when their first choice is unavailable. The policy gradient agent uses this signal to learn intelligent substitution strategies, maximizing revenue even during stockouts.

• Estimates cannibalization and halo effects across the catalog
• Trains on historical purchase sequences where preferred items were missing
• Enables the agent to proactively display substitutes before a stockout occurs

The streaming data infrastructure that captures impressions, clicks, and add-to-carts on every product display. This telemetry provides the immediate feedback loop required to compute the policy gradient and update the agent's weights.

• Ingests events at sub-second latency via Apache Kafka or similar brokers
• Feeds the reward function with session-level revenue and conversion signals
• Enables online policy updates without batch retraining cycles

A dense vector representation that fuses a product's static attributes with its real-time stock status. The policy gradient agent uses these embeddings as input features, allowing the neural network to natively reason about availability.

• Concatenates catalog embeddings with inventory scalar features
• Trained jointly with the policy network for end-to-end optimization
• Eliminates the need for separate post-processing filtering rules

A causal inference methodology that isolates the incremental revenue generated by the policy gradient agent's decisions. This technique separates the agent's true impact from confounding factors like price changes or seasonal demand shifts.

• Uses difference-in-differences or propensity score matching
• Validates that the learned policy drives genuine lift, not correlation
• Provides the business metric for evaluating policy gradient ROI