Reward Signal

The signal must accurately attribute credit to the action that caused it. Immediate rewards (e.g., a click) are trivial to attribute. Delayed rewards (e.g., a purchase 48 hours later) require sophisticated credit assignment mechanisms. Without proper attribution, the agent learns from noise. Common solutions include eligibility traces or session-window attribution models that decay the credit of earlier actions over time.

An effective reward signal must be monitored for concept drift. A signal that is stationary today (e.g., a 10% discount driving a 2x lift) may degrade tomorrow due to competitor actions or seasonality. The reward function should be paired with a drift detection module that triggers retraining or signals human operators when the statistical properties of the reward distribution change significantly, preventing the bandit from optimizing for an outdated world.

Often, the true business objective (e.g., long-term customer lifetime value) is unobservable in real-time. We substitute a proxy reward (e.g., a click). The danger is reward hacking, where the agent maximizes the proxy at the expense of the true goal (e.g., clickbait). An effective signal must be validated through counterfactual evaluation to ensure the proxy's monotonic relationship with the true north-star metric holds.

A sparse reward signal (e.g., only a final purchase) makes learning extremely slow. Reward shaping introduces intermediate, dense feedback to guide the agent. For instance, a product view might yield a +0.1 reward, an add-to-cart +0.5, and a purchase +1.0. This heuristic injects domain knowledge, but must be carefully tuned to avoid introducing unintended bias that prevents the discovery of novel, high-value paths.

A high-variance reward signal slows convergence. Techniques to reduce noise include capping outliers (winsorizing) to prevent a single anomalous transaction from skewing the model, and using importance sampling to reduce the variance of off-policy estimates. An effective signal is one where the difference in reward between a good and bad action is statistically distinguishable from the background noise of random user behavior.

Real-world systems rarely optimize a single metric. An effective signal is often a scalarized composite of multiple objectives. For example: Reward = w1*(Click) + w2*(Revenue) - w3*(Return_Probability). The choice of weights (w1, w2, w3) is a critical business decision. Pareto frontier analysis can help visualize the trade-offs between conflicting objectives, like maximizing engagement versus minimizing churn risk.

The reward signal must directly encode the business objective the bandit is tasked with maximizing. In e-commerce, this is rarely a single binary event.

• Click-through rate (CTR): A common but often misleading proxy. It optimizes for curiosity, not necessarily purchase intent.
• Conversion rate: A stronger signal, but sparse. A user may only convert once per session, creating a delayed reward problem.
• Revenue per session: The gold standard for retail. This naturally balances order value and conversion frequency.
• Profit margin-weighted revenue: Prevents the bandit from over-recommending high-revenue, low-margin items that erode profitability.

The chosen metric must be observable, attributable to the specific action, and aligned with long-term customer lifetime value, not just immediate session outcomes.

Real-world personalization requires balancing multiple, often conflicting, objectives. A scalarized composite reward combines them into a single numerical value.

• Linear scalarization: R = w1*revenue + w2*engagement - w3*returns
• Constrained optimization: Maximize revenue subject to a minimum diversity constraint to prevent filter bubbles.
• Lexicographic ordering: Prioritize a primary objective (e.g., conversion) and only optimize a secondary one (e.g., revenue) when the primary is equal.

Critical pitfall: Poorly tuned weights can lead to reward hacking, where the agent finds an unintended shortcut to maximize the composite score without achieving the true business goal. For example, a high weight on clicks can cause the bandit to recommend clickbait that never converts.

The time between an action (showing a recommendation) and the reward (a purchase) can span hours or days. This credit assignment problem must be engineered explicitly.

• Session-level attribution: Assign the final session outcome (e.g., total revenue) to all actions in the session, discounted by recency.
• N-step temporal difference (TD): Update the value of an action based on the immediate reward plus the predicted value of the subsequent state.
• Hindsight replay: At session end, replay the trajectory and assign the final outcome to each action, enabling offline learning from sparse signals.
• Surrogate signals: Use intermediate engagement (add-to-cart, wishlist) as a dense proxy reward while waiting for the final conversion signal.

For extremely sparse events like subscriptions or high-value purchases, importance sampling can re-weight historical data to make rare positive rewards more salient during training.

Raw reward magnitudes can destabilize bandit learning. A $5,000 purchase creates a gradient orders of magnitude larger than a $20 purchase, causing catastrophic updates.

• Reward clipping: Cap extreme values at a predefined percentile (e.g., 99th) to prevent outliers from dominating the loss function.
• Logarithmic scaling: Apply log(1 + revenue) to compress the dynamic range while preserving ordering.
• Running z-score normalization: Continuously track the mean and standard deviation of rewards and standardize them: R_norm = (R - μ) / σ.
• Inverse propensity scoring (IPS): Correct for the bias introduced by the logging policy, ensuring that rewards for frequently chosen actions are down-weighted.

Normalization is not a one-time step. It must be computed online using exponential moving averages to adapt to non-stationary reward distributions during seasonal events like Black Friday.

You cannot directly observe the reward of actions you did not take. Off-policy evaluation estimates what would have happened under a different policy using only logged data.

• Direct Method (DM): Train a regression model to predict the reward for any context-action pair, then evaluate the target policy against this model.
• Inverse Propensity Scoring (IPS): Re-weight observed rewards by 1 / p(action|context) where p is the probability under the logging policy. Unbiased but high variance.
• Doubly Robust (DR): Combines DM and IPS. R_DR = V_DM + (R_observed - V_DM) / p. Remains unbiased if either the reward model or the propensity model is correctly specified.
• Capped IPS: Clip propensity weights to a maximum value to trade a small amount of bias for a large reduction in variance.

This is the scientific method for bandits. Never deploy a new reward design without first validating it offline against historical production logs.

Consumer preferences shift. A reward signal that was optimal in January may be miscalibrated by December. The bandit must detect and adapt to concept drift.

• Sliding window estimation: Only use the last N days of data to compute reward statistics, discarding stale observations.
• Exponential decay weighting: Apply a decay factor γ to historical rewards: weight = γ^(age_in_days). A γ of 0.99 gives a half-life of ~69 days.
• Changepoint detection: Monitor the reward distribution for statistically significant shifts using CUSUM or Bayesian online changepoint algorithms. Trigger a model reset or increased exploration when detected.
• Contextual drift monitoring: Track the distribution of input features. If the user population changes (e.g., a new marketing campaign attracts a different demographic), the old reward model's assumptions are violated.

A robust system combines passive adaptation (decay) with active detection (changepoint monitoring) to gracefully handle both gradual evolution and abrupt market shocks.