Distributional RL

Unlike classical RL which learns only the expected value (mean) of future returns, distributional RL learns the complete probability distribution of the random return variable Z(s,a). This captures higher-order moments like variance (risk) and skewness (asymmetry), providing a richer signal for policy optimization.

• Learns P(Z | s, a) instead of just E[Z]
• Exposes tail risk and upside potential
• Enables risk-sensitive policy selection

A foundational distributional algorithm that discretizes the return distribution into a fixed set of support atoms (e.g., 51 bins) spanning possible value ranges. The network outputs a categorical distribution over these atoms, trained via a projection of the distributional Bellman update onto the fixed support.

• Uses V_min and V_max to bound support
• KL divergence between projected target and prediction
• Demonstrates that distributional learning alone improves stability

Represents the return distribution through quantiles rather than fixed-value bins. The network learns the inverse cumulative distribution function at pre-specified quantile fractions (e.g., median, 90th percentile). This avoids the support discretization bias of C51 and focuses learning on distributional statistics most relevant to decision-making.

• Learns N quantile locations τ_i = i/N
• Uses quantile Huber loss for robustness
• Naturally captures heavy-tailed distributions common in logistics delays

Extends QR-DQN by conditioning the network on a continuous quantile fraction τ sampled from U[0,1] during training. This allows the agent to query arbitrary quantiles at test time, enabling dynamic risk policies without retraining. A risk-sensitive policy can be derived by distorting the sampling distribution toward pessimistic or optimistic quantiles.

• Continuous τ ∈ [0,1] via embedding
• Enables on-the-fly risk adjustment
• State-of-the-art sample efficiency on Atari benchmarks

The theoretical foundation: instead of the scalar Bellman equation, distributional RL applies the distributional Bellman operator that transforms the entire return distribution. Under the Cramér distance metric, this operator is a contraction, guaranteeing convergence to the true return distribution in the limit.

• Z(s,a) := R(s,a) + γ Z(S', A')
• Equality holds in distribution, not expectation
• Preserves multimodality and stochasticity through bootstrapping

Distributional RL directly addresses supply chain risk management by enabling policies conditioned on risk metrics like Conditional Value at Risk (CVaR). An agent can optimize for the worst 5% of delivery time outcomes rather than average performance, critical for cold chain integrity and contractual service-level agreements.

• CVaR optimization via quantile distortion
• Trade off mean performance vs. tail risk
• Natural fit for disruption-prone routing problems

The formal mathematical framework for sequential decision-making under uncertainty. An MDP is defined by a tuple (S, A, P, R, γ) where S is the state space, A is the action space, P is the state transition probability, R is the reward function, and γ is the discount factor. Distributional RL operates directly on this framework but models the full distribution of the random return Z(s,a) rather than its expectation Q(s,a).

• Standard RL solves for E[Z]; Distributional RL solves for Z itself
• Captures aleatoric uncertainty inherent in logistics environments
• Enables risk-sensitive policies without modifying the underlying MDP

A seminal Distributional RL algorithm that parameterizes the return distribution using a fixed set of N quantiles. Instead of learning a single expected value, QR-DQN learns the value at each quantile τᵢ = i/N using the Huber quantile loss. This provides a non-parametric representation of the full return distribution without assuming any specific shape.

• Uses quantile regression to approximate the inverse CDF of returns
• Enables explicit risk-sensitive policy extraction via distortion operators
• Demonstrated state-of-the-art performance on Atari benchmarks

The first successful Distributional RL algorithm, C51 models the return distribution as a discrete categorical distribution over a fixed set of 51 atoms (support points) spanning a predefined value range V_min to V_max. The Bellman update is performed using a projection step that maps the target distribution back onto the fixed support.

• Parametric representation with fixed support and learnable probabilities
• Introduced the distributional Bellman equation as a contraction in Wasserstein space
• Proved that modeling distributions, not just expectations, improves learning stability

An extension of QR-DQN that learns the full quantile function implicitly rather than at fixed quantile positions. IQN samples τ ~ U([0,1]) during training and uses an embedding function φ(τ) to condition the network on the desired quantile level. This enables risk-sensitive policies to be extracted at inference time without retraining.

• Continuous quantile function: Z_τ(s,a) for any τ ∈ (0,1)
• Risk-averse policies: sample from τ ~ U([0, β]) with β < 1
• Risk-seeking policies: sample from τ ~ U([β, 1]) for optimistic planning

The broader paradigm that Distributional RL enables natively. By learning the full return distribution, agents can optimize for risk-adjusted objectives beyond expected value. Common distortion risk measures include:

• CVaR (Conditional Value at Risk): Expected return in the worst α% of outcomes
• Wang transform: Exponentially reweights quantiles toward pessimism or optimism
• CPW (Cumulative Prospect Weighting): Models human-like asymmetric risk perception

In logistics, this translates to optimizing for worst-case delivery times rather than average performance.

The theoretical insight that the distributional Bellman operator is a contraction in the Wasserstein metric, guaranteeing convergence of distributional methods. Key properties:

• Distributional Bellman Equation: Z(s,a) = R(s,a) + γZ(S', A') in distribution
• Preserves multimodality: captures scenarios with distinct high-reward and low-reward outcomes
• Auxiliary task hypothesis: Predicting distributions provides a richer learning signal than scalar regression
• Mitigates the winner's curse in Q-learning by reducing overestimation bias through distributional smoothing