Reinforcement Learning (RL)

The fundamental architecture of RL is a closed-loop system. The agent is the learner and decision-maker. The environment is everything the agent interacts with. At each time step t, the agent receives a representation of the environment's state (S_t), selects an action (A_t), and in the next step receives a numerical reward (R_{t+1}) and a new state (S_{t+1}). This interaction generates a trajectory of experience from which the agent learns.

All goals in RL can be framed as the maximization of the expected cumulative reward signal. The agent's objective is not to maximize immediate reward, but the return—the total accumulated reward over the long run. This is formalized as G_t = R_{t+1} + γR_{t+2} + γ^2R_{t+3} + ..., where γ (gamma) is the discount factor (0 ≤ γ ≤ 1) that determines the present value of future rewards. A γ close to 0 makes the agent myopic; a γ close to 1 makes it farsighted.

A policy (π) is the agent's strategy, mapping states to probabilities of selecting each possible action. It is the core output of an RL algorithm.

• Deterministic Policy: a = π(s). A direct mapping from state to a single action.
• Stochastic Policy: π(a|s) = P(A_t=a | S_t=s). A probability distribution over actions given a state, enabling exploration. The goal is to find the optimal policy (π)* that maximizes expected return from all states.

Value functions estimate how good it is to be in a state or to perform an action in a state, in terms of expected future return.

• State-Value Function V(s): The expected return starting from state s and following policy π thereafter: V_π(s) = E_π[G_t | S_t = s].
• Action-Value Function Q(s,a): The expected return starting from state s, taking action a, and following policy π thereafter: Q_π(s,a) = E_π[G_t | S_t = s, A_t = a]. These are linked by the Bellman Equations, which express a recursive relationship between the value of a state and the values of its successor states.

This is the central dilemma of RL. To maximize reward, an agent must exploit actions it knows yield high rewards. However, to discover better actions, it must also explore actions it hasn't tried before. The agent cannot do both simultaneously. Common strategies include:

• ε-greedy: With probability ε, choose a random action (explore); otherwise, choose the best-known action (exploit).
• Upper Confidence Bound (UCB): Select actions based on their potential for high reward, considering uncertainty.
• Thompson Sampling: A Bayesian approach that selects actions according to their probability of being optimal.

RL algorithms are categorized by whether they use a model of the environment.

• Model-Based: The agent learns or is given a model that predicts the next state and reward for any action. It can use this model to plan by simulating future trajectories before acting. This is sample-efficient but computationally intensive.
• Model-Free: The agent learns a policy or value function directly from real experience without a model. It is simpler to implement but often requires many more interactions with the environment. Examples include Q-Learning and Policy Gradients.

An RL agent learns a policy to assign delivery tasks to drivers in real-time, maximizing fleet utilization and minimizing idle time. The agent observes the state of all vehicles and pending orders, then takes an action to dispatch or withhold.

• State Space: Vehicle locations, remaining capacity, driver hours-of-service, and order queue.
• Action Space: Assign order k to vehicle j or hold.
• Reward Function: Positive reward for on-time delivery; negative penalty for late arrivals, empty miles, and SLA breaches.
• Key Benefit: Outperforms greedy heuristics by sacrificing short-term efficiency for long-term fleet balance, anticipating future demand surges in specific geofences.

An RL agent continuously adjusts a vehicle's planned route in response to stochastic events like traffic congestion, road closures, or new on-demand orders. The agent learns to balance the cost of deviation against the risk of a missed delivery window.

• Context: Operates on top of a solved Vehicle Routing Problem with Time Windows (VRPTW) baseline.
• Mechanism: The agent treats each intersection or waypoint as a decision step, choosing the next segment based on live traffic and remaining slack time.
• Reward Shaping: Penalizes every minute of lateness exponentially, encouraging a policy that is risk-averse with respect to hard time windows.
• Outcome: Learns non-intuitive shortcuts and waiting strategies that static solvers cannot replicate.

An RL agent learns an optimal pricing and acceptance policy for on-demand delivery slots. The agent must decide whether to accept a new order at a given price, considering current fleet capacity and the probability of a more profitable order arriving later.

• Problem Framing: Modeled as a Markov Decision Process (MDP) with a finite horizon (the operating day).
• State: Time remaining, number of active vehicles, current backlog, and order value.
• Action: Accept order at price p, reject, or offer a counter-price.
• Objective: Maximize total revenue while maintaining a target On-Time In-Full (OTIF) rate.
• Key Insight: The agent learns to raise prices dynamically in zones with low driver density and to reject low-value orders that would jeopardize high-value commitments.

For electric vehicle fleets, an RL agent jointly optimizes routing and charging decisions. The agent learns when to divert a vehicle to a charging station, balancing the time cost of charging against the risk of range depletion and delivery failure.

• Constraint: Vehicles have a limited range and non-linear charging curves (fast charging from 20-80% is more time-efficient than 80-100%).
• State Augmentation: Standard VRP state plus battery state-of-charge, charging station locations, and queue lengths at stations.
• Reward Engineering: Heavy penalty for battery depletion requiring a tow; moderate penalty for charging stops that cause missed time windows.
• Result: The policy learns to opportunistically charge during driver breaks or near delivery points with available chargers, minimizing deadhead trips to stations.

In crowdsourced delivery models, multiple independent couriers are coordinated by a central RL agent or trained via Multi-Agent Reinforcement Learning (MARL). Each courier-agent learns a decentralized policy to bid on or accept nearby orders.

• Architecture: Centralized training with decentralized execution (CTDE). A global critic sees all courier locations during training, but each agent acts on local observations at execution time.
• Challenge: Non-stationarity—as one agent's policy changes, the environment becomes unpredictable for others.
• Solution: Parameter sharing and opponent modeling stabilize training.
• Application: Optimizes the matching of ad-hoc couriers to hyperlocal delivery tasks, minimizing customer wait time without explicit central dispatch.

An RL agent learns to schedule reverse logistics pickups by integrating new return requests into existing forward-delivery routes. The agent decides whether to accept a return pickup immediately, defer it, or bundle it with a future delivery in the same neighborhood.

• State: Existing delivery manifest, new return request location and time window, vehicle capacity.
• Action: Insert return pickup at position i in the route, defer to a later shift, or reject.
• Reward: Positive for each completed return; negative for exceeding vehicle capacity or violating delivery promises.
• Efficiency Gain: The agent learns to cluster returns geographically and temporally, turning a cost center into a marginal-cost operation by piggybacking on existing routes.