Value Function

The state-value function, denoted V(s), estimates the expected cumulative future reward (or return) an agent will receive starting from a given state s and following a specific policy π thereafter. It answers the question: "How good is it to be in this state?"

• Formal Definition: Vπ(s) = Eπ[Gt | St = s], where Gt is the return from time t.
• Primary Use: Used for policy evaluation, to determine the quality of a given policy.
• Example: In a gridworld, V(s) for a cell close to the goal would be high, while V(s) for a cell near a hazard would be low.

The action-value function, or Q-function (Q(s, a)), estimates the expected cumulative future reward for taking a specific action a in state s and thereafter following policy π. It answers: "How good is it to take this action in this state?"

• Formal Definition: Qπ(s, a) = Eπ[Gt | St = s, At = a].
• Primary Use: The foundation for policy improvement and control. Algorithms like Q-Learning and Deep Q-Networks (DQN) learn the optimal Q-function (Q*).
• Example: For a robot arm, Q(s, 'extend gripper') would estimate the long-term reward of that specific motor command from its current joint configuration.

The optimal value functions, V*(s) and Q*(s, a), represent the maximum expected return achievable from a state or state-action pair under any policy. They define the ceiling of performance.

• Bellman Optimality Equation: Q*(s, a) = E[ Rt+1 + γ * max a' Q*(St+1, a') ]. This recursive equation is the target for value-based RL methods.
• Relation to Policy: The optimal policy π* can be derived directly from Q*: π*(s) = argmax a Q*(s, a).
• Significance: Learning these functions is synonymous with solving the RL problem. Model-based planning algorithms often compute approximations of V* or Q*.

The advantage function, Aπ(s, a), measures the relative benefit of taking a specific action a in state s compared to the average action under the policy π. It is defined as Aπ(s, a) = Qπ(s, a) - Vπ(s).

• Key Property: It has zero mean over the actions for a given state under the policy.
• Primary Use: Central to actor-critic methods like Advantage Actor-Critic (A2C/A3C) and Proximal Policy Optimization (PPO). By focusing on the advantage, the policy update reduces variance and becomes more stable.
• Interpretation: A positive advantage means the action is better than average; a negative advantage means it is worse.

In complex environments with vast or continuous state spaces, exact tabular value functions are impossible. Value function approximation uses parameterized functions (like neural networks) to estimate V(s) or Q(s, a).

• Function Approximators: Deep Neural Networks are standard, leading to Deep Reinforcement Learning.
• Challenges: Introduces risk of instability and divergence due to non-stationary targets, correlated data, and overestimation bias.
• Stabilizing Techniques: Target networks, experience replay, and double Q-learning are essential engineering solutions to these challenges.

These are the two fundamental approaches for learning value functions from experience, differing in how they use the concept of bootstrapping.

• Monte Carlo (MC) Methods: Learn value estimates from complete episodes of experience. They are unbiased but have high variance and must wait until an episode ends. Example: MC Prediction for V(s).
• Temporal Difference (TD) Methods: Learn by bootstrapping—updating estimates based on other estimates. They can learn online, from incomplete sequences, with lower variance but some bias. TD(0) update: V(St) ← V(St) + α[ Rt+1 + γV(St+1) - V(St) ].
• TD(λ): A generalization that smoothly interpolates between TD(0) and Monte Carlo returns.

The state-value function estimates the expected cumulative future reward from a given state, assuming the agent follows a specific policy. It answers: "How good is it to be in this state?"

• Core Definition: V^π(s) = E_π[ Σ γ^k R_{t+k+1} | S_t = s ], where γ is a discount factor.
• Use in Policy Evaluation: Used in algorithms like Temporal Difference (TD) Learning to assess a policy's performance without a full model.
• Example: In a gridworld navigation task, V(s) assigns a high value to states close to the goal and a low value to states near hazards, guiding the agent's path.

The action-value function (Q-function) estimates the expected cumulative reward from taking a specific action in a given state and thereafter following a policy. It is central to model-free control.

• Core Definition: Q^π(s, a) = E_π[ Σ γ^k R_{t+k+1} | S_t = s, A_t = a ].
• Foundation for Q-Learning: The Bellman optimality equation for Q*, Q*(s,a) = E[ R + γ max_{a'} Q*(s', a') ], is the basis for the Q-Learning algorithm.
• Example: In playing Atari games, a deep Q-network (DQN) learns a Q-function that outputs the estimated value of each possible joystick action (e.g., move left, fire) for a given screen frame.

The advantage function measures the relative benefit of taking a specific action compared to the average action in that state, defined as A(s,a) = Q(s,a) - V(s).

• Reduces Variance: By subtracting the state baseline V(s), it provides a lower-variance signal for policy gradient updates.
• Key to Actor-Critic: Modern algorithms like Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C) use estimates of the advantage function to stabilize training.
• Example: For a robotic arm, V(s) estimates the average future reward for being in its current joint configuration. A(s,a) indicates how much better (or worse) moving a specific joint is versus a random movement.

In Soft Actor-Critic (SAC), a state-value function is learned explicitly to stabilize training in continuous action spaces.

• Maximum Entropy Objective: SAC maximizes both reward and policy entropy. Its value function, V(s), estimates the expected future reward plus entropy.
• Role in Critic Networks: SAC uses two Q-networks (critics) and a separate V-network. The V-network is trained to minimize the squared error against a target derived from the Q-networks and the policy's entropy.
• Impact: This architecture enables highly sample-efficient and robust learning for tasks like dexterous manipulation and legged locomotion.

For complex environments with high-dimensional state spaces (e.g., pixels), the value function is approximated using a deep neural network.

• Function Approximator: A neural network parameterized by weights θ, such as V_θ(s) or Q_θ(s,a), generalizes across unseen states.
• Training Signal: The network is trained via Temporal Difference (TD) error, δ = R + γV(s') - V(s), which serves as the loss function's target.
• Challenge of Non-Stationarity: Because the target (R + γV(s')) depends on the network's own changing parameters, techniques like target networks are used to stabilize training, as seen in DQN and DDPG.

In model-based reinforcement learning, a learned or given dynamics model is used with a value function for planning.

• Dyna Architecture: Agents use a model to generate simulated experiences (s, a, r, s'), which are then used to update the value function or policy, blending model-free and model-based learning.
• Monte Carlo Tree Search (MCTS): Algorithms like AlphaGo use a value network to evaluate board positions, guiding a look-ahead search through possible future states.
• Example: A warehouse robot can use an internal model of the facility to simulate navigation paths, using its value function to estimate the long-term efficiency of each potential route before executing a move.

The Q-function, or action-value function, is a specific type of value function denoted as Q(s, a). It estimates the expected cumulative future reward of taking a specific action a in a given state s, and thereafter following a particular policy. It is the core component of Q-Learning and Deep Q-Networks (DQN). The optimal Q-function, Q*(s,a), satisfies the Bellman optimality equation and directly defines the optimal policy by selecting the action with the highest Q-value in each state.

The Bellman equation provides the recursive, self-consistent mathematical foundation for value functions. It decomposes the value of a state into the immediate reward plus the discounted value of the successor state. For a state-value function V(s) under policy π, it is expressed as: Vπ(s) = Σ_a π(a|s) Σ_s' P(s'|s,a)[R(s,a,s') + γVπ(s')]. This recursion is the basis for dynamic programming and Temporal Difference (TD) Learning algorithms, which perform iterative updates to converge on accurate value estimates.

Temporal Difference (TD) Learning is a class of model-free algorithms that learn value functions by bootstrapping—updating estimates based on other estimates. A TD update for the state-value function is: V(s) ← V(s) + α [r + γV(s') - V(s)], where the term in brackets is the TD error. This method learns from incomplete episodes without requiring a model of the environment. TD(λ) and the TD target used in Deep Q-Learning (r + γ max_a' Q(s',a')) are key extensions of this core idea.

The advantage function Aπ(s, a) quantifies how much better a specific action is compared to the average action in a given state, under policy π. It is defined as Aπ(s, a) = Qπ(s, a) - Vπ(s).

• A positive advantage indicates the action is better than average.
• It is central to Actor-Critic methods and policy gradient algorithms like Advantage Actor-Critic (A2C/A3C) and Proximal Policy Optimization (PPO).
• Using the advantage reduces variance in policy gradient estimates, leading to more stable and efficient learning.

The Actor-Critic architecture is a foundational RL framework that explicitly separates the two core components: the Actor (a policy network that selects actions) and the Critic (a value function network that evaluates the state or state-action pair). The Critic's role is to estimate V(s) or Q(s,a), providing a training signal—often the advantage—to the Actor. This separation enables more stable learning than pure policy gradient methods and is the basis for algorithms like A3C, PPO, and SAC.

In Model-Based Reinforcement Learning, an agent learns or is given an explicit model of the environment's dynamics (transition function T(s'|s,a) and reward function R(s,a)). The value function becomes a tool for planning within this learned model. Algorithms like Model Predictive Control (MPC) or Monte Carlo Tree Search (MCTS) use the model to simulate future trajectories and estimate their cumulative reward (i.e., their value) to select optimal actions, often without directly learning a policy network.