Q-Value Decomposition

The core architectural assumption is that the global Q-value can be expressed as a sum of independent per-component utility functions. This linear decomposition enforces that the joint value is strictly the element-wise sum of sub-values, eliminating complex non-linear interactions. The constraint simplifies credit assignment because the gradient of the global temporal-difference error flows directly to the responsible sub-module without confounding cross-terms. This is the foundational principle behind architectures like Value Decomposition Networks (VDN).

A critical consistency condition for factored Q-functions, the IGM principle states that the optimal joint action derived from the global Q-value must be equivalent to the collection of individually optimal actions from each sub-Q-function. This ensures decentralized execution is consistent with centralized training. Algorithms like QMIX enforce this by using a monotonic mixing network with non-negative weights, guaranteeing that a local argmax translates to a global argmax without communication during execution.

Instead of processing a monolithic state vector, the architecture partitions the observation space by logical entities (e.g., individual robots, inventory items, or vehicles). Each entity's local state is fed into a shared or entity-specific utility network that outputs a per-entity Q-value. This structural prior injects strong inductive bias, forcing the model to learn reusable, interpretable sub-policies. The global critic then combines these entity-values, often using attention mechanisms to handle dynamic entity counts.

Decomposition is often coupled with TD(λ) eligibility traces to solve long-horizon credit assignment. When a delayed reward is received, the trace decays the credit exponentially backward through time, updating the specific decomposed sub-Q-function that was active at each step. This prevents the 'structural credit assignment' problem where a sub-module is penalized for a global failure it did not cause, by linking the temporal sequence of activations to the final decomposed value estimate.

Advanced decomposition methods replace static summation with multi-head attention to compute context-dependent weights for each sub-value. The global Q-value becomes a weighted sum where the mixing weights are a function of the full state. This allows the model to dynamically prioritize certain agents or features based on the scenario. The attention weights themselves serve as a direct interpretability tool, visualizing which sub-component the system is 'focusing on' at any given timestep.

A counterfactual evaluation technique used to validate decomposed Q-values. A difference reward calculates an agent's marginal contribution by subtracting the team reward when that agent takes a default 'null' action from the actual team reward. This provides a ground-truth signal for training the decomposition, ensuring the factored Q-values align with causal responsibility rather than statistical correlation, and is particularly effective in multi-agent reinforcement learning (MARL) environments.

The foundational architecture that additively decomposes a joint Q-value into a sum of per-agent Q-values. Each agent's Q-function conditions only on its local observation history, enforcing the assumption that the joint value is the sum of individual contributions. This linear decomposition provides a direct credit assignment mechanism: the magnitude of an agent's Q-value directly indicates its contribution to the team's expected return. VDN is the simplest form of Q-value decomposition and serves as the baseline for more expressive methods.

The central challenge that Q-value decomposition addresses: determining which agent's action contributed to a shared team reward. Without decomposition, the environment returns a single scalar reward, making it impossible to distinguish between an agent that acted optimally and one that was carried by teammates. Decomposition methods solve this by learning to factor the joint value function into individual utility functions. Alternative approaches include difference rewards, which estimate an agent's marginal contribution by comparing the actual reward to a counterfactual where the agent's action is replaced with a default action.

An extension of VDN that relaxes the additive assumption by using a monotonic mixing network. A hypernetwork takes the global state as input and generates the weights and biases of a mixing network that combines per-agent Q-values into a joint Q-value. The key constraint is that the mixing network's weights are constrained to be non-negative, ensuring monotonicity: the joint Q-value increases monotonically with each agent's individual Q-value. This guarantees that the argmax over joint actions is consistent with the argmax over individual actions, enabling decentralized execution with centralized training.

A game-theoretic approach to credit assignment that computes the Shapley value for each agent's Q-function. The Shapley value is the average marginal contribution of an agent across all possible coalitions of other agents. In the context of Q-value decomposition, this provides a fair and axiomatic distribution of the joint value: it satisfies efficiency (the sum equals the joint value), symmetry (identical agents get equal credit), and null player (agents with zero contribution get zero credit). The computational cost is exponential in the number of agents, requiring Monte Carlo sampling for practical use.

A counterfactual credit assignment method that computes an agent's contribution as the difference between the actual global reward and the reward that would have been received if the agent had taken a default or baseline action. This directly measures the marginal impact of an agent's action on the team outcome. Unlike Shapley values, difference rewards consider only one counterfactual per agent, making them computationally efficient. However, they require defining a meaningful default action and can be noisy in stochastic environments where the counterfactual outcome is difficult to estimate.

A framework that decomposes a task into a hierarchy of sub-policies operating at different temporal scales. A high-level policy selects options (temporally extended actions), while low-level policies execute primitive actions to achieve sub-goals. This provides a temporal decomposition of value that complements the spatial decomposition of Q-value methods. The value of an option can be interpreted as the expected return for completing a specific sub-task, making the agent's long-horizon behavior more explainable by inspecting which sub-goals are being pursued at each level of the hierarchy.