Action Space

A discrete action space consists of a finite, countable set of distinct actions. The agent selects from a list of predefined options, such as {left, right, up, down} in grid navigation or specific joystick button presses. This type is common in board games (chess moves) and simple robotic tasks (selecting a pre-programmed grip).

• Mathematical Representation: Defined as a finite set (A = {a_1, a_2, ..., a_n}).
• Learning Algorithms: Typically optimized with algorithms like Deep Q-Networks (DQN) that estimate the value of each discrete action.
• Advantage: Simpler to explore and learn, as the policy outputs a probability distribution over a limited set.
• Limitation: Cannot represent fine-grained, continuous control required for precise torque or velocity adjustments.

A continuous action space is defined over one or more real-valued intervals, allowing the agent to output any numerical value within specified bounds. This is essential for physical control tasks like setting joint torques [-2.0 Nm, 2.0 Nm], steering angles, or motor velocities.

• Mathematical Representation: Defined as a bounded subset of (\mathbb{R}^n), e.g., (a \in [low, high]^d).
• Learning Algorithms: Requires policy-gradient methods like Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC) that directly parameterize a probability distribution (e.g., Gaussian) over the continuous range.
• Advantage: Enables precise, high-dimensional control necessary for dexterous manipulation and locomotion.
• Challenge: Vastly larger and more complex to explore, often requiring more sophisticated exploration strategies.

A hybrid (or mixed) action space combines both discrete and continuous components within a single decision step. This models complex real-world tasks where an agent must choose a discrete high-level action and its continuous parameters. A classic example is a robotic arm that must select a grasp type (discrete: pinch, power) and the continuous force to apply.

• Mathematical Representation: The space is the Cartesian product of discrete and continuous sets: (A = A_{disc} \times A_{cont}).
• Learning Algorithms: Presents a significant challenge. Solutions include parameterized action-space Markov Decision Processes (PAMDPs) and specialized architectures that branch network outputs.
• Real-World Relevance: Critical for hierarchical control and tasks in autonomous driving (e.g., change lane and set acceleration) or robotic manipulation.

The dimensionality of an action space refers to the number of independent control variables the agent outputs per timestep. A simple thermostat has a 1D space (heat on/off). A humanoid robot may have a 50D space (angles for dozens of joints).

• Low-Dimensional Spaces: Easier and faster to learn. Example: CartPole (1D: push left/right force).
• High-Dimensional Spaces: Pose the curse of dimensionality; the volume of the space grows exponentially, making exploration exponentially harder. Example: Shadow Hand manipulation (~20D).
• Action Bounds: Practical systems require hard constraints (e.g., joint limits, max voltage) or soft constraints (penalties in the reward function) to ensure safe, physically plausible actions. Bounds are typically enforced by clipping or using squashing functions (e.g., tanh) in the policy network.

This distinction concerns the granularity and semantic level of the actions available to the agent.

• Primitive (Low-Level) Actions: Direct, instantaneous motor commands. Examples: apply 0.5 Nm torque to joint_4, set wheel velocity to 1.2 m/s. These are the direct output of a visuomotor policy.
• Parameterized (High-Level) Actions: Abstract operations that are executed over time and may have arguments. Examples: move_to(x=1.5, y=3.2), grasp(object_id=7, force=5.0). These are used in hierarchical reinforcement learning and task-and-motion planning (TAMP), where a high-level planner selects parameterized actions that a low-level controller then executes.
• Trade-off: Low-level spaces offer maximum flexibility but are hard to learn. High-level spaces simplify learning by abstracting away physics but require careful engineering of the action repertoire.

The Options Framework extends the basic action space to include temporally extended actions, known as options. An option is a triple (\langle I, \pi, \beta \rangle): an initiation set I, an internal policy π, and a termination condition β. This allows an agent to choose macro-actions (e.g., navigate_to_kitchen) that persist for multiple timesteps.

• Relation to Action Space: It creates a hierarchical action space where the agent picks among options at a coarse time scale, and the option's internal policy picks primitive actions at a finer scale.
• Purpose: Drastically improves exploration and learning efficiency in long-horizon tasks by reducing the effective planning horizon.
• Implementation: Central to Hierarchical Reinforcement Learning (HRL). Methods like Option-Critic learn both the options and the policy over options end-to-end. This is key for scalable embodied AI in complex environments.

A discrete action space consists of a finite set of distinct, mutually exclusive actions. This is common in classic control problems and video games.

• Examples: The moves in a board game (e.g., chess), pressing a button on a game controller (up, down, left, right, A, B), or selecting a gear in a vehicle (Park, Reverse, Neutral, Drive).
• Learning Implication: Algorithms like DQN (Deep Q-Network) are designed for discrete spaces, where the network outputs a Q-value for each possible action.

A continuous action space allows the agent to output actions as real-valued vectors within a defined range. This is essential for precise physical control.

• Examples: Applying a specific torque to a robot joint (e.g., -2.5 Nm to +2.5 Nm), setting the steering angle and throttle of a car, or controlling the x,y,z force vector of a drone.
• Learning Implication: Requires algorithms like PPO (Proximal Policy Optimization) or SAC (Soft Actor-Critic) that can model a probability distribution (e.g., Gaussian) over the action range.

A hybrid (or mixed) action space combines discrete and continuous components within a single decision. This models complex real-world tasks where an agent must choose a high-level action type and its parameters.

• Example: A robotic manipulation task where the agent must first choose a discrete grasp type (e.g., pinch, power) and then specify continuous parameters for the gripper position and force.
• Learning Implication: More complex, often requiring specialized policy architectures that branch to handle different action types.

In multi-agent reinforcement learning, the joint action space is the Cartesian product of all individual agents' action spaces. The dimensionality grows exponentially with the number of agents.

• Example: A fleet of warehouse robots where each robot can choose from actions like move north, move south, pick up, set down. The joint action is the combination of all robots' choices.
• Challenge: The curse of dimensionality makes centralized learning intractable, often leading to decentralized policies or centralized training with decentralized execution (CTDE).

A parameterized action space is a specific type of hybrid space where each discrete action is associated with a set of continuous parameters. The agent selects both the action and how to execute it.

• Example: In a real-time strategy game, the discrete action might be Attack, which then requires continuous parameters for the target coordinates (x, y) on the map.
• Application: Bridges high-level strategic decisions with low-level continuous control, common in robotics and complex game AI.

Standardized benchmarks illustrate how action space design defines a problem's character.

• OpenAI Gym / Gymnasium - CartPole: Discrete (2 actions: push cart left or right).
• OpenAI Gym / Gymnasium - MuJoCo Ant: Continuous (8 actions: torques applied at each joint).
• DeepMind Control Suite - Humanoid Run: Continuous (21 actions: joint actuations).
• StarCraft II Learning Environment (SMAC): Hybrid (each unit has a set of discrete abilities, some with targeted parameters).

The observation space defines the set of all possible sensory inputs or states that an agent can perceive from its environment. It represents the agent's partial or complete view of the world, which may include pixel values from a camera, joint angles, LiDAR point clouds, or processed features. The observation is the input to the agent's policy, which then selects an action from the action space. A key distinction is that the observation may be a noisy or incomplete subset of the true underlying state of the environment.

A reward function is a mathematical function that provides a scalar feedback signal to an agent based on its action and the resulting state of the environment. It numerically encodes the task objective that the agent must learn to maximize over time. The design of the reward function is critical: a sparse reward (e.g., +1 for success, 0 otherwise) is challenging to learn from, while a dense, shaped reward can guide learning but may lead to unintended reward hacking behaviors. The reward signal, combined with the action space, defines the core optimization problem in reinforcement learning.

A policy is the agent's strategy or decision-making function that maps observations from the observation space to actions in the action space. It can be:

• Deterministic: Directly outputs a specific action.
• Stochastic: Outputs a probability distribution over actions. The policy is the core component learned through reinforcement learning. Its architecture (e.g., neural network) must be compatible with the action space's structure—outputting logits for discrete actions or mean/variance for continuous actions.

The state space represents the set of all possible configurations of the environment. It is the true, underlying description that fully determines the future dynamics, whereas the observation space is what the agent actually sees. In a fully observable Markov Decision Process (MDP), the observation equals the state. In a Partially Observable MDP (POMDP), the observation is a function of the state, and the agent must maintain an internal belief state. The state space's complexity directly influences the difficulty of learning a successful policy over the action space.

A Markov Decision Process (MDP) is the formal mathematical framework used to model decision-making in reinforcement learning. It is defined by the tuple (S, A, P, R, γ):

• S: State space.
• A: Action space.
• P: State transition probability function.
• R: Reward function.
• γ: Discount factor. The action space (A) is a core component of this tuple. The MDP assumes the Markov property, meaning the future state depends only on the current state and action, not the full history.

The environment is the world with which the agent interacts. It receives an action from the agent's action space, transitions to a new internal state, and emits an observation and a reward. Environments are categorized by their action space type (discrete, continuous, hybrid) and include:

• Simulated: Software-based, like OpenAI Gym or MuJoCo, for fast, parallelized training.
• Physical: Real-world systems like robots or industrial controllers, where action execution has real consequences. The environment's dynamics and physics impose the fundamental constraints that shape a viable action space.