Action Space

This is the most fundamental classification of an action space. A discrete action space consists of a finite, countable set of distinct actions, like pressing a button on a game controller (e.g., {UP, DOWN, LEFT, RIGHT, A, B}). In contrast, a continuous action space is defined over one or more real-valued intervals, allowing for infinitely many possible actions, such as specifying the exact torque to apply to a robotic joint (e.g., torque ∈ [-5.0 Nm, 5.0 Nm]).

• Discrete Example: A chess-playing agent's action is to move a piece to one of 64 squares.
• Continuous Example: A self-driving car's steering angle is a continuous value between -30 and +30 degrees.

This distinction determines the choice of learning algorithms; Q-learning is classic for discrete spaces, while policy gradient methods like PPO or DDPG are designed for continuous control.

The dimensionality of an action space refers to the number of independent control variables that must be specified simultaneously for a single timestep. A high-dimensional action space exponentially increases the complexity of search and learning.

• 1-Dimensional: A thermostat with a single action: {HEAT, COOL, OFF}.
• Multi-Dimensional: A robotic arm with 7 degrees of freedom requires specifying a 7-dimensional action vector, where each element is the target position or torque for a joint.
• Very High-Dimensional: A humanoid robot with over 20 actuated joints, or a text-generating AI where the action is choosing the next word from a vocabulary of 50,000+ tokens.

High dimensionality is a primary driver of the curse of dimensionality, making exhaustive exploration infeasible and necessitating sophisticated function approximation and sampling strategies.

An action space is rarely a simple, unconstrained set. Real-world systems impose kinematic, dynamic, and safety constraints that define which actions are valid from a given state.

• Kinematic Constraints: A robot's joint limits prevent actions that demand physically impossible angles.
• Dynamic Constraints: Maximum acceleration and torque limits restrict how quickly an action can change the system's state.
• Temporal Constraints: Some actions have minimum execution durations or cannot be interrupted.
• State-Dependent Validity: The action GRASP is only valid if the robot's end-effector is near an object. The action JUMP is invalid if the agent is mid-air.

These constraints create a feasible action set that is a strict subset of the nominal action space. Planning and learning algorithms must respect this to generate executable behaviors, often modeled via preconditions in symbolic planning or masking in reinforcement learning.

Complex tasks are managed by structuring the action space into hierarchical levels. High-level actions are abstract operators that are decomposed into sequences of lower-level, more primitive actions.

• High-Level (Task): MakeCoffee()
• Mid-Level (Skill): NavigateToKitchen(), GraspMug(), PressBrewButton()
• Low-Level (Primitive): Joint position commands or motor voltages.

This hierarchy is central to Hierarchical Reinforcement Learning (HRL) and Hierarchical Task Network (HTN) planning. It reduces planning complexity by allowing reasoning at an abstract level and provides temporal abstraction, where a high-level action can persist over many timesteps. The choice of motion primitives or skills in the lower level defines the building blocks available to the agent.

Actions are often parameterized, meaning a single action type can be executed with different arguments. This combines elements of discrete and continuous spaces.

• Action Type: MOVE_TO(x, y)
• Parameters: The continuous coordinates (x, y) that specify the goal location.
• Other Examples: GRASP(object_id, force), SAY(text_string), ROTATE(joint, angle).

Parameterization allows for a compact representation of a vast action space. Learning and planning then involve both selecting the correct action type and optimizing its continuous parameters. This is a core challenge in task and motion planning (TAMP), where symbolic action selection must be coupled with geometric parameter optimization.

The action space does not exist in isolation; it is intrinsically linked to the state space and observation space. The dynamics model (or transition function) T(s, a) -> s' defines how an action a transitions the agent from state s to a new state s'.

• Underactuation: When the dimensionality of the action space is less than the state space, the agent has limited immediate control (e.g., a car cannot move directly sideways).
• Perception-Action Loop: The agent's policy π(o) -> a maps from an observation o (a potentially partial view of the state) to an action. The complexity of this mapping is driven by the size and structure of the action space.
• Reward Guidance: The reward function R(s, a) provides a scalar signal evaluating the quality of taking action a in state s, guiding the agent to explore its action space effectively.

Understanding this triad—state, action, and reward—is essential for designing agents that can learn and plan competently within their defined world.

In classic board and video games, the action space is a finite set of discrete moves. This makes the planning problem combinatorial but enumerable.

• Chess/Go Agent: Actions are legal moves (e.g., e2-e4, place stone at (3,4)).
• Text Adventure Game Bot: Actions are commands from a predefined vocabulary (e.g., GO NORTH, TAKE KEY, USE LAMP).
• Atari-playing RL Agent: Actions correspond to joystick positions (e.g., FIRE, UP, LEFT, RIGHT), often 4 to 18 discrete options.

These are tabular or discrete policy problems, where value iteration or policy gradient methods can directly map states to one of N actions.

For a robotic arm, the action space typically defines the commands sent to its joints. This is often a continuous, high-dimensional space.

• Velocity Control: Action is a vector of joint velocities [θ̇₁, θ̇₂, ..., θ̇ₙ].
• Torque Control: Action is a vector of joint torques [τ₁, τ₂, ..., τₙ].
• Position Control: Action is a vector of target joint angles [θ₁, θ₂, ..., θₙ].

For a 7-DoF arm, the action space is a 7-dimensional continuous box. Inverse kinematics solvers may be used to convert desired end-effector poses into this joint space. Constraints like joint limits and self-collision define the feasible action subspace.

A self-driving car's action space blends discrete high-level decisions with continuous low-level control.

• High-Level (Tactical): Discrete actions like CHANGE_LANE_LEFT, FOLLOW_LANE, MERGE.
• Low-Level (Control): Continuous actions like steering angle δ ∈ [-30°, 30°], acceleration a ∈ [-4 m/s², 3 m/s²], and brake pressure.

Modern architectures often use a hierarchical action space. A high-level planner selects a maneuver (discrete), and a low-level Model Predictive Control (MPC) or neural network policy executes it via continuous throttle/steering commands.

For robots like quadrupeds or bipeds, the action space must produce stable, dynamic gait cycles. It is high-dimensional and highly constrained.

• Joint-Level: Direct target positions or torques for each of 12+ joints (e.g., 3 joints per leg * 4 legs).
• Task Space: Desired foot swing trajectories and body posture, which are then resolved into joint commands via inverse kinematics.
• Parameterized Motion Primitives: Actions modulate parameters of cyclic gait generators (e.g., step frequency, swing height, body attitude).

This space is constrained by dynamics, stability (zero-moment point), and actuator limits. Reinforcement learning policies are often trained in simulation to map proprioception directly to joint torques.

A multi-rotor drone's action space is defined by the rotational speeds of its propellers, which generate thrust and torque.

• Low-Level: Direct motor PWM signals [ω₁, ω₂, ω₃, ω₄] for a quadcopter.
• High-Level: Desired body rates [p, q, r] (roll, pitch, yaw) and collective thrust T.
• Task-Level: Waypoint commands (x, y, z, ψ).

The mapping from high-level to low-level actions uses an attitude controller and mixing matrix. The action space must respect strict physical limits to prevent motor saturation and loss of controllability.

Many real-world problems require actions that have both a discrete type and continuous parameters.

• Robotic Skill Selection: Action = (SKILL_ID, PARAMETERS). Example: (GRASP, {location: [x,y,z], width: 0.05m}) or (PUSH, {direction: [dx,dy], force: 10N}).
• Hierarchical RL: A meta-controller picks a sub-policy (discrete), which then outputs primitive actions (continuous).
• Monte Carlo Tree Search for Continuous Domains: The tree branches on discrete decisions, while continuous parameters are optimized within each node.

This structure is modeled as a parameterized action space or option framework, requiring specialized algorithms like Parameterized Action DQN or Hybrid SAC.

The set of all possible configurations or situations that a dynamical system, such as a robot, can occupy. It is defined by a vector of state variables (e.g., joint angles, positions, velocities). The State Space and the Action Space are fundamental duals in control and reinforcement learning: the state describes where the agent is, and the action defines what it can do to transition to a new state.

• Key Relationship: A policy or planner maps states to actions.
• Example: For a mobile robot, the state space might be its (x, y) coordinates and heading; the action space is the set of velocity commands.

A specific, geometric type of State Space where each point represents a unique placement of all a robot's joints, ignoring dynamics. Obstacles in the physical world become forbidden regions in this abstract space. Motion planning often occurs in C-Space.

• Core Function: Transforms the problem of moving a multi-jointed body into moving a point through a landscape of obstacles.
• Dimensionality: Determined by the robot's degrees of freedom (DOF). A 6-DOF arm has a 6-dimensional C-Space.
• Connection to Action Space: Actions cause transitions between points in C-Space.

A short, parameterized, fundamental movement that serves as a building block for complex motion. Motion Primitives are often the elements that populate a discrete Action Space or are the output of a continuous action policy.

• Examples: A straight-line move, a specific joint rotation, a grasping trajectory, or a walking gait cycle.
• Usage in Planning: High-level task planners sequence primitives from a Skill Library.
• Benefits: Encodes stability, dynamics, and safety constraints into reusable modules.

The process of computing a smooth, time-parameterized path (a trajectory) that minimizes a cost function (e.g., energy, time, jerk) while satisfying dynamic constraints and avoiding collisions. It refines a coarse plan into an executable motion.

• Input/Output: Often takes a geometric path from a planner and outputs velocities and accelerations.
• Methods: Includes techniques like direct collocation and Model Predictive Control (MPC).
• Relation to Action Space: Defines the quality of a continuous action sequence within the space of possible motions.

A function, often a neural network, that maps an agent's perception of the State Space directly to an action in the Action Space. It is the core decision-making component in reinforcement learning and optimal control.

• Stochastic vs. Deterministic: A stochastic policy outputs a probability distribution over actions; a deterministic policy outputs a single action.
• Visuomotor Policy: A policy that maps raw visual inputs (pixels) directly to low-level motor commands.
• Learning Goal: To find the optimal policy that maximizes cumulative reward.

The hierarchical process of breaking a complex, high-level goal (e.g., 'make coffee') into a structured sequence of simpler, executable subtasks and finally into primitive actions. It bridges abstract intent to the Action Space.

• Formalisms: Uses frameworks like Hierarchical Task Networks (HTN) or Behavior Trees.
• Example: 'Make coffee' → [Grasp cup, Move to machine, Press button]. Each step may require its own motion plan.
• Outcome: Produces a Task Graph or plan where leaf nodes are actions the robot can perform.