Action Space

A finite set of distinct, mutually exclusive actions. This is common in game-playing agents (e.g., move up/down/left/right) and high-level task selection.

• Representation: Typically a categorical distribution. The policy outputs a probability for each possible action.
• Learning: Often uses algorithms like DQN (Deep Q-Network) or policy gradients with a softmax output layer.
• Example: A robot's high-level planner choosing from {pick_up, move_to_kitchen, place_on_table}.

An infinite set of actions where each action is a vector of real numbers. This is essential for precise low-level motor control, such as joint torques or velocities.

• Representation: A multivariate probability distribution, often a Gaussian or Beta distribution. The policy outputs the parameters (e.g., mean and log standard deviation).
• Learning: Requires algorithms like PPO, SAC, or DDPG that can handle continuous outputs.
• Example: A robotic arm where the action is a 7-dimensional vector of torque commands for each joint.

A combination of discrete and continuous components within a single action step. This models complex real-world tasks requiring both categorical choices and precise parameterization.

• Representation: A hierarchical or factored distribution. For example, a discrete choice selects a skill, and a conditional continuous vector parameterizes it.
• Learning: More complex, often using specialized network heads or algorithms like Parameterized Action-Space MDP (PAMDP) approaches.
• Example: A drone choosing to hover (discrete) or move_to a specific 3D coordinate (continuous x, y, z).

The number of degrees of freedom in the action vector. High-dimensional action spaces (e.g., for humanoid robots with >20 joints) present a significant exploration and optimization challenge.

• Impact: The size of the space grows exponentially with dimensionality, making naive exploration infeasible. This is the curse of dimensionality.
• Mitigation: Techniques include using structured policies (e.g., decentralized sub-policies), hierarchical RL, and curriculum learning to build skills incrementally.

Physical and safety limits that bound the valid action space. A policy must respect actuator limits, self-collision avoidance, and environmental constraints.

• Types:
  • Box Constraints: Simple min/max bounds on each action dimension (e.g., joint limits).
  • Non-Linear Constraints: Complex, coupled limits (e.g., torque-speed curves, dynamic stability).
• Enforcement: Can be baked into the policy output (e.g., using a tanh activation scaled to limits) or handled by a safety layer or constrained RL objective.

The granularity of time over which an action is applied. A primitive action (e.g., 10ms torque command) differs from an option or skill that executes a sequence of primitives.

• Low-Level (Primitives): Direct motor commands. High frequency, precise, but requires extensive planning.
• High-Level (Options): Macro-actions or skills that abstract a sequence. Reduces planning horizon but requires robust skill design or learning.
• Connection: Hierarchical policies explicitly manage this abstraction, with a high-level manager invoking low-level controllers.

A finite set of distinct, mutually exclusive actions. Common in classic reinforcement learning benchmarks and games.

Key Examples:

• Atari Games (e.g., Pong, Breakout): Actions like LEFT, RIGHT, FIRE, NOOP.
• Board Games (e.g., Chess, Go): Actions are moves on the board, represented as a large but finite set.
• High-Level Robot Commands: PICK_UP, MOVE_FORWARD, TURN_LEFT.

Technical Impact: Enables the use of policy gradient methods that output a probability distribution over actions (e.g., via a softmax layer).

An infinite set of actions where each dimension is a real-valued number. Essential for precise physical control.

Key Examples:

• Robotic Arm Control: A 7-degree-of-freedom arm's action space is a 7D vector of joint torques or target velocities.
• Autonomous Driving: Steering angle (continuous) and throttle/brake pressure (continuous).
• Drone Flight: Continuous thrust and pitch/roll/yaw rates for each rotor.

Technical Impact: Requires algorithms like Soft Actor-Critic (SAC) or Proximal Policy Optimization (PPO) with Gaussian or Beta distributions to sample actions. Model Predictive Control (MPC) also operates in continuous spaces.

Combines discrete and continuous components within a single decision step. Models real-world tasks requiring both choice and parameterization.

Key Examples:

• Robot Manipulation: Discrete action: GRASP; Continuous parameters: grasp pose (x, y, z, orientation) and gripper force.
• Strategy Games with Parameterized Actions: ATTACK (discrete) with a continuous parameter for attack intensity.
• Hierarchical Robot Control: High-level discrete skill selection (NAVIGATE_TO_KITCHEN) with low-level continuous motion primitives.

Technical Impact: Presents a significant architectural challenge, often addressed with specialized policy networks that have separate heads for discrete and continuous outputs.

A specific type of hybrid space where each discrete action is associated with a set of continuous parameters. The agent must choose what to do and how to do it.

Key Examples:

• Simulated Humanoid Soccer: KICK (discrete) with parameters for kick direction (azimuth, elevation) and power.
• Warehouse Robotics: PLACE_ON_SHELF with parameters for the precise (x, y, z) placement coordinates.
• Drone Package Delivery: DROP_PACKAGE with parameters for release altitude and velocity.

Technical Impact: Often tackled using actor-critic methods where the critic evaluates the joint choice of action type and its parameters.

For end-to-end visuomotor control, the action space is the direct output of a neural network consuming pixel inputs. Its design is critical for stability and performance.

Common Representations:

• Joint Space: Target joint positions, velocities, or torques. Direct but requires good internal proprioceptive feedback.
• Task Space (Cartesian): End-effector delta poses (Δx, Δy, Δz, Δroll, Δpitch, Δyaw). More intuitive for visual servoing.
• Impedance/Force Control: Desired forces and torques at the end-effector, crucial for dexterous manipulation and contact-rich tasks.

Design Choice: A visuomotor policy predicting in task space often generalizes better across robot morphologies than joint-space control.

Physical systems impose hard limits that must be baked into the action space definition or enforced during execution to prevent damage and ensure feasibility.

Common Constraints:

• Joint Limits: Minimum and maximum angles for each robotic joint.
• Velocity & Torque Limits: Maximum safe speeds and motor torques.
• Dynamic Feasibility: Actions must respect physics (e.g., a legged robot cannot arbitrarily change foot position mid-air).
• Collision Avoidance: Self-collision and environment collision constraints.

Implementation: Constraints can be enforced via clipping (post-processing), tanh/sigmoid activation functions (scaling network output), or through the use of a dynamics model in Model Predictive Control (MPC) to predict and avoid invalid states.

The observation space defines the set of all possible sensory inputs an agent can receive from its environment. It is the informational counterpart to the action space. In visuomotor control, this is often high-dimensional, consisting of raw pixels from cameras, depth sensors, or processed features. The structure of the observation space (e.g., image dimensions, proprioceptive data format) directly influences the policy architecture's design. A Partially Observable Markov Decision Process (POMDP) formalizes scenarios where the agent only receives partial, noisy observations of the true underlying state.

The state space represents the set of all possible configurations of the environment and the agent itself. It is the ground-truth information that fully describes the system at a given time. The agent's observation space provides a (often partial) view of this state. In robotics, the state might include the precise 3D pose of all objects and the robot's joint angles. The action space defines how the agent can transition from one state to another. Learning a world model often involves learning a compact, predictive representation of this state space.

A policy is the function or model that maps from the agent's observations (or states) to actions in the action space. It defines the agent's behavior. In visuomotor control, this is typically a visuomotor policy—a neural network that consumes images and outputs motor commands.

• Deterministic Policy: Outputs a specific action for a given input.
• Stochastic Policy: Outputs a probability distribution over the action space.
• Hierarchical Policy: Decomposes complex tasks, where a high-level policy selects sub-goals and a low-level policy executes primitive actions.

A Markov Decision Process is the foundational mathematical framework for sequential decision-making. It formally defines the interaction loop between an agent and its environment. The core components are:

• State Space (S): All possible states.
• Action Space (A): All possible actions.
• Transition Function (T): Probability of moving to a new state given a state and action.
• Reward Function (R): Scalar feedback for taking an action in a state. The agent's goal is to learn a policy that maximizes cumulative reward by selecting actions from A. A Partially Observable MDP (POMDP) extends this when the agent cannot see the full state.

The reward function provides a scalar signal that defines the task's objective, guiding the agent's exploration of the action space. It is a critical component of the Markov Decision Process. In reinforcement learning for robotics, designing a reward function that is dense enough to guide learning but aligned with the true goal is a major challenge. Inverse Reinforcement Learning (IRL) is the technique of inferring a reward function from expert demonstrations, sidestepping manual reward engineering. The reward signal directly shapes the policy gradient used to update the agent.

A trajectory (or episode) is a sequence of states, actions, and rewards experienced by an agent over time: τ = (s₀, a₀, r₀, s₁, a₁, r₁, ...). It represents one complete instance of the agent interacting with the environment using its policy to select actions from the action space. Trajectories are the fundamental data unit for training in both imitation learning (where they are expert demonstrations) and reinforcement learning (where they are collected via agent exploration). They are stored in an experience replay buffer for off-policy learning algorithms.