Observation Space

A core distinction is whether the observation space provides a full state or a partial observation of the environment.

• Fully Observable: The observation (e.g., a robot's joint angles and target position) contains all information needed to describe the environment's true state. This is formalized as a Markov Decision Process (MDP).
• Partially Observable: The agent receives only sensory clues (e.g., a single camera image) that are insufficient to determine the full underlying state. This is formalized as a Partially Observable Markov Decision Process (POMDP), requiring the agent to maintain an internal belief state over time.

Observations can range from highly structured numerical vectors to raw, high-dimensional sensory streams.

• Structured/Low-Dimensional: Examples include joint positions, lidar point clouds, or game scores. These are computationally efficient for traditional RL algorithms.
• Unstructured/High-Dimensional: Examples include raw RGB images, audio waveforms, or tactile sensor arrays. Processing these requires deep learning components like Convolutional Neural Networks (CNNs) to extract relevant features, forming an observation encoder.

Observations are categorized by their source relative to the agent's body.

• Proprioceptive Observations: Sense the internal state of the agent's own body. Examples: joint angles, motor torque, battery level, and velocity. These are typically low-dimensional and structured.
• Exteroceptive Observations: Sense the external world. Examples: camera images, microphone audio, and lidar scans. These are often high-dimensional and unstructured. Robust agents fuse both modalities for effective state estimation and control.

A single observation at time t is often insufficient. Agents must integrate observations over time to understand dynamics and overcome partial observability.

• Frame Stacking: A simple technique where the last k observations (e.g., video frames) are concatenated and presented as the current observation, providing implicit temporal information.
• Recurrent Networks: Architectures like LSTMs or GRUs are used within the policy network to maintain a hidden state, explicitly forming a memory over the observation history. This is critical for tasks like navigation where objects move in and out of view.

Raw observations are rarely fed directly into a learning algorithm. Standard preprocessing is critical for stable and efficient training.

• Normalization/Standardization: Scaling numerical values (e.g., joint angles) to have zero mean and unit variance prevents gradients from exploding and helps with generalization.
• Dimensionality Reduction: Techniques like PCA or learned encoders compress high-dimensional observations (e.g., images) into a compact latent representation.
• Data Augmentation: Applying random transformations (e.g., color jitter, cropping) to observations during training improves robustness and helps bridge the sim-to-real gap.

The observation space does not exist in isolation; its design is intrinsically linked to the action space and the underlying state space.

• Action Space Coupling: The observation must contain information relevant to choosing a good action. For example, a visual navigation agent's observation must contain cues about obstacle locations.
• State Inference Goal: In a POMDP, the agent's objective is to infer the true state from its observation history. The belief state is a probability distribution over the state space, conditioned on all past observations and actions. The complexity of this inference directly impacts learning difficulty.

A Discrete Observation Space consists of a finite set of distinct, countable states. The agent receives one of a predefined number of possible observations.

• Examples: The current room number in a grid-world, a specific game screen in a finite set of levels, or a categorical sensor reading (e.g., 'object present' or 'object absent').
• Model Impact: Often paired with tabular reinforcement learning methods or neural networks using embedding layers to convert the discrete index into a dense vector.
• Key Consideration: While simple, discrete spaces can suffer from the curse of dimensionality if the number of possible states is very large, making learning intractable.

A Continuous Observation Space, often implemented as a Box space, is defined by an n-dimensional array of real numbers, each bounded within a specified interval. This is the most common type for robotic sensors and physical simulations.

• Examples: Joint angles (radians), LiDAR point distances (meters), pixel intensities in an image (0-255), or the (x, y, z) position of an end-effector.
• Model Impact: Requires neural network architectures designed for continuous input, such as fully connected or convolutional layers. Normalization of inputs is often critical for stable training.
• Key Consideration: The dimensionality of the box (e.g., 64x64x3 for an RGB image) directly dictates the required network capacity and computational cost.

A Multi-Discrete Observation Space represents a vector where each component can take on a discrete value from its own independent set. It generalizes a single discrete space to multiple, simultaneous categorical observations.

• Examples: The status of multiple independent switches (each on/off), the discrete type of multiple objects in a scene, or a tuple representing (grid_x, grid_y) position.
• Model Impact: Each component is typically handled with its own embedding layer, and the resulting embeddings are concatenated before being passed to the policy network.
• Key Consideration: Useful for structured discrete data where the agent must reason about multiple independent categorical factors from a single observation.

A Dictionary Observation Space is a composite space that allows multiple, heterogeneous observation components to be grouped under named keys. Each key maps to its own subspace (e.g., Box, Discrete).

• Examples: A robot observation containing a {'camera': Box(shape=(84,84,3)), 'lidar': Box(shape=(360,)), 'joint_positions': Box(shape=(7,))}. This is standard in complex embodied AI setups.
• Model Impact: Requires a neural network with parallel processing streams (e.g., a CNN for camera, an MLP for lidar) whose outputs are fused (e.g., concatenated) before the policy head. Frameworks like RLlib and Stable-Baselines3 provide native support.
• Key Consideration: Essential for multimodal perception, enabling the agent to learn from complementary data types like vision, proprioception, and language instructions.

An Image Observation Space is a specific, high-dimensional instance of a Box space where the observations are pixel arrays, typically representing camera feeds or rendered frames from a simulator.

• Examples: An RGB image of shape (height, width, 3) with values 0-255, or a grayscale depth image of shape (height, width, 1).
• Model Impact: Mandates the use of convolutional neural networks (CNNs) or vision transformers (ViTs) within the policy to extract spatial features. This is computationally intensive but necessary for tasks requiring scene understanding.
• Key Consideration: Prone to the sim-to-real gap; visual features learned in simulation (lighting, textures) may not transfer directly to physical cameras. Techniques like domain randomization are applied to the observation space to mitigate this.

This is not a separate type but a critical property of an observation space. In a Partially Observable Markov Decision Process (POMDP), the observation is a function of the underlying state and does not contain full information.

• Fully Observable: The observation o_t equals the true environment state s_t. Common in board games (like Go) or full-state simulators (like MuJoCo providing all joint data).
• Partially Observable: The observation is incomplete. Examples include a single first-person camera view (hiding what's behind the agent), or a robot with limited sensor range.
• Model Impact: Partial observability requires the agent to maintain an internal state or memory (e.g., via Recurrent Neural Networks (RNNs) or Transformers) to integrate information over time, a process known as belief state estimation.

Defines the set of all possible control commands an agent can execute. It is the output counterpart to the observation space.

• Discrete Action Space: A finite set of distinct actions (e.g., {left, right, up, down}).
• Continuous Action Space: A real-valued vector space (e.g., [torque_joint_1, torque_joint_2] ∈ ℝ²). The policy function maps from the observation space to the action space.

Represents the complete, underlying configuration of the environment, which may be partially observable. The observation space is a (often noisy) projection of the true state space.

• Fully Observable: Observation space = State space (e.g., board games).
• Partially Observable (POMDP): Observation is an incomplete signal (e.g., robot with a single camera). The agent must use a belief state or memory to infer the true state from a history of observations.

A mathematical function R(s, a, s') that provides the agent with a scalar reward signal based on the state, action, and resulting next state. It defines the task's objective.

• Sparse Rewards: Given only upon task completion (e.g., +1 for winning).
• Dense Rewards: Provide frequent, incremental feedback (e.g., distance to goal). The agent's goal is to learn a policy that maximizes the cumulative discounted reward, despite only having access to observations.

Learned neural networks that internally simulate the dynamics of the environment. They compress the observation space into a compact latent state representation to enable prediction and planning.

• Dynamics Model: Predicts the next latent state given the current state and action.
• Observation Model: Decodes a predicted latent state back into the expected observation. Frameworks like Dreamer use world models to train agents purely in latent space, improving sample efficiency.

A technique where multiple independent environment instances run in parallel, producing a batch of observations for the agent at each step.

• Input: Becomes a tensor of shape (num_envs, *observation_space.shape).
• Purpose: Dramatically accelerates data collection by decorrelating samples and fully utilizing GPU/CPU resources.
• Implementation: Common in libraries like Gymnasium, Stable-Baselines3, and Ray RLlib for training robustness.

The process of training a policy in a simulated environment and deploying it on a physical robot. A major challenge is the reality gap—discrepancies between the simulation's and real world's observation spaces.

• Domain Randomization: Actively varies simulation parameters (e.g., lighting, textures, physics) during training to force the policy to be robust to a wide distribution of observations.
• Domain Adaptation: Uses techniques to align the simulated observation space with the real one, often using unlabeled real-world data.