State Representation

A state representation is sufficient if it contains all the information necessary to predict future states and rewards, given the agent's actions. This is formalized by the Markov Property, where the future is conditionally independent of the past given the present state. For example, in chess, a sufficient representation is the board's current configuration (piece positions), not the entire move history. Insufficient representations lead to Partially Observable Markov Decision Processes (POMDPs), requiring agents to maintain a belief state over history.

An effective representation is a compressed version of raw sensory data (e.g., pixels), discarding irrelevant noise and redundancy. This reduces the curse of dimensionality for planning algorithms. Techniques include:

• Autoencoders for learning low-dimensional latent codes.
• Disentangled representations that separate independent factors of variation (e.g., object position, color).
• Object-centric representations that encode scenes as sets of entities, promoting compositionality. The goal is to balance compression with the retention of task-relevant information, often guided by the information bottleneck principle.

The representation should evolve smoothly and predictably over time with the agent's actions. Small changes in the environment should correspond to small changes in the state vector (except for discrete events). This property is crucial for learning accurate dynamics models and for Model-Predictive Control (MPC). Architectures like Recurrent State-Space Models (RSSM) explicitly model temporal dynamics with deterministic and stochastic components. Inconsistent encodings make learning and planning unstable.

The representation should enable the agent to apply learned knowledge to new, unseen situations. This requires abstraction—ignoring superficial details to capture underlying structure. For instance, a representation for a navigation agent should encode the layout of walls, not their exact texture. Methods to promote this include:

• Learning bisimulation metrics that cluster states with similar long-term outcomes.
• Causal state representations that identify variables with true causal influence.
• Successor representations that capture expected future state occupancy.

The representation must be computable from the agent's available observations (sensors) in real-time. This is the domain of state estimation, using filters (e.g., Kalman, particle) to infer hidden state from noisy data. The computational cost of deriving the representation must be feasible for the agent's hardware, especially for embodied AI and real-time robotic perception. Representations that are theoretically ideal but prohibitively expensive to compute are impractical for deployment.

The representation should be structured to facilitate the specific decision-making tasks required. For reinforcement learning, this often means the state space should be amenable to value function approximation or direct policy learning. For Task and Motion Planning (TAMP), a hierarchical representation separating symbolic goals from geometric details is beneficial. This property underscores that there is no universally "best" representation; it is defined by its utility for prediction, planning, and control within the target domain.

In perfect-information games, the state representation is the complete board configuration. For AI systems like AlphaZero, this is encoded as a set of binary feature planes.

• Each plane represents the presence of a specific piece type (e.g., white pawns, black knights) on an 8x8 grid.
• Additional planes may encode game meta-information like castling rights or move count.

This compact, structured tensor (e.g., 19x8x8 for chess) allows the neural network to efficiently evaluate positions and plan moves via Monte Carlo Tree Search (MCTS), abstracting away the visual rendering of the board entirely.

In settings like chemical plants or power grids, the state is a curated set of telemetry signals from sensors and SCADA systems.

• Continuous variables: Temperatures, pressures, flow rates, voltage levels.
• Discrete states: Valve positions (open/closed), pump status (on/off/error).
• Derived features: Often, key performance indicators (KPIs) like efficiency or yield are calculated and included.

This structured vector, often normalized, is used by Model-Predictive Control (MPC) algorithms. The dynamics model predicts how these values will evolve over a horizon given control actions (e.g., adjusting a valve), allowing the system to optimize for stability and efficiency.

Trading algorithms construct state from high-frequency market data to inform buy/sell decisions. The representation is a temporal snapshot combining:

• Order book dynamics: Price levels and volumes for bids and asks up to a certain depth.
• Recent price history: A window of past mid-prices or returns, often as a technical indicator.
• Macro-features: Volatility indices, sector ETF movements, or news sentiment scores.

This multivariate time-series vector defines the "market state." A reinforcement learning agent uses this state to estimate the value of taking a trading action, learning a policy that maps complex, noisy financial signals to executable orders.

The mathematical foundation for sequential decision-making. An MDP formally defines the problem an agent solves, consisting of:

• State Space (S): All possible configurations of the environment.
• Action Space (A): All possible moves the agent can make.
• Transition Function T(s'|s,a): The probability of moving to state s' from state s after taking action a.
• Reward Function R(s,a,s'): The immediate feedback signal.

The state in an MDP is the ideal, fully observable representation the agent uses for optimal decision-making. State representation learning aims to construct a latent space that approximates this ideal MDP state from raw, high-dimensional observations.

The real-world extension of the MDP framework. In a POMDP, the agent cannot directly observe the true state. Instead, it receives an observation (e.g., a pixel image) that is a noisy, incomplete function of the state.

This makes state representation critical. The agent must maintain a belief state—a probability distribution over possible true states—by integrating its history of observations and actions. Techniques like Recurrent Neural Networks (RNNs) or Bayesian filters are used to learn or compute this belief state, which then serves as the input to the policy.

A learned internal simulator. A world model is an agent's learned representation of the environment's dynamics. It typically consists of two components learned jointly:

• Representation Model: Encodes the current observation (and past context) into a compact state vector.
• Dynamics Model: Predicts the next state vector given the current state and a proposed action.

World models enable planning and model-based reinforcement learning (MBRL). The agent can perform "imagined" rollouts inside its world model to evaluate actions without costly real-world interaction. The Recurrent State-Space Model (RSSM) used in the Dreamer algorithm is a prominent example.

A paradigm that leverages learned models. Unlike model-free RL, which learns a policy or value function directly from experience, MBRL algorithms first (or concurrently) learn a model of the environment—its dynamics and often its reward function.

This model is then used for:

• Planning: Searching for action sequences that maximize predicted reward (e.g., using Model-Predictive Control (MPC) or Monte Carlo Tree Search (MCTS)).
• Generating synthetic experience: Training a policy on "dream" trajectories from the model, dramatically improving sample efficiency.

The quality of the MBRL agent is directly limited by the accuracy and generality of its learned state representation and dynamics model.

A property of an ideal state vector. A representation is disentangled when individual latent dimensions correspond to single, semantically meaningful generative factors of the observed data.

For example, in a scene with multiple objects, a disentangled state might have separate dimensions for:

• An object's X position
• An object's Y position
• The object's color
• The object's shape

This property is highly desirable because it promotes interpretability, robustness, and compositional generalization. An agent can understand that changing one dimension (e.g., "position") affects the environment in a predictable, isolated way. Learning such representations is an active area of research using techniques like β-VAE.

A hybrid representation linking states to future states. The Successor Representation (SR) of a state s under a policy π is a vector where each component is the expected discounted future occupancy of a specific state (or feature).

It decomposes the value function: V(s) = SR(s) · w, where w is a vector of immediate rewards. This is powerful because:

• Fast Reward Re-valuation: If rewards change (w changes), the agent can recompute optimal behavior instantly without re-learning the SR.
• Transfer Learning: The SR captures the "geometry" of the environment under a policy, which can be reused for new tasks.

The SR is a form of state representation that explicitly encodes long-term predictive knowledge, not just the immediate present.