Trajectory

The state sequence is the ordered list of observations the agent receives from the environment. In robotics, a state can be high-dimensional, including:

• Proprioceptive data: Joint angles, velocities, and motor torques.
• Exteroceptive data: Camera images, LiDAR point clouds, or depth sensor readings.
• Task-specific variables: The position of a target object or the remaining time in an episode. The sequence (s₀, s₁, ..., s_T) captures the environment's evolution, forming the primary observational input for the agent's policy and value functions.

The action sequence comprises the decisions made by the agent's policy at each timestep. For robotic control, actions are typically continuous vectors specifying:

• Joint position or torque targets for a manipulator arm.
• Velocity commands for a mobile robot base.
• Gripper open/close signals. The sequence (a₀, a₁, ..., a_{T-1}) represents the agent's direct influence on the environment. The policy's role is to map from the current state s_t to an action a_t that maximizes cumulative reward.

The reward signal r_t is a scalar feedback provided by the environment after each action. It quantifies the desirability of the transition from state s_t to s_{t+1} after taking action a_t. In robotics, rewards are carefully engineered to guide learning:

• Sparse rewards: A large positive reward only upon task completion (e.g., +1000 for inserting a peg).
• Dense rewards: Shaped feedback at every step (e.g., -distance to goal, -effort penalty).
• Composite rewards: A weighted sum of multiple objectives (e.g., α * (progress) - β * (energy_used)). The trajectory's total return, G_t = Σ γ^k * r_{t+k}, is the ultimate objective the agent seeks to maximize.

The terminal condition defines when a trajectory ends. It is a crucial component for structuring the learning problem and defining an episode's horizon. Common terminal conditions in robotic RL include:

• Task Success: The goal is achieved (e.g., object grasped, door opened).
• Task Failure: An irreversible error occurs (e.g., robot falls over, object dropped).
• Timeout: A maximum episode length T_max is reached.
• Constraint Violation: A safety limit is breached (e.g., joint torque exceeds a threshold). Terminal states are often absorbing, meaning no further actions or rewards occur. Distinguishing between terminal states due to success or failure is vital for credit assignment.

A trajectory is fundamentally decomposed into a sequence of transition tuples (s_t, a_t, r_t, s_{t+1}, done_t). This atomic unit of experience is the core data structure for most RL algorithms.

• s_t, a_t, r_t, s_{t+1}: Used in Temporal Difference (TD) learning and Q-Learning to update value functions via the Bellman equation.
• done_t: A boolean flag indicating if s_{t+1} is terminal. These tuples are stored in a replay buffer for off-policy algorithms like DDPG and SAC, enabling sample-efficient learning by breaking temporal correlations through random minibatch sampling from past experiences.

The return G_t is the total discounted future reward from timestep t onward: G_t = r_t + γr_{t+1} + γ²r_{t+2} + ... + γ^{T-t-1}r_{T-1}.

• Discount factor (γ): A value between 0 and 1 that determines the present value of future rewards. A γ near 1 makes the agent far-sighted, while a γ near 0 makes it short-sighted.
• Undiscounted return: Used in episodic tasks with a natural horizon (γ=1). The value function V(s) or Q(s,a) is an estimate of the expected return, and the entire trajectory is used to compute or refine these estimates, either through Monte Carlo methods (using the full return) or TD methods (using bootstrapped estimates).

An episode is synonymous with a trajectory, representing one complete trial from an initial state to a terminal state. It is the atomic unit for Monte Carlo methods in RL, where the total return is calculated at the end. Episodes can be of fixed length or terminate upon reaching a goal or failure condition.

• Key Distinction: While 'trajectory' emphasizes the sequence of transitions, 'episode' often refers to the entire experiential unit, including its final outcome and return.

A rollout refers to the process of executing a policy in an environment to generate a trajectory. It is both a noun (the resulting data) and a verb (the act of collection). In model-based RL, rollouts are generated from a learned dynamics model for planning. In on-policy algorithms like PPO, each training step involves collecting a batch of fresh rollouts from the current policy.

• Primary Use: Generating data for policy evaluation, planning, or training updates.

An experience tuple (or transition) is the fundamental data point within a trajectory, typically represented as (s, a, r, s').

• State (s): The observation at time t.
• Action (a): The action taken.
• Reward (r): The scalar feedback received.
• Next State (s'): The resulting observation.

These tuples are stored in a replay buffer for off-policy learning (e.g., DDPG, SAC), where they are sampled randomly to break temporal correlations and improve sample efficiency.

The return (G) is the total discounted future reward accumulated over a trajectory. For a trajectory starting at time t, it's calculated as: G_t = r_t + γr_{t+1} + γ²r_{t+2} + ... where γ (gamma) is the discount factor between 0 and 1.

• Purpose: It defines the objective the agent aims to maximize. The value function V(s) is the expected return starting from state s.
• Credit Assignment: Analyzing which actions in a trajectory contributed to a high or low return is a core challenge in RL.

The horizon defines the temporal scope of a trajectory. It can be:

• Finite Horizon: The trajectory has a maximum fixed number of steps (e.g., 1000). Common in episodic tasks.
• Infinite Horizon: Tasks continue indefinitely, requiring a discount factor γ < 1 to ensure the return is finite. Common in continuing tasks.

Effective Horizon: The number of steps into the future where rewards significantly impact the current decision, roughly 1/(1-γ). A high gamma (e.g., 0.99) implies a long effective horizon (~100 steps), encouraging long-term planning.

These are two fundamental approaches for learning from trajectories:

• Monte Carlo (MC) Methods: Learn directly from complete trajectories. They wait until the end of an episode, calculate the actual return G_t, and use it to update value estimates. High variance, zero bias.
• Temporal Difference (TD) Methods: Learn from incomplete trajectories. They update estimates based on the observed reward and the current estimate of the next state's value (a bootstrapped estimate). Lower variance, some bias. TD(λ) provides a spectrum between pure TD (one-step) and MC (full-trajectory) updates.

This trade-off is central to value function learning and policy evaluation.