Success weighted by Path Length (SPL)

SPL is formally defined as: SPL = (1/N) Σ_i^N S_i * (l_i / max(p_i, l_i)). Where:

• N is the number of evaluation episodes.
• S_i ∈ {0,1} is a binary success indicator for episode i.
• l_i is the length of the shortest path (optimal path) from start to goal.
• p_i is the length of the path actually taken by the agent.

The term l_i / max(p_i, l_i) is the path efficiency ratio, which is 1.0 only if the agent's path equals the optimal length, and decreases as the agent's path grows longer.

SPL produces a single scalar score between 0.0 and 1.0 that holistically evaluates an agent.

• SPL = 1.0: The agent succeeded in every episode and took the optimal path in each case (perfect performance).
• SPL = 0.0: The agent failed every episode, or succeeded but took infinitely long paths.
• SPL = Success Rate: If all successful agents took the optimal path, SPL equals the simple success rate.
• SPL < Success Rate: This indicates successful agents are taking sub-optimal, longer paths. The magnitude of the difference quantifies the average efficiency penalty.

SPL was introduced to address critical flaws in using success rate alone for navigation:

• Penalizes Inefficiency: An agent that wanders randomly until eventually stumbling upon the goal would score a 1.0 in success rate. SPL heavily penalizes this behavior.
• Encourages Optimality: It provides a direct training signal or evaluation criterion that rewards finding shorter paths.
• Real-World Relevance: For physical robots, path length correlates with time, energy consumption, and operational cost. SPL measures practical utility.
• Granular Differentiation: It allows for ranking agents with identical success rates based on their average path efficiency.

While standard, SPL has known limitations that researchers account for:

• Dependence on Optimal Path: Requires a pre-computed shortest path (l_i), which can be ambiguous in continuous spaces or complex environments.
• No Partial Credit: An agent that stops 1cm from the goal scores the same (S_i=0) as one that gets completely lost. Metrics like Success weighted by Distance (SD) or Normalized Inverse Path Length (NIPL) offer alternatives.
• Ignores Execution Dynamics: Does not consider smoothness of motion, collisions, or time taken (only path length).
• Sensitive to Stop Condition: The definition of 'success' (e.g., proximity threshold) directly impacts the score. Results are only comparable under identical thresholds.

Implementing SPL evaluation involves specific steps:

1. Episode Definition: Each test case is a tuple: (Start Pose, Goal Specification, Instruction).
2. Optimal Path Pre-computation: Using a planner (e.g., Dijkstra's algorithm) on a known environment graph to find l_i.
3. Agent Execution: Run the agent's policy to produce its trajectory, recording the path length p_i.
4. Success Determination: Check if the agent's final position is within a threshold distance (e.g., 3 meters) of the goal.
5. Aggregation: Calculate the mean SPL across all episodes in the validation or test set (e.g., on the Room-to-Room (R2R) dataset).

SPL is often reported alongside other metrics to give a fuller performance picture:

• Success Rate (SR): The fraction of episodes where the goal was reached. Provides the upper bound for SPL.
• Oracle Success Rate: The fraction of episodes where the agent ever passed within the success threshold during its trajectory, measuring navigation potential.
• Path Length (PL): The average length of trajectories, reported separately for successful episodes.
• Normalized Dynamic Time Warping (NDTW): Measures the spatial similarity of the agent's path to the reference path, offering partial credit for getting close.
• Goal Progress (GP): Measures the average reduction in distance to the goal over the trajectory.

SPL is used to evaluate the navigation performance of Cross-Modal Transformer architectures and Vision-Language-Action Models. It provides a single, comparable figure that reflects how well the model's integrated visual and linguistic understanding translates into effective physical (or simulated) movement. It answers: 'Does better cross-modal alignment lead to more efficient action?'

Researchers use SPL to measure an agent's ability in zero-shot navigation and instruction decomposition. A high SPL on unseen environments or novel instructions indicates strong generalization. It also helps evaluate hierarchical planners: a low SPL may indicate failures in waypoint prediction or high-level planning, even if the low-level control is competent.

Vision-and-Language Navigation (VLN) is the core task that SPL evaluates. An embodied agent must follow a natural language instruction to navigate through a real or simulated 3D environment, using only egocentric visual perception. SPL directly measures the efficiency and success of an agent's performance on this task.

• Primary Benchmark: The Room-to-Room (R2R) dataset is the foundational VLN benchmark.
• Agent Input: A natural language instruction and a stream of first-person visual observations.
• Agent Output: A sequence of low-level navigation actions (e.g., move_forward, turn_left).

The Room-to-Room (R2R) dataset is the standard benchmark environment for evaluating VLN agents and calculating SPL. It provides the ground-truth data necessary for the metric.

• Environment: Uses Matterport3D 3D reconstructions of real building interiors.
• Data Structure: Contains human-annotated trajectory-instruction pairs.
• SPL Calculation: For each instruction, the dataset provides the expert's shortest path length, which is the L* in the SPL formula, against which the agent's actual path length (L) is compared.

Embodied Instruction Following is a broader category of tasks that includes VLN but extends to interactive object manipulation. SPL is specifically designed for the navigation component of these tasks.

• Scope: Includes navigation-to-and-interaction, as seen in benchmarks like ALFRED.
• Task Complexity: Instructions are often long-horizon and compositional (e.g., 'Go to the kitchen, pick up the mug, and place it on the table').
• Metric Adaptation: While SPL measures navigation efficiency, success in these tasks may require additional metrics for interaction success.

Success Rate (SR) is the foundational component of the SPL metric. It is a binary measure of whether an agent successfully completed its navigation task within a specified distance of the goal.

• Calculation: SR = (Number of Successful Episodes) / (Total Episodes).
• Relationship to SPL: SPL refines SR by incorporating path efficiency. An agent with a high SR but inefficient paths will have a significantly lower SPL.
• Limitation: SR alone does not penalize meandering, long paths that eventually reach the goal, which is why SPL was introduced.

Normalized Dynamic Time Warping (nDTW) is an alternative or complementary metric to SPL that measures the similarity of the agent's path to the optimal path, rather than just its final success and length.

• Core Idea: Measures the spatial alignment between two paths over time, forgiving small temporal deviations.
• Advantage over SPL: Can assign partial credit for paths that come close to the goal or follow the general route but fail to stop exactly at the target.
• Use Case: Often reported alongside SPL to provide a more nuanced view of path quality, especially in REVERIE-style tasks where the goal is an object, not a point.

Path Length (L) and Shortest Path Length (L)* are the two geometric variables in the SPL formula. L is the agent's actual traveled distance, while L* is the length of the optimal path, as determined by an oracle or planner.

• L (Agent Path Length): The cumulative distance of all steps taken by the agent during an episode.
• *L (Shortest Path Length)**: The minimum feasible distance between the start and goal points in the environment. This is a property of the environment and instruction, not the agent.
• SPL Formula: SPL = S * (L* / max(L, L*)). The ratio L*/L is the path efficiency penalty, which reduces the success credit S for suboptimal paths.