Reflexion

The core architecture separates the Actor (which generates actions and reasoning) from the Evaluator (which provides feedback). The Actor produces a trajectory using an LLM, and the Evaluator scores the outcome. On failure, the Evaluator's scalar reward or textual critique is stored in episodic memory, enabling the Actor to self-correct on the next trial without gradient updates.

Reflexion maintains a persistent episodic memory that stores reflective summaries of past failures. Unlike short-term context windows, this buffer retains lessons across multiple independent trials. Each entry captures:

• The original reasoning trajectory
• The failure signal from the environment
• A verbal self-reflection diagnosing the error This allows the agent to accumulate experiential knowledge over long horizons.

Instead of updating model weights through traditional RL, Reflexion uses natural language as the reinforcement signal. The Evaluator generates a textual critique explaining why a trajectory failed. This verbal feedback is more sample-efficient than scalar rewards alone, as it provides dense, semantic guidance that the Actor can parse and act upon in subsequent prompts.

The reflection generation process follows a structured heuristic. When a task fails, the agent is prompted to analyze:

• What went wrong in the reasoning chain
• Which specific action caused the failure
• What alternative strategy would avoid the error This structured introspection produces actionable insights that are appended to the agent's working memory for the next attempt.

A defining characteristic of Reflexion is that it improves performance without any gradient-based optimization. The underlying LLM weights remain frozen. All learning occurs through in-context retrieval of past reflections. This makes the framework lightweight, model-agnostic, and immediately deployable with any sufficiently capable language model, avoiding the compute cost of RLHF or fine-tuning.

Reflexion was originally demonstrated on sequential decision-making benchmarks including:

• AlfWorld: Embodied household tasks requiring multi-step planning
• HotPotQA: Multi-hop question answering over Wikipedia
• HumanEval: Code generation with unit test feedback In each domain, the agent iteratively refines its policy by reflecting on execution failures, achieving significant gains over baseline ReAct agents.

The paradigm underlying Reflexion where an agent's textual self-critique serves as a reward signal. Unlike traditional RL that uses scalar rewards, verbal RL uses linguistic feedback stored in episodic memory to guide policy improvement. This enables agents to learn from sparse failure signals—like a compiler error or a unit test result—by generating a reflective text summary of what went wrong and why, which is then conditioned on in future trials.

A memory architecture that stores trajectories of agent experiences—sequences of actions, observations, and outcomes. In Reflexion, the agent writes its self-critique into an episodic memory buffer after each trial. On subsequent attempts, the agent retrieves relevant past reflections to avoid repeating mistakes. This differs from semantic memory (facts) and procedural memory (skills), focusing instead on temporally-bound learning events that capture the context of failure.

An iterative prompting framework where a single LLM generates an initial output, provides its own feedback, and uses that critique to produce a refined version. Self-Refine operates within a single context window without persistent memory, whereas Reflexion maintains a growing episodic memory across multiple trials. Both share the core mechanism of linguistic self-feedback as a driver of improvement, but Reflexion adds a long-term learning loop.

A prompting paradigm that interleaves reasoning traces and action steps, enabling an LLM to generate task-specific actions and process external observations in a synergistic loop. Reflexion builds on ReAct by adding a post-hoc evaluation phase: after a ReAct trajectory completes, the agent reflects on the full sequence to generate a verbal critique. This critique then informs the reasoning in future ReAct trajectories.

A training methodology that provides feedback on each intermediate step of a reasoning chain, rewarding correct logical progression rather than just the final outcome. Reflexion's self-generated critiques function as a form of autonomous process supervision—the agent learns to identify which specific step in its reasoning led to failure. This contrasts with outcome supervision, where only the final answer correctness is evaluated.

A reasoning trace that accurately reflects the true causal process by which the model arrived at its answer, free from post-hoc rationalization. Reflexion's requirement that the agent critique its own reasoning chain creates pressure toward faithfulness—the agent must accurately report what it did to identify the error. This helps combat the Clever Hans effect, where models rely on spurious correlations for wrong reasons.