Reflexion

Reflexion operates through a modular feedback loop that separates task execution from self-critique:

• Actor: A language model that generates actions and reasoning traces in an environment (e.g., code generation, tool use, or question answering).
• Evaluator: The same or a separate model instance that inspects the actor's output, comparing it against environmental feedback or internal consistency checks.
• The evaluator produces a verbal reinforcement signal—a structured critique that identifies specific errors, explains why they occurred, and suggests corrective strategies.
• This loop mirrors the actor-critic architecture in reinforcement learning but replaces scalar rewards with semantically rich natural language feedback.

Reflexion stores self-generated critiques in a persistent episodic memory that accumulates across trials:

• After each failed attempt, the verbal feedback is serialized and appended to a sliding window of past experiences.
• On subsequent trials, the actor retrieves this memory and conditions its generation on both the current task and the history of prior mistakes.
• This creates a form of in-context learning from failure—the model sees what went wrong before and adjusts its strategy without gradient updates.
• The memory buffer typically retains the last 3–5 experiences to stay within context window limits while providing sufficient corrective signal.
• This mechanism is distinct from short-term memory (current trajectory) and long-term memory (external vector stores); it is a dedicated trial-level learning cache.

The evaluator module relies on heuristic signals rather than human labels to generate feedback:

• Binary Success/Failure: For code generation, unit tests provide an unambiguous pass/fail signal. The evaluator uses the test output to pinpoint which function or line caused the failure.
• Internal Consistency: For reasoning tasks, the evaluator checks for logical contradictions, missing steps, or factual inconsistencies within the generated chain-of-thought.
• Environmental Feedback: In embodied or tool-use scenarios, error messages, API responses, or simulation states serve as grounding signals.
• The evaluator synthesizes these heuristics into a structured critique containing: (1) the identified error, (2) the root cause hypothesis, and (3) a suggested remediation.
• This approach avoids the credit assignment problem by using language to explicitly attribute failure to specific reasoning steps.

Reflexion improves decision-making in sequential environments by adding a reflection token to the action space:

• In the AlfWorld embodied reasoning benchmark, the agent can choose to reflect instead of acting, triggering a self-evaluation of its current trajectory.
• This reflection is stored in memory and conditions future actions, enabling the agent to recover from dead-end states without external intervention.
• The mechanism is particularly effective for sparse reward environments where failure signals arrive only after long action sequences.
• By verbalizing what went wrong, the agent performs credit assignment in natural language, attributing failure to specific earlier decisions.
• This converts a reinforcement learning problem into a language-conditioned policy optimization problem, where the policy is guided by self-generated textual critiques.

Reflexion achieves state-of-the-art results on code generation benchmarks by treating compiler errors and test failures as evaluation signals:

• The actor generates a code solution; the evaluator executes it against a test suite.
• On failure, the evaluator produces a reflection containing: the specific test case that failed, the error traceback, and a hypothesis about the bug's root cause.
• The actor then regenerates the solution conditioned on the original problem statement plus the accumulated reflections from prior failed attempts.
• This creates a self-improving loop where each iteration benefits from the explicit diagnosis of previous mistakes.
• On the HumanEval benchmark, Reflexion with GPT-4 achieves 91% Pass@1 accuracy, surpassing prior methods that required multiple samples or external retrieval.

While both techniques improve reasoning, Reflexion and Chain-of-Thought (CoT) operate at different levels of the agent architecture:

• Chain-of-Thought generates intermediate reasoning steps within a single forward pass, improving the model's ability to solve complex problems through decomposition.
• Reflexion operates across multiple trials, using the output of one trial to improve the next. It is a meta-cognitive wrapper around the base reasoning process.
• CoT addresses intra-episode reasoning; Reflexion addresses inter-episode learning.
• The two techniques are complementary: an agent can use CoT to generate detailed reasoning traces and Reflexion to critique and refine those traces across attempts.
• Reflexion can also be combined with Self-Consistency by sampling multiple reflected trajectories and marginalizing over their outputs.