Can an agent built on a frozen language model learn from its failures within a handful of attempts, without any weight update? Reflexion's answer is to turn the failure signal into text. After an attempt (a trial) ends in failure, a language model reads the attempt and writes a few sentences on what went wrong and what to do instead; these self-reflections are stored and given to the agent at the start of its next trial on the same task. The memory is small by design: at most one to three reflections are kept. With it, a ReAct agent in the ALFWorld household environments solved 130 of 134 tasks over repeated trials, where the same agent retrying without reflections stopped improving at about three quarters (Section 4.1, Figure 3). The paper calls this "verbal reinforcement learning"; it is learning across trials of the same task, and on two of the three task families the signal that a trial failed comes from the environment or the gold answer.
Version of record. The paper appeared in Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 8634 to 8652, with five authors. This note follows the proceedings PDF. It was first posted to arXiv (2303.11366) on 20 March 2023, the date given above, under another title ("Reflexion: an autonomous agent with dynamic memory and self-reflection"), with three authors and experiments on ALFWorld and HotPotQA only. arXiv v4 (10 October 2023) has the proceedings title, adds a sixth author (Edward Berman) and notes "a few additional experiments"; a word-level comparison of v4 with the proceedings PDF found no difference in the text other than the author list and the page headers. Versions 2 and 3 were not read. Code is released at github.com/noahshinn024/reflexion (named in the abstract; not read).
Terms used in this note
- Trial. One complete attempt at a task, producing a trajectory of actions and observations. Learning in Reflexion happens between trials on the same task, not across tasks.
- Actor, Evaluator, Self-Reflection model. The paper's three components, $M_a$, $M_e$ and $M_{sr}$ (Section 3). The Actor acts; the Evaluator scores a trajectory; the Self-Reflection model writes the reflection.
- Short-term and long-term memory. The paper's terms: the current trajectory is short-term memory, and the stored reflections are long-term memory (Section 3, "Memory"). The paper also calls the reflection buffer "episodic memory".
- "Verbal reinforcement learning", "semantic gradient". The authors' framing: the reflection "acts as a 'semantic' gradient signal" (Section 1). It is an analogy; no gradient is computed and no parameter changes.
- pass@1. Whether the single submitted program passes the benchmark's hidden tests. The authors argue Reflexion's programming results are eligible for pass@1 because the agent tests itself only with tests it wrote, never with the hidden ones (Sections 2 and 4.3).
How the memory works
What is stored, and when it is written
One thing is stored: a reflection, a short paragraph in the first person written by $M_{sr}$ after a trial is judged a failure. The appendix's ALFWorld example reads, in part, "I should have looked for the desklamp first, then looked for the mug" (Appendix B, Figure 5). The Self-Reflection model is given "a sparse reward signal, such as a binary success status (success/fail), the current trajectory, and its persistent memory" (Section 3). After each trial $t$, the reflection $sr_t$ is appended to the buffer mem, and the loop repeats until the Evaluator passes the trajectory or a maximum number of trials is reached (Algorithm 1).
What triggers a reflection depends on the Evaluator:
- ALFWorld (decision making): either a hand-written heuristic, which calls for a reflection "if the agent executes the same action and receives the same response for more than 3 cycles, or if the number of actions taken in the current environment exceeds 30", or a language model classifying the trajectory as success or failure (Section 4.1).
- HotPotQA (reasoning): exact match of the answer against the dataset's answer, "using the environment to give a binary success signal" (Section 4.2).
- Programming: the agent's own unit tests. A model generates tests with chain-of-thought prompting, syntactically invalid ones are filtered by trying to parse them, and up to six are sampled into a suite (Section 4.3).
How it is retrieved, and what enters the context
There is no retrieval step. The buffer is bounded by a maximum number of reflections, $\Omega$, "usually set to 1-3" (Section 3): the last three in ALFWorld and HotPotQA, one in programming. All stored reflections for the task are placed in the Actor's prompt, after the task's few-shot examples, together with the current trajectory. For programming the Actor's prompt follows the form instruction, previous implementation, unit-test feedback, self-reflection, then the instruction for the next implementation (Appendix C). The buffer is a sliding window, and the paper names "vector embedding databases or traditional SQL databases" as future extensions (Section 5).
Why the paper calls it reinforcement learning
The authors map the setup onto policy-based reinforcement learning: the policy is "an agent's memory encoding paired with a choice of LLM parameters", $\theta = \{M_a, mem\}$ (Section 1 and Algorithm 1), and only mem changes. They list as advantages that it needs no fine-tuning, allows feedback more specific than a scalar reward, and gives "a more explicit and interpretable form of episodic memory", and as disadvantages that it relies on the model's self-evaluation and has no "formal guarantee for success" (Section 1).
What the paper measures
Three task families, each comparing an agent with and without the reflection buffer, given the same number of trials. The paper reports no variance across runs for its main results.
ALFWorld: sequential decision making
Setup (Section 4.1). 134 ALFWorld text environments across six task types, following ReAct's setup, with ReAct as the Actor and the same two few-shot trajectories as ReAct "with GPT-3 for the LLM" (the exact model is not named further). The baseline also retries: when a reflection would be called for, it skips the reflection, resets and starts a new trial. So the comparison isolates the stored reflections, not the retries.
Results (Figure 3; trial-by-trial values read from the figure). Both agents start at about 0.63 of environments solved. ReAct with Reflexion and the heuristic Evaluator reached 130 of 134 (the text; about 0.97 on the figure); with the language-model Evaluator, about 0.94. ReAct alone stopped improving between trials 6 and 7, at about 0.75. The text says Reflexion learns "in 12 consecutive trials" and describes "a steady increase over the next 11 trials"; Figure 3a plots the success curves for trials 0 to 10 (eleven trials), while the Reflexion lines in Figure 3b, the failure classification, run to trial 11 (twelve trials). The authors' failure analysis attributes most baseline failures to the agent believing it holds an item it does not have, and reports that the ReAct-only agent converged at "a hallucination rate of 22%" (Section 4.1, Figure 3b).
HotPotQA: reasoning
Setup (Section 4.2). 100 HotPotQA questions (Figure 4 caption). Three Actors: chain-of-thought from the question alone (CoT, 6-shot), chain-of-thought given the gold supporting paragraphs (CoT (GT)), and ReAct with a Wikipedia API (2-shot); self-reflection prompts are 2-shot, memory holds 3 reflections. Between trials, the agent learns whether its answer exactly matched the gold answer. Reflexion agents retried a question until three consecutive failed attempts. The baselines, sampling at temperature 0.7, solved no question in later trials that they had failed in the first.
Results (Figure 4; values read from the figure, which prints none). ReAct with Reflexion rose from about 0.32 to about 0.54 over six trials; CoT with Reflexion from about 0.32 to about 0.40; the baselines stayed near 0.33. For CoT (GT), the text reports that the agent failed 39% of questions and that Reflexion improved its accuracy "by 14%".
The episodic-memory ablation (Figure 4c). To separate reflection from simply seeing the last attempt, the authors added the most recent trajectory to the CoT (GT) prompt ("EPM", episodic memory) and compared with EPM plus a reflection. Read from the figure: CoT (GT) alone about 0.61, with EPM about 0.66, with EPM and Reflexion about 0.74; the text states "an 8% absolute boost over the episodic memory learning advantage". This is the paper's most direct evidence that the written reflection adds something beyond replaying the failed trajectory.
Other models (Appendix A, Table 5, 100 questions): Reflexion raised accuracy for every model tested, for example CoT (GT) with gpt-4 from 0.68 to 0.80 and ReAct with gpt-3.5-turbo from 0.26 to 0.38. The main-text figures do not name the model they used.
Programming
Setup (Section 4.3). Python and Rust function writing on HumanEval and MBPP (Rust via the MultiPL-E translators), and LeetcodeHardGym, a new set of 40 hard LeetCode problems released after 8 October 2022, which the paper gives as GPT-4's pre-training cutoff. The Actor is GPT-4 (Tables 1 to 3).
Results (Table 1, pass@1, %):
| Benchmark | Previous best (as cited) | GPT-4 | Reflexion |
|---|---|---|---|
| HumanEval (Python) | 65.8 (CodeT + GPT-3.5) | 80.1 | 91.0 |
| HumanEval (Rust) | 60.0 | 68.0 | |
| MBPP (Python) | 67.7 (CodeT + Codex) | 80.1 | 77.1 |
| MBPP (Rust) | 70.9 | 75.4 | |
| LeetcodeHard (Python) | 7.5 | 15.0 |
Reflexion was worse than plain GPT-4 on MBPP Python. Table 2 explains this by the quality of the self-written tests: when all of the agent's tests passed, the program was nonetheless wrong with probability 0.16 on MBPP Python against 0.01 on HumanEval Python (the text gives 16.3% and 1.4%). A wrong program that passes weak tests is submitted early, and no reflection is written.
Ablation (Table 3, the 50 hardest HumanEval problems in Rust, GPT-4): base model 0.60; reflections without self-written tests 0.52; tests without reflections 0.60; both 0.68. Without tests the agent could not tell when to stop and made "harmful edits"; without reflections, test failures did not lead to useful fixes.
Where it did not help
- A weaker model. With starchat-beta on HumanEval Python, baseline and Reflexion both scored 0.26, averaged over 8 trials (Appendix A, Table 4); the authors conclude that specifying self-corrections "is an emergent quality of stronger, larger models".
- WebShop. A ReAct with Reflexion agent on 100 shopping tasks showed no improvement after four trials, and the runs were stopped; the authors report that it did not generate helpful reflections and conclude that Reflexion cannot solve tasks "that require a significant amount of diversity and exploration" (Appendix B.1, Figure 6).
The instructions used
The method is a set of prompts. The appendix prints the programming instructions, for example "Apply the necessary changes below by responding only with the improved body of the function" (Appendix C), and full example trajectories with reflections for ALFWorld and HotPotQA (Appendices B and D); the full prompts are in the repository. In the appendix as printed, the "Self-reflection instruction" for programming repeats the text of the Actor instruction, so the instruction actually given to the Self-Reflection model is not visible in the paper. Few-shot reflection examples are used for HotPotQA (2-shot). The wording of the reflection prompts was not varied.
Limitations and open questions
- The failure signal. In HotPotQA, whether a trial failed is decided by exact match against the gold answer, so the agent is told, between trials, that its answer is wrong. In ALFWorld it comes from a heuristic or a classifier, and in programming from self-written tests. The paper names reliance on "the power of the LLM's self-evaluation capabilities (or heuristics)" as a disadvantage (Section 1), but does not run HotPotQA without the gold-answer signal.
- Later work on that point. Huang et al. (2024, ICLR) argue that the HotPotQA improvement depends on these "oracle labels". On the same 100 questions, closed-book and with Reflexion's prompt, they report GPT-4 at 49.0% with standard prompting and 59.0% with label-guided self-correction, but 49.0% and then 43.0% after one and two rounds of self-correction without labels (their Tables 2 and 3);
gpt-3.5-turbo-0613scored 26.0%, 29.0% with labels and 25.0% after each round without them. Their setting differs from Reflexion's (closed-book, at most two rounds, no ReAct retrieval), and they test reasoning, not ALFWorld or programming; they write that external feedback such as code execution can support correction (their Section 6). - Within-task learning only. Reflections are written for and read on the same task; transfer to new tasks is not tested.
- Small, fixed memory. One to three reflections; the authors list richer stores as future work (Section 5).
- No variance. Main results are single curves or single numbers; the 8-trial average in Table 4 is the only repeated measurement reported.
- Inconsistencies in the text. The ALFWorld text says 12 trials; the success-rate panel (Figure 3a) plots trials 0 to 10, and only the failure-classification panel (Figure 3b) reaches trial 11, so the success rate after the eleventh trial is not shown. The programming analysis gives baseline pass@1 of "82% and 80%" for HumanEval and MBPP Python (Section 4.3), where Table 1 gives 80.1 for both and Table 2 gives 0.80 for both.
Research questions
- How much of the HotPotQA gain needs the gold answer? Running Reflexion with a model judge or a self-check in place of exact match, on the same questions, would measure the gain without oracle feedback.
- Do reflections transfer across tasks? Keeping reflections from solved ALFWorld tasks and retrieving them for new tasks of the same type would test whether the memory generalizes beyond the task it was written for.
- Does a larger, retrieved reflection memory help? Replacing the 1 to 3 entry window with an embedding-indexed store of all reflections, retrieved by task similarity, would test the authors' proposed extension.
- How often are reflections wrong? Labeling reflections for whether they identify the actual cause of failure, and relating that to success on the next trial, would show how reflection quality drives the effect.
- What variance do the curves hide? Repeating the ALFWorld and HotPotQA runs with several seeds would give intervals for the trial-by-trial gains.
How it connects
- Within this unit. For the planned memory-for-language-model-agents, Reflexion is the simplest case of a model writing its own memory: one kind of entry, written on failure, always in the prompt. This placement is the note's.
- Among the notes written with it. Generative Agents: Interactive Simulacra of Human Behavior (which this paper cites) also stores model-written reflections, but triggers them by accumulated importance and retrieves them from a large stream by score. Voyager: An Open-Ended Embodied Agent with Large Language Models uses Reflexion as a baseline, re-implemented with its own self-verification module, and calls its self-verification "more comprehensive than self-reflection" because it both checks success and critiques. Cognitive Architectures for Language Agents (CoALA), whose authors include two of this paper's (Yao and Narasimhan), classifies Reflexion's reflections as semantic knowledge written by a learning action, where Reflexion itself calls the buffer episodic memory; the two papers use the terms differently, and this note records both. MemGPT: Towards LLMs as Operating Systems keeps a small always-present block (working context) with large external stores behind it, which Reflexion lacks. The comparisons are the note's.
- In the library. The components of agent memory reports an Anthropic test in which telling Claude models to "record only failures" reduced their gain from memory on one task, a rule close to Reflexion's failure-triggered writing, though on a different task, model generation and memory design; this connection is the note's.
References
- Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pp. 8634 to 8652. arXiv:2303.11366 (v1 and v4 read; v4 lists Edward Berman as an additional author). record
- Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. (2024). Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR 2024). arXiv:2310.01798 (v2 read). record