Voyager: An Open-Ended Embodied Agent with Large Language Models

Wang and colleagues' agent plays Minecraft through code, keeping every program that GPT-4 writes and a second GPT-4 call verifies in a skill library keyed by an embedding of the program's description, retrieving the top five skills into each new code-writing prompt, and choosing its own next task with a GPT-4 curriculum; over three runs of 160 prompting iterations it collected about three times as many distinct items as the best re-implemented baseline and reached diamond tools once, and in ablations removing the curriculum or self-verification cut items by 93% and 73%.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar (NVIDIA, Caltech, UT Austin, Stanford, UW Madison) · 2023-05-25 · original

What should an agent remember from its own experience if it is to keep getting more capable in an open world? Voyager's answer is: the programs that worked. The agent acts in Minecraft by writing JavaScript against a game-control library; when a program achieves its task, as judged by a second GPT-4 call, it is stored in a skill library under an embedding of a short description of what it does, and later code-writing prompts include the five most similar stored skills, which new programs can call. A GPT-4 automatic curriculum proposes each next task from the agent's state and its record of completed and failed tasks, under the instruction to "discover as many diverse things as possible". Over three runs of 160 prompting iterations each, Voyager collected 63 distinct items (the paper's figure), about 3.3 times the best baseline, and in one of three runs reached diamond tools, which no baseline reached (Section 3.3, Table 1). In a new world with an emptied inventory, Voyager with its learned library solved four unseen tasks in every run, in fewer iterations than Voyager without the library (which solved 11 of 12 runs), and the library gave a re-implemented AutoGPT baseline partial success where it had none (Table 2).

Version of record. The paper was published in Transactions on Machine Learning Research (TMLR) in 2024 (OpenReview id ehfRiF0R3a; the TMLR listing gives March 2024). It was first posted to arXiv (2305.16291) on 25 May 2023, the date given above; v2 followed on 19 October 2023. The OpenReview copy could not be retrieved (scripted access was refused), so this note follows arXiv v2, read with its LaTeX source; differences from the TMLR text, if any, are not known. Affiliations are as listed on the arXiv paper. Code, prompts and the learned skill library are released at github.com/MineDojo/Voyager (named in the paper; not read for this note).

Terms used in this note

  • Prompting iteration. One call to GPT-4 to generate code; the paper's unit of time for every curve and table.
  • Skill. An executable JavaScript function, written by GPT-4, that completed a task proposed by the curriculum (for example craftStoneShovel()), stored with a model-written description.
  • Control primitive APIs. Functions the authors wrote (such as mineBlock, craftItem, smeltItem, killMob, exploreUntil) and Mineflayer library calls (pathfinding, equipping, placing), which generated code may call (Appendix A.4.1).
  • Self-verification. A separate GPT-4 call that receives the agent's state and the task, decides whether the task was completed, and, if not, writes a critique (Section 2.3).
  • Lifelong learning. The authors' term for continually acquiring and reusing skills during one long run. Their claim that composing skills "alleviates catastrophic forgetting" (Section 1) is an argument; no forgetting experiment is reported.

How the memory works

What is stored, and when it is written

The skill library is a vector database. "The key is the embedding vector of the program description (generated by GPT-3.5), while the value is the program itself" (Figure 4 caption). A program is added only at one moment: when self-verification judges that it completed its task. The loop (Section 2.3, Appendix A.1 pseudocode) is:

  1. The curriculum proposes a task.
  2. Up to four rounds: retrieve skills, generate code, execute it in the game, collect environment feedback and execution errors, ask self-verification whether the task succeeded.
  3. On success, add the program to the skill library and the task to the completed list; otherwise add the task to the failed list and ask the curriculum for another.

Two other records persist. The curriculum keeps the lists of completed and failed tasks, which enter its prompt as the agent's "exploration progress". And the game state itself (inventory, nearby blocks, position) is read fresh each time, not stored as memory. Failed programs are not stored, and stored skills are not edited later in the paper's description.

How it is retrieved and scored

When a new task arrives, GPT-3.5 is asked for "a general suggestion for solving the task", which "is combined with environment feedback as the query context"; the query's embedding is matched against the skill keys, and the top five skills are retrieved (Figure 4 caption, Section 2.2). Embeddings come from OpenAI's text-embedding-ada-002 (Section 3.1). Ranking is by embedding similarity alone, with no recency or usage count.

The authors measured retrieval on 309 samples (Appendix B.4.4, Table A.4): the needed skill was the top result 80.2% ± 3.0 of the time, within the top two 89.3% ± 1.8, top three 93.2% ± 0.7, top four 95.2% ± 1.8 and top five 96.5% ± 0.3. How the samples were labeled, and over what the ± is computed, is not described.

What enters the context

The code-generation prompt to GPT-4 contains (Section 2.2, Appendix A.4.1): guidelines for writing reusable code ("Your function will be reused for building more complex functions. Therefore, you should make it generic and reusable."); the control primitive APIs; the retrieved skills, as code; the previous round's code, the environment feedback (chat messages the primitives emit, such as "I cannot make an iron chestplate because I need: 7 more iron ingots"), execution errors and the critique; the agent's state; the task and a GPT-3.5 suggestion for it; and an instruction to reason before writing code. Retrieved skills are given as callable code, so a new program can call them; the library becomes hierarchical in that sense.

The automatic curriculum

The curriculum prompt (Section 2.1, Appendix A.3, Prompt 1) combines directives, the agent's state (inventory, equipment, nearby blocks and entities, biome, time, health, hunger, position, known chests), the completed and failed task lists, and question-answer pairs that GPT-3.5 generates and answers, optionally using a Minecraft wiki. A warm-up schedule adds parts of the state as the agent completes tasks: the core inventory, equipment, nearby blocks and position from the start, nearby entities after 5 completed tasks, the full inventory after 7, recently seen blocks and biome after 10, and health, hunger, time and the question-answer context after 15 (Appendix A.3.3, Table A.1). The authors describe the curriculum as "an in-context form of novelty search" (Section 1). It runs at temperature 0.1; every other call at temperature 0 (Section 3.1).

What the paper measures

Setup (Section 3.1, Appendix B.1). GPT-4 (gpt-4-0314) for code, curriculum and self-verification; GPT-3.5 (gpt-3.5-turbo-0301) for question answering, task suggestions and skill descriptions. The game runs in MineDojo with Mineflayer controls, so the agent acts through high-level APIs on a text description of the state, not on pixels; the authors exclude pixel-based Minecraft agents from comparison for this reason. When the bot dies it is resurrected near the ground with its inventory preserved. Each method was run three times for at most 160 prompting iterations.

Baselines (Section 3.2, Appendix B.2). There was no LLM agent for Minecraft to use off the shelf, so the authors re-implemented ReAct, Reflexion and AutoGPT in their environment, each given the task "explore the world and get as many items as possible" and the same observations; Reflexion was given execution errors and Voyager's self-verification module. None of the baselines has a skill library or a curriculum (Table A.3). These are the authors' adaptations of methods designed for text tasks, not the original systems.

Exploration

Voyager "discover[ed] 63 unique items within 160 prompting iterations, $3.3 \times$ many novel items compared to its counterparts" (Section 3.3, Figure 1). The per-run item lists in Appendix B.4.1 contain 63, 58 and 67 items for Voyager, 11, 22 and 25 for AutoGPT, 4, 4 and 2 for ReAct, and 5, 5 and 5 for Reflexion (our count of the lists). The means, 62.7 and 19.3, give a ratio of 3.2 (our arithmetic); the paper does not say how its 3.3 was computed.

Tech tree

Table 1 gives the mean prompting iterations to reach each tool level, with the number of runs (of three) that reached it:

Method Wooden Stone Iron Diamond
ReAct 0/3 0/3 0/3 0/3
Reflexion 0/3 0/3 0/3 0/3
AutoGPT 92 ± 72 (3/3) 94 ± 72 (3/3) 135 ± 103 (3/3) 0/3
Voyager without skill library 7 ± 2 (3/3) 9 ± 4 (3/3) 29 ± 11 (3/3) 0/3
Voyager 6 ± 2 (3/3) 11 ± 2 (3/3) 21 ± 7 (3/3) 102 (1/3)

The paper's "15.3×, 8.5× and 6.4× faster" are the ratios of AutoGPT's means to Voyager's (92/6, 94/11, 135/21; our check). Voyager without the skill library reached the first three levels in a similar number of iterations (7, 9 and 29 against 6, 11 and 21, with overlapping spreads); the library's clearest effect in this table is the one diamond run.

Map coverage

Voyager traversed distances 2.3 times longer than the baselines (Figure 7), measured by the smallest circle enclosing each trajectory (Appendix B.4.2).

Zero-shot generalization

In a new world with an empty inventory, each method was given four tasks it had not seen (diamond pickaxe, golden sword, lava bucket, compass), decomposed into subgoals by GPT-4, with at most 50 prompting iterations (Table 2):

Method Diamond pickaxe Golden sword Lava bucket Compass
ReAct, Reflexion, AutoGPT 0/3 0/3 0/3 0/3
AutoGPT with Voyager's skill library 39 (1/3) 30 (1/3) 0/3 30 (2/3)
Voyager without skill library 36 (2/3) 30 ± 9 (3/3) 27 ± 9 (3/3) 26 ± 3 (3/3)
Voyager 19 ± 3 (3/3) 18 ± 7 (3/3) 21 ± 5 (3/3) 18 ± 2 (3/3)

The library was learned in an earlier run of Voyager; the authors present AutoGPT's gain as evidence that the library is "a plug-and-play asset".

Ablations

Figure 9 plots distinct items against prompting iterations, with shaded bands; the number of runs per ablation is not stated (values read from the figure except where the text states them):

  • Curriculum. A random curriculum (drawn from 101 items Voyager had obtained) cut distinct items by 93% (text); a manual ten-step curriculum toward a diamond reached about 17 items by about iteration 50, where its solid line ends; the figure continues it as a flat dashed line, which the paper does not explain.
  • Skill library. Without it, about 36 items at 160 iterations against about 62, with a plateau in later iterations.
  • Feedback. Without self-verification, items fell by 73% (text; about 16 on the figure); without environment feedback, about 25; without execution errors, about 26.
  • Model. GPT-4 obtained 5.7 times as many items as GPT-3.5 for code generation (text; about 10 for GPT-3.5).

Appendix B.4.5 reports that rerunning with gpt-4-0613 gave "roughly the same" performance (Figure A.4).

Human feedback

Without vision, Voyager could build structures such as a Nether Portal and a house when a human acted as the critic or as the curriculum (Section 3.5, Figure 10). This is a demonstration, not a measured result.

The instructions used

The paper prints its six system prompts (Appendix A, Prompts 1 to 6): the curriculum, question asking and answering, code generation, skill description and self-verification. The curriculum's criteria include "The next task should not be too hard since I may not have the necessary resources or have learned enough skills to complete it yet" and an instruction to avoid tasks that cannot be verified from the player's status, with "Placing 4 torches" given as a task that would "require visual confirmation from the screen", followed by "All the placing, building, planting, and trading tasks should be avoided" (Prompt 1). The code-generation guidelines ask for generic, reusable functions (Prompt 4). The wording of the prompts was not varied; the ablations remove whole components.

Limitations and open questions

The authors list (Section 4):

  • Cost. GPT-4 is "15× more expensive than GPT-3.5", and GPT-3.5 could not do the code generation (Figure 9).
  • Inaccuracies. The agent sometimes gets stuck; self-verification sometimes fails, for example "not recognizing spider string as a success signal of beating a spider".
  • Hallucinations. The curriculum proposed items that do not exist in the game ("copper sword"); code used cobblestone as fuel or called functions that do not exist.

Further points from the paper's setup:

  • Three runs per method in the main comparison (Appendix B.4.1, Tables 1 and 2), with means and spreads; shaded bands in Figures 1 and 9 whose meaning is not defined; no significance tests.
  • Baselines are re-implementations adapted by the authors, given a single open-ended goal and no curriculum.
  • High-level actions and text state. Skills call authors' primitives that already mine, craft and smelt; the agent does not perceive pixels, and resurrection preserves inventory.
  • What the library does not do. Skills are never revised or removed in the paper's description, the library's growth is not reported, and whether stale or wrong skills accumulate is not measured.

Research questions

  1. How does retrieval quality bear on success? Logging, for each task, whether the needed skill was among the five retrieved and whether the task succeeded would relate Table A.4's accuracy to outcomes.
  2. Does the library degrade as it grows? Running far beyond 160 iterations and tracking library size, retrieval accuracy and success rate would show whether near-duplicate or wrong skills crowd out good ones.
  3. Would revising stored skills help? Allowing the agent to replace a skill when a retrieved one fails, and comparing with append-only storage, would test whether editing procedural memory pays.
  4. How much does self-verification err? Checking a sample of verdicts against the game state by hand would give the false-positive rate, which decides what enters the library.
  5. Does the curriculum's effect hold with a stronger baseline? Giving AutoGPT the same curriculum but no library, and Voyager's library but no curriculum, would separate the two components' contributions to exploration.

How it connects

  • Within this unit. The planned memory-for-language-model-agents can treat the skill library as procedural memory stored as retrievable code, distinct from stores of text about events or facts. This placement is the note's.
  • Among the notes written with it. Cognitive Architectures for Language Agents (CoALA) uses Voyager as its example of an agent with all four kinds of action, whose learning action adds "code-based grounding procedures" to procedural memory (CoALA Section 5, Table 2). Voyager uses Reflexion: Language Agents with Verbal Reinforcement Learning as a baseline and contrasts its self-verification with Reflexion's self-reflection. Generative Agents: Interactive Simulacra of Human Behavior stores descriptions of experience and retrieves by recency, importance and relevance; Voyager stores only verified programs and retrieves by relevance alone. MemGPT: Towards LLMs as Operating Systems lets the model choose when to search; in Voyager retrieval is a fixed step before every code generation. These comparisons are the note's, apart from CoALA's classification.
  • In the library and the course. Skill retrieval is dense retrieval over short model-written descriptions (Sentence and Passage Encoders), and because the key is a model-written description rather than the code, what can be found depends on how well that description states what the program does. The curriculum's appeal to novelty search connects to Quality Diversity: MAP-Elites and Novelty Search in the evolutionary computing course, and the library of reusable programs to Genetic Programming, where programs are evolved rather than written by a language model. Project EVE: Claude plays MMORPGs reports another set of game-playing agents. These connections are the note's.

References