Language-agent papers name the same parts differently ("tool use", "grounding", "actions", "memory"), which makes it hard to compare agents or build new ones from shared parts. This paper proposes a common vocabulary, CoALA (Cognitive Architectures for Language Agents), taken from the cognitive architectures of symbolic AI such as Soar. Under CoALA an agent has a working memory and three long-term memories (episodic, semantic and procedural); it acts through external actions on the world (grounding) and internal actions on its memories (retrieval, reasoning, learning); and a decision procedure runs a loop of cycles, each of which plans with retrieval and reasoning and then executes one grounding or learning action. The paper reports no experiments. Its contributions are the framework, a classification of five existing agents (Table 2), and a list of design recommendations the authors call "actionable insights" (Section 6). The claims are therefore arguments and classifications, to be judged by whether they describe agents accurately and usefully.
Version of record. The paper was published in Transactions on Machine Learning Research (02/2024, per the paper's header), reviewed on OpenReview (id 1i6ZCvflQJ), with the TMLR certifications "Survey" and "Featured". It was first posted to arXiv (2309.02427) on 5 September 2023, the date given above; arXiv v3 (15 March 2024) is described by its authors as "TMLR camera ready version", and this note follows it, read with its LaTeX source. Version 1 was compared for structure: it has the same sections and case studies, and v3 is longer. The arXiv comment states that "The first two authors contributed equally, order decided by coin flip." Two authors (Yao and Narasimhan) are also authors of ReAct and Reflexion, which the paper uses as examples.
Terms used in this note
All definitions are the paper's (Section 4).
- Language agent. A system that uses a large language model to interact with the world (Section 1). A cognitive language agent adds internal processes such as reasoning, planning and memory management (Figure 1C).
- Production system. A set of rules, each a precondition and an action; the paper's starting analogy is that a language model is a probabilistic production system, defining a distribution over continuations $X \rightarrow X\,Y$ of a prompt $X$ (Section 3.1).
- Working memory. It "maintains active and readily available information as symbolic variables for the current decision cycle": perceptual inputs, active knowledge from reasoning or retrieval, and information carried over, such as goals (Section 4.1).
- Episodic memory. It "stores experience from earlier decision cycles": input-output pairs, event histories, trajectories from earlier episodes.
- Semantic memory. It "stores an agent's knowledge about the world and itself", which may start from an external database and grow by writing in conclusions drawn by reasoning.
- Procedural memory. Two forms: "implicit knowledge stored in the LLM weights, and explicit knowledge written in the agent's code", the code covering both the action procedures and the decision procedure itself.
- Grounding, retrieval, reasoning, learning actions. Grounding acts on the external environment; retrieval reads from long-term memory into working memory; reasoning reads and writes working memory using the language model; learning writes to long-term memory (Section 4, Figure 5).
The framework
Where it comes from
Sections 2 and 3 trace a line from Post's production systems through Markov algorithms to Newell and Simon's production-rule agents and the Soar architecture, which has working memory, procedural, semantic and episodic long-term memories, grounding in sensors and actuators, a propose-evaluate-select decision cycle and several kinds of learning (Section 2.3, Figure 2). The authors argue that a language model "define[s] a distribution over changes or additions to text" as productions define possible rewrites, and that prompting methods (chain-of-thought, self-critique, selection-inference) amount to control flow over this probabilistic production system (Sections 1 and 3.2, Table 1). The analogy motivates borrowing Soar's organization for language agents; it is an argument from correspondence, not a tested claim.
Memory
CoALA starts from the observation that "Language models are stateless: they do not persist information across calls" (Section 4.1), so any state an agent keeps lives outside the model. The four memories differ in what they hold and how they change:
- Working memory is "a data structure that persists across LLM calls", more general than the model's own context. On each call the prompt is built from a subset of working memory (a template plus variables), and the model's output is parsed back into variables, such as an action name and arguments (Figure 3A). It is "the central hub" connecting the model, long-term memory and grounding.
- Episodic memory can be retrieved into working memory during planning and written from working memory as a learning action.
- Semantic memory is often read-only in earlier work (retrieval-augmented models reading Wikipedia are given as an example of retrieving from a semantic memory of unstructured text), but agents may also write new knowledge into it.
- Procedural memory "must be initialized by the designer". Writing to it is possible but, the authors argue, "significantly riskier than writing to episodic or semantic memory, as it can easily introduce bugs or allow an agent to subvert its designers' intentions" (Section 4.1).
Text is "the de facto internal representation", which the authors say simplifies memory compared with symbolic architectures (Section 4).
The action space
- Grounding (Section 4.2) covers physical environments, dialogue with humans or other agents, and digital environments (games, APIs, websites, code execution). Tools such as a search API or calculator are treated as "single-use" digital environments.
- Retrieval (Section 4.3) "reads information from long-term memories into working memory", by rule-based, sparse or dense methods. The paper's examples: Voyager loading code skills by dense retrieval from procedural memory, and Generative Agents retrieving events from episodic memory by recency (rule-based), importance (reasoning-based) and relevance (embedding-based). The authors write that "adaptive and context-specific recall remains understudied in language agents".
- Reasoning (Section 4.4) reads and writes working memory, for example summarizing an observation or a trajectory; its results can feed learning or decisions.
- Learning (Section 4.5) writes long-term memory. The paper lists, in this order: adding experiences to episodic memory; adding knowledge to semantic memory (Reflexion's reflections and Generative Agents' reflections are the examples); fine-tuning the model's weights (implicit procedural memory); and updating the agent's code, including prompts, grounding skills, retrieval procedures, and even learning or decision procedures. The only risk ranking the paper states is that writing to procedural memory is riskier than writing to episodic or semantic memory (Section 4.1), and it calls updates to learning or decision procedures "risky both for the agent's functionality and alignment". The authors found no language agent that learns new retrieval, learning or decision procedures, and note that "modifying and deleting (a case of 'unlearning') are understudied" (Section 4.5).
The decision cycle
The decision procedure is "the top-level or 'main' agent program", structured into cycles (Section 4.6, Figure 4B). In each cycle a planning stage uses reasoning and retrieval to propose candidate actions, evaluate them (by heuristic, model likelihood, learned values or model reasoning) and select one (argmax, softmax, majority vote) or loop back; an execution stage then performs the chosen grounding or learning action, and an observation may start the next cycle. Learning is thus an action the agent chooses, on a par with acting on the world. The authors observe that most agents so far propose a single action without evaluation, and that Tree of Thoughts and RAP are early examples of propose-evaluate-select planning (Section 4.6).
The case studies
Table 2 casts five agents into the framework (writable long-term memory only, since every agent has procedural memory in its code and weights):
| Agent | Long-term memory | Grounding | Internal actions | Decision making |
|---|---|---|---|---|
| SayCan | none | physical | none | evaluate |
| ReAct | none | digital | reason | propose |
| Voyager | procedural | digital | reason, retrieve, learn | propose |
| Generative Agents | episodic, semantic | digital, agent | reason, retrieve, learn | propose |
| Tree of Thoughts | none | digital (submit answer only) | reason | propose, evaluate, select |
The text (Section 5) describes Voyager's skill library as procedural memory of code-based grounding procedures, written by a learning action after the agent judges a task complete. It describes Generative Agents as writing reflections on episodic memory "to long-term semantic memory" and retrieving them when planning. Elsewhere it gives Reflexion's reflections as an example of storing inferences "as semantic knowledge" (Section 4.5). These are CoALA's classifications. The original papers organize memory differently: Generative Agents keeps observations, reflections and plans in one list scored by one formula, and Reflexion calls its buffer of reflections "episodic memory". In Section 4.5 the paper also says Voyager "maintains a curriculum library"; Voyager's own name for the store is the skill library.
The actionable insights
The authors' recommendations (Section 6), kept as their positions:
- Modular agents. "Perhaps our most important suggestion is that agents should be structured and modular", with shared abstractions (
Memory,Action,Agentclasses) as OpenAI Gym standardized reinforcement learning interfaces. They suggest using code "sparingly" for generic algorithms that complement the model, such as tree search. - Designing an agent. Decide which memories are needed, define read and write access to each, then define the decision procedure. Their worked example is a retail assistant with semantic memory of the catalog, episodic memory of each customer's history, write access only to episodic memory, and learning deferred to the end of each interaction by summarizing the episode before storing it.
- Long-term memory beyond retrieval augmentation. Retrieval-augmented models only read human-written corpora; agents can also write self-generated content, combine human knowledge with their own experience, and interleave retrieval with reasoning during planning.
- Learning. Meta-learning by modifying agent code, for example learning better retrieval procedures or storing metadata about when knowledge will be useful; unlearning by deleting memory items.
- Action space. Choose "the minimal action space necessary"; learning actions that delete or modify procedures can cause internal harm, grounding actions external harm, and the action space may need to be specified and ablated "for worst-case scenario prediction and prevention".
- Decision making. Mix language reasoning with code-based planning; extend deliberate planning beyond toy tasks; allocate computation adaptively ("metareasoning"); address calibration, alignment and hallucination in self-evaluation.
The discussion (Section 7) poses open questions: language-only or multimodal reasoning; where an agent ends and its environment begins (the authors propose controllability and coupling as criteria, so that an offline copy of Wikipedia only the agent can write to counts as internal memory while the live site is external); physical versus digital environments; how agents should decide when to learn; and how designs change as models grow, including the thought that "a longer context window could reduce the importance of long-term memory".
The instructions used
The paper studies no system of its own and reports no prompts. It treats prompts as part of the agent's procedural memory: "prompt templates" are among the procedures the agent's code contains (Figure 4A), and inferring better prompt instructions (APE is the example) is classified as a learning action that updates reasoning (Section 4.5).
Limitations and open questions
- No empirical test. The framework's usefulness is argued and illustrated, not measured; no agent is built from it in the paper, and no comparison shows that CoALA-structured agents perform better.
- Classification is interpretive. Whether a store counts as episodic or semantic, or code execution as internal or external, depends on choices the paper makes case by case; Section 7 acknowledges that some boundaries are conceptual and that practitioners "may also just choose their preferred framing".
- Coverage as of its date. The case studies are five agents from 2022 and 2023. MemGPT, posted about five weeks after v1, is not discussed in v1 or v3.
- Analogy as motivation. The production-system analogy motivates the design but is not a result.
- Authors' own work. ReAct, Reflexion and Tree of Thoughts, three of the examples, are by authors of this paper; the classification of their own systems is not independently checked in the paper.
Research questions
- Do practitioners classify the same agent the same way? Asking several researchers to cast a set of agents into CoALA's categories and measuring agreement would test whether the taxonomy is precise enough to standardize terms.
- Does a modular, CoALA-structured implementation of a known agent match the original? Re-implementing an agent such as Generative Agents from separate memory, action and decision modules and comparing behavior would test the modularity recommendation.
- Does treating learning as a chosen action help? Comparing an agent that writes memory on a fixed schedule with one that decides when to write, on the same task, would test the claim in Section 7.
- Is writing to procedural memory riskier in practice? Measuring error rates introduced by writes to episodic, semantic and procedural stores in agents that support all three would test the risk ordering of Section 4.1.
- Does longer context reduce the value of long-term memory? Varying context length on a long-horizon task, with and without a retrieval store, would test the authors' conjecture.
How it connects
- Within this unit. The planned memory-for-language-model-agents can use CoALA's four memories and four kinds of action as one vocabulary for comparing designs, alongside the evaluation benchmarks in LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory and Evaluating Very Long-Term Conversational Memory of LLM Agents. The paper's remark that retrieval-augmented models read from a semantic memory ties the unit to Retrieval-Augmented Generation. These placements are the note's.
- Among the notes written with it. CoALA classifies Generative Agents: Interactive Simulacra of Human Behavior, Voyager: An Open-Ended Embodied Agent with Large Language Models and Reflexion: Language Agents with Verbal Reinforcement Learning directly (Sections 4.5 and 5). It does not discuss MemGPT: Towards LLMs as Operating Systems; read in CoALA's terms, MemGPT's prompt is working memory, recall storage is episodic memory, archival storage semantic memory, and its memory-editing functions are learning actions the model chooses, which is the "learning as an action" design CoALA recommends. That mapping is the note's.
- In the library. The components of agent memory sorts agent memory by a different axis, layers from transcripts to injected context, and describes an offline pass that rewrites notes, which in CoALA's terms is learning deferred out of the decision cycle; Building effective agents distinguishes workflows from agents by who directs the process, close to CoALA's question of where the decision procedure lives. These connections are the note's.
References
- Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L. (2024). Cognitive architectures for language agents. Transactions on Machine Learning Research. https://openreview.net/forum?id=1i6ZCvflQJ. arXiv:2309.02427 (v3 read with its LaTeX source; v1 compared). record