MemGPT: Towards LLMs as Operating Systems

Packer and colleagues' preprint treats the context window as main memory and gives the model function calls to page text in and out of two external stores, recall storage (every message) and archival storage (a vector database), prompted by memory-pressure warnings and chained by a heartbeat flag; in its February 2024 version MemGPT raised accuracy on a multi-session recall task from 32.1% to 92.5% with GPT-4 against a summary-only baseline, held document-QA accuracy flat as more documents were available, and completed nested key-value lookups that the base models failed.

Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez (UC Berkeley) · 2023-10-12 · original

How can a model with a fixed context window behave as if its context were unbounded? This paper borrows the answer operating systems give for memory: virtual memory, in which a program sees more memory than physically exists because the system pages data between fast main memory and slow disk. In MemGPT the prompt is main memory, two databases are the disk, and the paging is done by the language model itself, through function calls it chooses to make. The system also gives the model a writable block of text inside the prompt for facts about the user and itself, and warns it when the prompt is nearly full so that it can save what matters before old messages are evicted. On a recall task built from multi-session chat, MemGPT with GPT-4 answered 92.5% of questions correctly, against 32.1% for GPT-4 given a summary of the earlier sessions (Table 2, judged by a language model); on document question answering its accuracy did not fall as the number of available documents grew, while baselines that had to truncate documents did (Figure 5).

Version of record. The paper exists only as an arXiv preprint (2310.08560). Version 1 was posted on 12 October 2023, the date given above; version 2 on 12 February 2024. The v2 source compiles with the ICML 2024 style file in its "accepted" mode, but the paper is not among the 2,610 papers of the ICML 2024 proceedings (PMLR volume 235, checked by title), and no other peer-reviewed version was found. This note follows v2, read with its LaTeX source, and reports where v1 differs; the two versions report different numbers for the same task. The authors' affiliation is given as UC Berkeley. Code and data were released (the paper's website link, research.memgpt.ai).

Terms used in this note

  • Main context. The paper's name for the prompt tokens: everything "in-context" that the model reads on an inference step (Section 2).
  • External context. Anything held outside the context window, which "must always be explicitly moved into main context" before the model can use it (Section 2).
  • Recall storage. The message database: every message and model output, kept "indefinitely" (Section 2.2). The paper calls it recall storage and, in places, recall memory.
  • Archival storage. "A read/write database storing arbitrary length text objects" (Section 2.2), searched by embedding similarity. In places called archival memory.
  • Working context. A "fixed-size read/write block of unstructured text, writeable only via MemGPT function calls" inside the prompt (Section 2.1). The appendix's persona prompt calls the same block "core memory".
  • Virtual context management. The paper's name for the whole technique.

How the memory works

The three tiers

The prompt is split into three contiguous parts (Section 2.1, Figure 3):

  1. System instructions, read-only: how MemGPT's control flow works, what each memory level is for, and the schemas of the memory functions, with natural-language descriptions (Section 2.3).
  2. Working context, read-write by function call only. In conversation it holds "key facts, preferences, and other important information about the user and the persona the agent is adopting".
  3. A FIFO queue of recent messages: user and agent messages, system messages such as memory warnings, and function calls with their outputs. Its first entry is a system message holding "a recursive summary of messages that have been evicted from the queue".

Outside the prompt are recall storage (the full message history) and archival storage (arbitrary text, including uploaded documents). Archival search in the experiments ran on PostgreSQL with the pgvector extension and an HNSW index over OpenAI text-embedding-ada-002 embeddings, by cosine similarity (Section 3.2.1). The v1 text describes three ways to query external context, by timestamp, by text and by embedding (v1, Section 2.2); v2 does not repeat the list.

When things are written

  • Every message is logged automatically. The queue manager appends each incoming message to the queue and writes both it and the model's output to recall storage (Section 2.2). Nothing the model sees is lost; it may only leave the prompt.
  • Eviction is automatic, saving is the model's choice. When the prompt passes a "warning token count" (the example given is 70% of the window), the queue manager inserts a "memory pressure" warning so the model can use functions to copy important information from the queue into working context or archival storage. At the "flush token count" (example: 100%), the manager evicts a block of messages (example: 50% of the window) and folds them into a new recursive summary (Section 2.2). The percentages are given as examples; the settings used in each experiment are not stated.
  • Self-directed edits. The model may write to working context or archival storage at any time, not only under pressure. In v2 the example figures show the model appending "Birthday is February 7" and "Boyfriend named James" to working context after a memory-pressure alert (Figure 1), and later replacing "Boyfriend named James" with "Ex-boyfriend named James" when the user says they broke up (Figure 4). v1's figures show a write with no memory warning (a birthday and a favorite cake, v1 Figure 2) and a correction from "I watch horror movies." to "I like romantic comedies." (v1 Figure 4).

How it is retrieved

Retrieval is a function call the model decides to make, with a query it writes: recall_storage.search("six flags") for an earlier conversation, archival_storage.search("nobel physics", page=2) for documents (figures in Sections 1 and 3). Results are paginated so that a search cannot overflow the window; the document-QA figure shows "Showing 10 of 124 results (page 1/13)". Retrieved messages are appended to the back of the queue, so they re-enter the prompt as ordinary queue entries (Section 2.2). Ranking inside archival search is the vector index's cosine similarity; there is no recency or importance term. The model's own judgment decides when to search, what to search for, and when to stop paging.

Function chaining and interrupts

Inference is driven by events: user messages, system messages such as memory warnings, user actions such as a login or a finished upload, and "timed events that are run on a regular schedule", which let MemGPT run "unprompted" (Section 2.4). After each function call the model can set a flag, request_heartbeat=true, asking to run again immediately with the function's output in context; without it the processor yields and waits for the next external event "(e.g. a user message or scheduled interrupt)" (Section 2.4, Figure 3). Chaining is what allows multi-step retrieval, such as paging through results or following a chain of keys. The function executor parses each output, runs valid calls and feeds back results and errors, including errors such as trying to add to working context when it is full (Section 2.3). The v1 abstract described the design as using "interrupts to manage control flow between itself and the user"; v2's abstract drops the word, and its conclusion keeps "hierarchical memory management and interrupts".

What enters the context

On every step, the model reads: the fixed system instructions and function schemas, the current working context, the recursive summary of evicted messages, and the queue of recent messages, including any search results the model has pulled in. Everything else is outside until the model asks for it. The v1 text notes the cost of this design: the system instructions and function schemas take part of the window, so fewer documents fit in the prompt at once than for a baseline using the same model (v1, Section 3.2).

What the paper measures

Three OpenAI models were used, as the processor of MemGPT and as baselines without it: gpt-4-1106-preview ("GPT-4 Turbo", 128,000 tokens), gpt-4-0613 ("GPT-4", 8,192) and gpt-3.5-turbo-1106 (16,385) (Section 3). The paper reports no repeated runs, variances or confidence intervals for any result.

Deep memory retrieval (consistency)

Task (Section 3.1). The Multi-Session Chat dataset (Xu et al. 2021) has chats of five sessions, each "roughly a dozen messages", between crowdworkers playing consistent personas. The authors added a sixth session with one question that refers back to an earlier session, generated by a separate language model told to write a question answerable only from the old chat log, not the persona (prompt in the appendix). The number of questions is not stated.

What was varied. The baseline model saw "a lossy summarization of the past five conversations"; MemGPT had the full history in recall storage, reachable by paginated search.

Grading. An LLM judge labeled each answer CORRECT or WRONG against the gold answer. Neither version names the judge model; the text cites work showing that GPT-4 agrees highly with human evaluators, which implies but does not state that GPT-4 was the judge, and the paper reports no validation of its own judge. The judge prompt tells it to "be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT", and counts "I'm sorry, I don't remember" as wrong. ROUGE-L recall is also reported, because answers were longer than the gold labels.

Results (Table 2, v2):

Model Accuracy ROUGE-L (R)
GPT-3.5 Turbo 38.7% 0.394
+ MemGPT 66.9% 0.629
GPT-4 32.1% 0.296
+ MemGPT 92.5% 0.814
GPT-4 Turbo 35.3% 0.359
+ MemGPT 93.4% 0.827

Version 1 reported different numbers for the same task. v1's Table 2 names its models only as "gpt-3.5" and "gpt-4" and uses different baseline variants: non-oracle baselines from 55.6% to 79.2% (GPT-3.5 and GPT-4, given a summary of sessions 1 to 5 or a summary of sessions 1 to 4 plus session 5 verbatim; GPT-4 with the summary of sessions 1 to 5 scored 63.0%), MemGPT 82.4% (ROUGE-L recall 0.743), and oracle baselines given the gold personas at 70.0% and 79.8%, with MemGPT given persona and recall at 84.0%. In v2 the comparable figures are 32.1% for GPT-4 with a summary and 92.5% for MemGPT with GPT-4. v2 does not mention or explain the change; the baseline prompt printed in v2's appendix (below) is one candidate difference, but the paper does not say whether v1 used it.

Conversation opener (engagement)

Task (Section 3.1). Write the first message of the next session so that it draws on what is known about the user. Openers were scored by similarity to the gold persona statements (SIM-1, the closest persona fragment; SIM-3, the mean of the top three) and to the human-written opener (SIM-H). The similarity measure (embedding model) is not named.

Results (Table 3, v2): human opener 0.800 / 0.800 / 1.000; MemGPT with GPT-3.5 Turbo 0.830 / 0.812 / 0.817; with GPT-4 0.868 / 0.843 / 0.773; with GPT-4 Turbo 0.857 / 0.828 / 0.767 (SIM-1 / SIM-3 / SIM-H). There is no baseline without MemGPT in v2. The text says "storing information in working context is key to generating engaging openers", a sentence carried over from v1, where it was supported by an ablation (v1 Table 3: with working context and recall storage, CSIM-1 0.817; working context only 0.820; recall storage only 0.749). v2 drops the ablation table but keeps the sentence. A higher similarity to persona statements than the human opener's means the opener mentions more persona facts; whether that makes it more engaging was not judged by people.

Multi-document question answering

Task (Section 3.2.1), from Liu et al. (2023): questions from NaturalQuestions-Open, Wikipedia from late 2018 as the corpus, and a retriever that ranks passages by text-embedding-ada-002 similarity. Baselines receive the top $K$ passages in the prompt, truncated to fit when $K$ is large; MemGPT has the whole corpus in archival storage and pages through search results itself. A subset of 50 questions was sampled. An LLM judge required both the answer and the supporting document text.

Results (Figure 5, which prints no values; all values read from the figure). MemGPT's accuracy is drawn as a horizontal line, about 0.72 with GPT-4 and GPT-4 Turbo and about 0.40 with GPT-3.5, since it does not depend on $K$. GPT-4 as a baseline peaks near 0.70 at about 10 documents and falls to about 0.12 at 200 as truncation removes text; GPT-4 Turbo stays between about 0.56 and 0.68 from 10 to 200 documents. The authors observe that MemGPT "will often stop paging through retriever results before exhausting the retriever database" (Section 3.2.1). v1 added that MemGPT sometimes paused to ask the user to narrow the query and that "these questions are counted as failed answers", and that MemGPT had "a lower average accuracy than GPT-4" because its instructions occupy part of the window (v1, Section 3.2); v2 does not repeat either point.

Nested key-value retrieval

Task (Section 3.2.2), a new variant of Liu et al.'s synthetic task: 140 key-value pairs of 128-bit UUIDs (about 8,000 tokens), where a value may itself be a key, so the answer needs up to several lookups in sequence. Thirty orderings were sampled per nesting level. The text says nesting levels run from 0 to 4; Figure 7 plots levels 0 to 3.

Results (Figure 7, values read from the figure). MemGPT with GPT-4 was correct at every plotted level (1.0). MemGPT with GPT-4 Turbo fell to about 0.7 at level 2 and 0.6 at level 3. The baselines collapsed: GPT-3.5 reached 0 at level 1; GPT-4 was about 0.3 at level 1 and 0 at level 3; GPT-4 Turbo about 0.5 at level 1 and 0 at level 3. MemGPT with GPT-3.5 fell like the baselines (about 0.2 at level 1), although the text groups it with GPT-4 Turbo as beginning "to drop off in performance at 2 nesting levels".

Where the key-value pairs were held is described two ways. The text says MemGPT with GPT-4 performs the lookup "by accessing the key-value pairs stored in main context repeatedly via function queries" (Section 3.2.2; v1 says "main memory"). The example figure shows it issuing one archival_storage.search per key until a search returns only one result, "indicating that it is not also a key" (Figure 8), and the task persona in the appendix tells it the answer "will ALWAYS be in your archival memory" (Appendix 6.1.6). The paper does not reconcile the two; the figure and the persona point to archival storage, which is the reading under which repeated function calls would be needed.

The instructions used

Each task used a written persona or preprompt, printed (shortened, the authors say) in the appendix; the full prompts are in the released code.

  • Chat persona (MemGPT, DMR): it tells the agent to stay in role, never to say it is an AI, and "If the user asks me a question, I should reply with a best guess using the information in core memory and conversation_search."
  • Chat baseline preprompt (DMR): the summary of prior conversations, followed by "If you do not have enough information to answer the question, reply 'NO ANSWER'."

The two conditions therefore differ in instruction as well as in memory: MemGPT is told to guess, the baseline is told to abstain, and the judge scores an abstention as wrong. The paper does not discuss this difference or test either condition with the other's instruction; the observation is the note's.

  • Document QA: MemGPT is told "The answer to the users question will ALWAYS be in your archival memory, so remember to keep searching if you can't find the answer." Baselines are told to reply "INSUFFICIENT INFORMATION" if no document contains the answer, and the judge scores that reply as incorrect.
  • Nested KV: the MemGPT persona adds "DO NOT STOP SEARCHING UNTIL YOU VERIFY THAT THE VALUE IS NOT A KEY." The baseline prompt explains the nested lookup with an example.

How the model manages memory is itself governed by instructions: the system prompt's description of the memory hierarchy and the function schemas are, in the authors' words, how "self-directed editing and retrieval" is implemented (Section 2.3). Their wording is not printed in the paper and was not varied.

Limitations and open questions

  • No peer review, and numbers changed between versions without explanation.
  • Single runs, no variances, an unstated number of DMR questions, and 50 document-QA questions.
  • Model-graded results. DMR and document QA are graded by a language model that the paper does not name, and openers by a similarity score whose model is not named; no validation is reported, and the DMR judge is instructed to be generous.
  • Instruction asymmetries between MemGPT and baselines in DMR and document QA (above).
  • Dependence on strong function calling. v1's limitations section reported that GPT-4 models "rarely made syntactic or semantic errors" on the MemGPT functions, while GPT-3.5 and Llama 2 70B variants "consistently generated incorrect function calls" or invented functions; v2 drops the section but its results show the same pattern (MemGPT with GPT-3.5 at about 0.40 on document QA).
  • Stopping early. The model ends searches before exhausting relevant results (Section 3.2.1), so retrieval depth is a model decision with no guarantee.
  • What is not tested. Writes to working context and archival storage are illustrated by examples, not evaluated: there is no measurement of what the model chose to save under memory pressure, whether saved facts were correct, or how the recursive summary degrades over time.

Research questions

  1. How much of the DMR gap is instruction rather than memory? Running the baseline with MemGPT's "best guess" instruction, and MemGPT with the baseline's abstention instruction, would separate the two.
  2. What does the model save under memory pressure? Logging every write to working context and archival storage over long chats, and checking each against the conversation, would give the precision and recall of self-directed saving.
  3. When does the model stop searching? Recording the rank of the gold passage against the last page MemGPT read, over many questions, would measure how early stopping limits document QA.
  4. Do the reported gains hold with repeated runs and human grading? Repeating DMR with several seeds and a human-validated judge would give intervals for Table 2.
  5. Does a fixed retrieval rule do as well as model-chosen paging? Comparing MemGPT's self-directed search with a fixed rule that always inserts the top recall results before answering would show what the model's choices add.

How it connects

References

  • Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., and Gonzalez, J. E. (2023). MemGPT: Towards LLMs as operating systems. arXiv:2310.08560 (v1, 12 October 2023; v2, 12 February 2024; v2 read with its LaTeX source, v1 read for comparison). record · 2 pages
  • Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv:2307.03172. (Source of the document-QA and key-value tasks; not read for this note.) record
  • Xu, J., Szlam, A., and Weston, J. (2021). Beyond goldfish memory: Long-term open-domain conversation. arXiv:2107.07567. (The Multi-Session Chat dataset; not read for this note.) record