LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Wu and colleagues' ICLR 2025 benchmark hides the evidence for 500 hand-written questions inside long user-assistant chat histories (about 115k tokens, or 500 sessions) and tests five memory abilities; long-context models lost 30% to 60% of their accuracy against reading only the evidence, and a study of memory as a key-value store found that storing rounds rather than sessions, adding extracted user facts to the index keys, filtering by an LLM-inferred time range, and reading with Chain-of-Note in JSON each helped, with a GPT-4o judge validated at 97% to 98% agreement with experts.

Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu (UCLA; Tencent AI Lab Seattle; UC San Diego) · 2024-10-14 · original

A chat assistant that remembers its user has to notice a fact mentioned in passing, keep it for weeks, combine it with others, update it when it changes, place it in time, and admit when it was never told. This paper turns those five requirements into a benchmark of 500 questions whose evidence is buried in long, mostly unrelated chat histories, and then uses the benchmark to study how a memory system should be built. Its two headline results are a gap and a recipe. The gap: long-context models answered 30% to 60% fewer questions when they read the whole history than when they read only the evidence sessions (Figure 3b), and two commercial assistants with memory did worse still on much shorter histories (Figure 3a). The recipe: treat memory as a key-value store and get four choices right, which together raised retrieval recall and answer accuracy (Sections 5.2 to 5.5).

Version of record. The paper was published at ICLR 2025; the arXiv version 2 (2410.10813v2, 4 March 2025) carries the header "Published as a conference paper at ICLR 2025" and is the version this note follows. Version 1 (14 October 2024, the date above) reports the same pilot-study numbers and retriever, but its summary figures for the design study differ: key expansion with user facts gave "4% higher recall@k" and "5% higher accuracy" in v1 against 9.4% and 5.4% in v2, and time-aware query expansion 11.4% and 6.7% against 11.3% and 6.8%; v1 also numbers its tables differently. This note did not compare the two versions line by line, and it did not retrieve the OpenReview copy. The benchmark and code are released at github.com/xiaowu0162/LongMemEval (named in the abstract); this note did not inspect the release.

Terms used in this note

  • Session, round. A session is one dated user-assistant conversation; a round is one user message and the assistant's reply (Section 3.1).
  • Evidence session. A session that contains a fact needed to answer a question. A history is the evidence sessions shuffled among unrelated ones.
  • LongMemEvalS and LongMemEvalM. The two standard history lengths: about 115k tokens per question (about 50 sessions), and 500 sessions (about 1.5 million tokens) (Section 3.2).
  • Oracle setting. The model reads only the evidence sessions.
  • Key, value. In the paper's framework, memory is a list of (key, value) items: the value is what is stored and later read (a session, a round, a summary, a fact); the key is what retrieval matches against (Section 4.1).
  • Chain-of-Note (CoN). A reading method (Yu et al., 2023) in which the model first writes notes extracting what each retrieved item says, then answers from the notes.

The benchmark

Five abilities, seven question types (Section 3.2, Figure 1). Information extraction (questions about something the user said, something the assistant said, or a stated preference that should shape a recommendation); multi-session reasoning (aggregate or compare across sessions, for example counting instruments owned); knowledge updates (the latest value of something that changed); temporal reasoning (using timestamps and time expressions); and abstention (30 questions modified from the others into "false premise" questions, which should be answered with "I don't know"). Read from the pie chart of Figure 9a, the shares of the six non-abstention types are 14% single-session user, 11% single-session assistant, 6% single-session preference, 27% multi-session, 16% knowledge update and 27% temporal reasoning.

How the questions were made (Section 3.2, Figure 2, Appendix A). The authors wrote an ontology of 164 user attributes in five categories (lifestyle, belongings, life events, situations, demographics). An LLM (Llama 3 70B Instruct unless stated) wrote background paragraphs for an attribute and proposed seed question-answer pairs; "human experts manually filter and rewrite all the questions", and decompose each answer into evidence statements, some with timestamps. Each evidence statement was then embedded in a task-oriented session simulated between a user LLM and an assistant LLM, with the user told to convey the fact indirectly (asking about car insurance rather than announcing a new car). Annotators screened and edited every evidence session for evidence inclusion, position, colloquial phrasing and timestamps; "roughly 70% of the sessions are human edited" (Appendix A.1). Most questions need evidence from more than one session, up to six (Figure 9b). Within an evidence session, the evidence most often sits in the first round: Figure 9c's caption says "Most evidence statements are located at the beginning of the chat".

How a history is built (Section 3.2, Appendix A.2). Following the needle-in-a-haystack idea, the evidence sessions are inserted among filler sessions drawn 25% from ShareGPT, 25% from UltraChat and 50% from simulated sessions on other, non-conflicting attributes, and timestamps are assigned in order. Because filler is sampled, the history length is configurable.

How answers are scored (Section 3.3, Appendix A.4). A prompt-engineered judge, gpt-4o-2024-08-06, is given the question, the correct answer (or, for preference questions, a rubric) and the response, and answers yes or no. There are separate instructions per type: for temporal questions it is told to "not penalize off-by-one errors for the number of days"; for knowledge updates a response that includes old information is correct "as long as the updated answer is the required answer"; for preferences the response "does not need to reflect all the points in the rubric". To validate it, the authors sampled 30 questions per type, took long-context answers from GPT-4o and Llama 3.1 8B Instruct, and compared the judge with expert judgments: agreement averaged 0.98 for GPT-4o's answers and 0.97 for Llama's, with the lowest cells 0.90 (27 of 30) for preference questions answered by GPT-4o and abstention questions answered by Llama (Table 6). The number of experts in this check is not given; the ethics statement says dataset construction and the commercial study were "mainly conducted by three expert annotators", in-house NLP researchers. When a system exposes what it retrieved, Recall@k and NDCG@k are computed against the annotated evidence locations.

What the pilot study found

  • Commercial assistants (Section 3.4, Appendix B, Figure 3a). Because ChatGPT and Coze offer memory only through their web interfaces, five annotators played 97 questions by hand over short histories of 3 to 6 sessions (about one tenth the length of LongMemEvalS), in the first two weeks of August 2024, clearing memory between questions. Accuracy: ChatGPT 0.5773 with GPT-4o and 0.7113 with GPT-4o-mini; Coze 0.3299 with GPT-4o and 0.2474 with GPT-3.5-turbo; GPT-4o reading the same short histories offline, 0.9184. This early version had no abstention questions and skipped temporal questions that need months of real time and questions about what the assistant said. The authors observed that ChatGPT recorded facts promptly but "often modify this information when it compresses the history", while Coze often failed to record facts given indirectly.
  • Long-context models (Figure 3b, LongMemEvalS against the oracle setting, no Chain-of-Note). GPT-4o 0.870 to 0.606 (a 30.3% drop); Llama 3.1 70B Instruct 0.744 to 0.334 (55.1%); Llama 3.1 8B Instruct 0.710 to 0.454 (36.1%); Phi-3 128k (14B) 0.702 to 0.380 (45.9%); Phi-3.5 Mini 0.660 to 0.342 (48.1%). With Chain-of-Note the oracle scores rose for GPT-4o (0.924), Llama 3.1 70B (0.848) and Phi-3 (0.722) and were flat or lower for the two smallest models, and no drop shrank (GPT-4o 30.7%, Llama 3.1 70B 66.3%).

No variance or repeated runs are reported for any result in the paper.

What the memory design study does

The framework (Section 4, Figure 4). A memory system has three stages. Indexing turns each session into one or more key-value items; retrieval forms a query and fetches the top k items; reading gives the items to an LLM to answer. Four control points: the value (granularity and form of what is stored), the key (what is matched), the query, and the reading strategy. Table 2 places nine systems in this frame, among them MemoryBank, RAPTOR, MemWalker, HippoRAG, ChatGPT and Coze (the last two with most choices unknown).

The setup (Section 5.1). Readers: GPT-4o, Llama 3.1 70B Instruct and Llama 3.1 8B Instruct. Retriever: dense retrieval with the 1.5B Stella V5 embedding model. Summaries, keyphrases, user facts and timestamped events for keys and values are extracted by Llama 3.1 8B Instruct. When sessions or rounds serve as keys, only the user's side is kept. Retrieved items are always sorted by timestamp before reading, and Chain-of-Note with JSON formatting is the default reading method. Experiments are on LongMemEvalM (500 sessions), where reading the whole history is not possible.

So, in the paper's recommended design (Table 2, "Our Design"): what is stored is the history split into rounds; each round is indexed under a key made of the round's text concatenated with user facts extracted from it, and also by the dates of the events it mentions; retrieval matches the question against these keys, restricted to a time range that an LLM infers from the question when the question is time-sensitive; and what enters the reader's context is the top-k rounds, sorted by timestamp and formatted as JSON, with an instruction to take notes on each before answering.

What it finds

  • Value: rounds beat sessions; facts lose information (Section 5.2, Figure 5). With GPT-4o as reader, splitting sessions into rounds "significantly enhances" accuracy; with Llama 3.1 8B the two were similar. Storing summaries or extracted facts in place of the text lowered accuracy, except on multi-session questions, where facts helped. The best retrieval budget depended on the reader: Llama 3.1 8B "drops sharply beyond 3k retrieved tokens" while GPT-4o kept improving past 20k (values read from Figure 5, which plots curves).
  • Key: add facts to the text, do not replace it (Section 5.3, Table 3). The paper reports that a summary, keyphrases or facts alone as the key did not improve recall over the value itself; Table 3 has one exception, facts alone with sessions as values, at Recall@10 (0.814 against 0.783). Concatenating extracted user facts to the value (document expansion) gave the best recall for both rounds and sessions, though not the best accuracy in every reader and top-k cell. For rounds, Recall@5 went from 0.582 to 0.644 and GPT-4o's top-5 accuracy from 0.615 to 0.657; for sessions, Recall@5 from 0.706 to 0.732 and GPT-4o top-5 accuracy from 0.670 to 0.714. The paper summarizes this as an average improvement of 9.4% in recall@k and 5.4% in accuracy across models; it does not say whether these are relative or absolute. The authors also tried merging retrieval pathways at query time by combining ranks, which did worse than merging at indexing (Section 5.3, footnote 4; details in Appendix E.3, not checked for this note).
  • Query: time-aware expansion (Section 5.4, Table 4, temporal-reasoning subset). Indexing values by event dates and filtering by a time range that GPT-4o extracts from the question raised recall "by an average of 11.3%" with rounds and 6.8% with sessions; for example, with fact-expanded keys and rounds, Recall@10 went from 0.550 to 0.722. When Llama 3.1 8B extracted the time range instead, recall was often no better or worse (with fact-expanded keys and rounds, Recall@5 0.481 against 0.489 without expansion), which the authors attribute to hallucinated or missed time cues.
  • Reading: Chain-of-Note and JSON (Section 5.5, Figure 6, oracle retrieval). For GPT-4o, accuracy was 0.862 with plain text and direct answering, 0.822 with JSON and direct answering, 0.910 with plain text and Chain-of-Note, and 0.924 with JSON and Chain-of-Note; for Llama 3.1 70B, 0.762, 0.744, 0.800 and 0.848. The authors' summary: "even with perfect retrieval, a suboptimal reading strategy results in up to a 10-point absolute performance drop", and JSON helps consistently only together with Chain-of-Note.

The prompts used

The benchmark's text is generated from prompts and then edited (Appendix A, Figure 8 for the user-simulator prompt); the judge's instructions are printed in Figure 10, and the paper states the judge prompt will be released for consistent comparison. The judge's leniencies (off-by-one days accepted; old information alongside the updated answer accepted; preference rubrics need not be fully met) define what "correct" means on this benchmark, and any score reported with a different judge or prompt is not directly comparable. The memory study's extraction prompts (summaries, keyphrases, facts, events, time ranges) are described in Section 4.2 and Appendix D, which this note did not check in detail.

Limitations and open questions

  • Synthetic histories. Evidence sessions are LLM-simulated and human-edited; filler comes from public chat logs and simulations. Real users' histories are not used.
  • Single runs, no intervals. None of the tables reports variance; differences of a few points between designs have no stated uncertainty.
  • The commercial study is small and different. 97 questions, short histories, hand-played by annotators on an early version of the dataset, and products that change over time. The authors present it as a pilot.
  • Judge leniency. Validated against experts at 90% or better per type, on long-context answers from two models only. Memory systems that answer differently (more hedged, more verbose) were not part of the validation.
  • Design results are on one retriever and one extractor. Stella V5 and Llama 3.1 8B; Appendix E.2 reports that with other retrievers, summary and keyphrase expansion could help sessions too.
  • Published use and critique. Zep's founders, in a post disputing Mem0's results on LoCoMo (Chalef and Rasmussen, 2025), recommend LongMemEval instead, as longer, testing knowledge updates, and "Human-curated and designed to be high quality". The post is a vendor's, and this note found no published critique of LongMemEval itself in the time spent.

Research questions

  1. How stable are the design rankings across retrievers and extractors? Repeating Table 3 and Table 4 with several retrievers and a stronger extraction model, with several runs each, would show which recommendations are robust.
  2. How well does the judge agree with experts on memory-system outputs? Repeating the Table 6 meta-evaluation on answers from retrieval-based systems rather than long-context readers would test the judge where it is now used.
  3. Does document expansion keep helping as facts accumulate? Measuring recall with fact-expanded keys as the history grows from 50 to 500 sessions and beyond would show whether extracted facts crowd the index.
  4. Do commercial memory products still overwrite facts? Re-running the 97-question protocol on current products, with dates recorded, would show whether the 2024 failure modes persist.

How it connects

  • LoCoMo is the benchmark this paper sets itself against: the paper says LoCoMo uses human-human conversations and does not test recall of what the assistant said or reasoning with updated information (Section 1), and its Table 1 describes LoCoMo with the figures of LoCoMo's arXiv version (50 histories, 7,512 questions), not those of its ACL version (10, 1,986). The paper makes the comparison; the version detail is this note's.
  • Mem0 stores extracted facts as its values, the form this paper found loses information except on multi-session questions; Mem0 reports results on LoCoMo, not LongMemEval. This note connects them.
  • Retrieval-Augmented Generation, Assembling the Context and Query Expansion cover the general techniques this paper adapts: what unit to retrieve, how to order and format retrieved text, and how to rewrite a query. Fact-augmented keys are document expansion, which the paper cites to Tao et al. (2006) and Efron et al. (2012); Query Expansion and Learned Sparse Retrieval describe document expansion in ad hoc retrieval. This note connects them.
  • Evaluating Retrieval-Augmented Systems discusses model judges and their agreement with people; this paper's Table 6 is a per-type check of a judge against experts. This note connects them.
  • The components of agent memory describes transcripts, notes and wikis kept by Claude; this paper's value choices (sessions, rounds, summaries, facts) are the same layers measured on a benchmark. This note connects them. GraphRAG and LAMA are the unit's notes on document stores and on weights.
  • The planned synthesis memory-for-language-model-agents will draw on this note for its design vocabulary (value, key, query, reading).

References

  • Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., and Yu, D. (2025). LongMemEval: Benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR 2025). arXiv:2410.10813 (v2, 4 March 2025; v1, 14 October 2024). record
  • Chalef, D., and Rasmussen, P. (2025, May 6; updated 2026, June 3). Lies, damn lies, & statistics: Is Mem0 really SOTA in agent memory? Zep blog. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/ record · 3 pages
  • Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., and Yu, D. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv:2311.09210. (The reading method; entry as the paper cites it; not read for this note.) record
  • Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., and Fang, Y. (2024). Evaluating very long-term conversational memory of LLM agents. In Proceedings of ACL 2024 (Volume 1: Long Papers), pages 13851–13870. record · 3 pages