How well does a language model remember a conversation that has run for months? Earlier dialogue benchmarks stopped at about five sessions and a thousand or so tokens (Section 1). This paper builds longer ones: conversations between two LLM agents, each with a persona and a timeline of life events, carried over up to 32 sessions and about 16,000 tokens, then edited by people for consistency. On questions about these conversations, every model tested fell far below the paper's human score (87.9 F1): the best, gpt-4-turbo reading the whole conversation, scored 51.6 (Table 2). Longer context helped on most questions and hurt on questions designed to have no answer, and a retrieval system did best when it retrieved short statements extracted about each speaker rather than raw turns (Table 3). The benchmark, LoCoMo, was later adopted as the test bed for memory systems, including Mem0.
Version of record. The paper appeared in the Proceedings of the 62nd Annual Meeting of the ACL (Volume 1: Long Papers), Bangkok, August 2024, pages 13851 to 13870, DOI 10.18653/v1/2024.acl-long.747. Its only arXiv version (2402.17753, 27 February 2024, the date above) describes a larger and shorter dataset: 50 conversations of about 300 turns and 9K tokens over up to 35 sessions, with 7,512 questions (arXiv Section 1 and Table 5), against 10 conversations of about 600 turns and 16K tokens over up to 32 sessions, with 1,986 questions, in the proceedings (Section 1 and Table 5). The model list and results also differ (the arXiv version mentions no Llama-3, Gemini or Claude model, and states improvements of "22-66%" and a gap to humans of 56%, against 12-20% and 36% in the proceedings). This note follows the proceedings throughout. Later papers cite both versions: LongMemEval describes LoCoMo with the arXiv figures (50 histories, 7,512 questions), Mem0 with the 10-conversation release. This note did not inspect the released dataset (project page snap-research.github.io/locomo).
Terms used in this note
- Session. One dated conversation between the two speakers; a conversation in LoCoMo is a sequence of sessions days to months apart.
- Temporal event graph. For each speaker, a list of dated life events with causal links between them ("caused_by"), generated before the conversation and used to steer it (Section 3.2).
- Observation. A short factual statement about a speaker extracted by an LLM from a session, with the turn IDs it came from, for example "Melanie is busy with kids and work" (Appendix A.2.1, Figure 9). Observations are both part of the generating agents' memory and one of the retrieval units tested.
- Session summary. An LLM summary of a session conditioned on the previous session's summary (Figure 8).
- F1 (partial match). Token-overlap F1 between predicted and gold answers after normalization (Section 4.1).
How the conversations were generated
The pipeline (Section 3, Figure 3; prompts in Appendix A):
- Personas. A 4 to 5 sentence persona from the MSC dataset is expanded by gpt-3.5-turbo into a full description with name, age, job, family and habits, using one in-context example (Section 3.1, Figure 5).
- Event graphs. text-davinci-003, conditioned on the persona, generates up to 25 events over 6 to 12 months, three at a time, each new batch caused by existing events (Section 3.2, Figure 6).
- Two agents talk. Each speaker is an agent on gpt-3.5-turbo built from the generative-agents architecture of Park et al. (2023). Its memory has two parts (Section 3.3): after each session it writes a summary, carried forward; and each turn is turned into an observation and stored in a long-term memory. A response in the next session is conditioned on the latest summary, on reflections drawn from retrieved observations, on the current session so far, on the persona, and on the events from the speaker's graph that fall between the last session's date and this one's. That last input is what makes speakers report new life events across sessions.
- Images. An agent may decide to share an image: it writes a caption, turns it into search keywords and fetches a web image; the receiving agent captions it with BLIP-2 and reacts (Section 3.3, Figure 10).
- Human editing. Annotators fixed long-range inconsistencies, removed or replaced irrelevant images, and checked that conversations matched the event graphs, removing events that never came up. They edited "nearly 15%" of turns and removed or replaced about 19% of images (Section 3.4). The annotators were in-house, and their number and demographics are not given (Appendix B.3).
The resulting dataset (Table 5): 10 conversations, 27.2 sessions and 21.6 turns per session on average, 16,618.1 tokens per conversation, 35.8 ground-truth events per conversation and 91.2 images. The dataset is released under CC BY-NC 4.0 (Appendix B.2).
The tasks
Question answering (Section 4.1; counts from Table 5):
| Category | What it requires | Questions |
|---|---|---|
| Single-hop | an answer from one session | 841 (42.3%) |
| Multi-hop | information from several sessions | 282 (14.2%) |
| Temporal reasoning | time cues in the conversation | 321 (16.1%) |
| Open-domain knowledge | a speaker's information plus commonsense or world knowledge | 96 (4.8%) |
| Adversarial | questions designed to trick the model, to be recognized as unanswerable | 446 (22.4%) |
Gold answers were taken from the conversation's wording "as much as possible", and models were instructed to copy the conversation's wording when they could, so that partial-match F1 is meaningful; each question is annotated with the turn IDs that hold its answer, which gives a recall@k for retrieval (Section 4.1). The paper does not say who wrote the questions or how the human score was obtained (how many people, with what access to the conversation).
Event summarization (Section 4.2). A model summarizes a speaker's events over a period; the summary is scored against the event graph with FactScore (Min et al., 2023) adapted to give precision (summary facts found in the graph) and recall (graph facts found in the summary), and their F1.
Multimodal dialogue generation (Section 4.3). MiniGPT-5 variants trained on 50 further generated conversations (not human-edited) predict the next turn and image, scored with MM-Relevance and other generation metrics.
What enters the model's context
Three setups were tested on question answering (Section 5, Appendix C.1):
- Base: models with short windows (Mistral-7B-Instruct-v0.2 with 8K tokens; Llama-2-70b-chat and Llama-3-70B-Instruct with 4K). Earlier dialogue is cut; the most recent turns that fit are given, followed by the question.
- Long context: as much of the conversation as the window holds: gpt-3.5-turbo at 4K, 8K, 12K and 16K, gemini-1.0-pro (1M), claude-3-sonnet (200K) and gpt-4-turbo (128K).
- RAG: DRAGON (Lin et al., 2023) retrieves the top k items and gpt-3.5-turbo reads them. Three kinds of item were compared: dialogue turns, observations, and session summaries.
The paper names the Gemini model inconsistently: Section 5, Table 2 and Section 6.2 say gemini-1.0-pro, while Section 6.1 says "gemini-pro-1.5". Which model was run cannot be settled from the paper; this note uses the table's label.
Images are replaced by their BLIP-2 captions for question answering and summarization. All runs used temperature 0 and are "a single inference run for each model" (Appendix C.2); no variance is reported. API models were used as of May 2024.
What it finds
Question answering, base and long-context models (Table 2, F1):
| Model (context) | Single-hop | Multi-hop | Temporal | Open-domain | Adversarial | Overall |
|---|---|---|---|---|---|---|
| Human | 95.1 | 85.8 | 92.6 | 75.4 | 89.4 | 87.9 |
| Llama-3-70B-Instruct (4K) | 17.0 | 17.0 | 12.0 | 13.0 | 80.0 | 30.1 |
| gpt-3.5-turbo (4K) | 23.8 | 18.0 | 15.6 | 20.4 | 34.8 | 23.9 |
| gpt-3.5-turbo (16K) | 52.6 | 36.7 | 24.3 | 24.0 | 14.8 | 35.9 |
| gemini-1.0-pro (1M) | 62.4 | 35.3 | 34.2 | 19.0 | 5.2 | 39.1 |
| claude-3-sonnet (200K) | 70.7 | 38.1 | 26.9 | 52.2 | 2.5 | 42.8 |
| gpt-4-turbo (128K) | 72.3 | 51.5 | 51.4 | 38.5 | 15.7 | 51.6 |
The other two base models scored 18.7 (Mistral) and 18.4 (Llama-2) overall. The pattern the authors draw out (Section 6.1): more context raised single-hop and multi-hop scores and lowered adversarial scores, which they read as models being "easily misled into generating hallucinations" in long contexts (Section 1 adds that long-context models are "especially prone to misassigning dialogs or events to the wrong speaker"); and the gap between single-hop and multi-hop scores suggests models can find a fact in a long context but struggle to reason over several. Temporal and open-domain questions were the hardest.
Question answering with retrieval (Table 3, gpt-3.5-turbo reader, overall F1, with recall@k of the evidence in parentheses):
| Retrieved unit | k = 5 | k = 10 | k = 25 |
|---|---|---|---|
| Dialogue turns | 38.8 (56.7) | 39.7 (66.2) | 41.0 (76.7) |
| Observations | 43.3 (56.2) | 42.8 (61.3) | 42.1 (67.5) |
Dialogue at k = 50 scored 40.5 (recall 82.7). Table 3 has a fourth observation row, labelled k = 5 a second time but by its position and its higher recall apparently k = 50: 40.6 (71.2). Session summaries scored 29.0, 30.9 and 32.0 at k = 2, 5 and 10 despite recall of 65.9 to 84.7 (for summaries, recall counts retrieving the summary of the right session); the authors attribute this to "loss of information" in summarizing. With no retrieved context the reader scored 22.4. The best retrieval result, 43.3 with five observations, is above gpt-3.5-turbo with a 16K window (35.9) but below gpt-4-turbo with the whole conversation (51.6), a comparison across readers. The authors' conclusion is that retrieval works best "when dialogues are transformed into a database of assertions (observations)" (Section 1) and that adding more retrieved observations lowered the score, so the signal-to-noise ratio of the context matters (Section 6.1).
Event summarization (Table 4, FactScore F1): gpt-4-turbo 48.9, gemini-1.0-pro 44.2, claude-3-sonnet 43.1, Llama-3-70B-Instruct with incremental summarization 37.8, Mistral 32.3. A manual analysis found five error types: missing information, hallucinated details, misread dialogue cues such as humor, wrong speaker, and unimportant exchanges treated as events (Section 6.2, Table 6).
Multimodal generation. Training MiniGPT-5 with retrieved observations as extra context gave the best MM-Relevance, and relevance fell as dialogue history grew (Section 6.3, Figure 4; values in Table 7 of the appendix, not checked for this note).
The prompts used
The dataset is written by prompted models, so its content follows from its prompts (Appendix A, Figures 5 to 10). The persona prompt asks for "crucial details" such as name, age, marital status and job. The event-graph prompt asks for events with dates and causes, and "as many causal connections as possible". The observation prompt asks for "objective factual information about the speaker that can be used as a database about them" and to avoid "abstract observations about the dynamics between the two speakers". The same kind of observation that the generating agents stored as memory is the retrieval unit that scored best on the QA task: the benchmark's text was produced by agents whose memory took the form that then won. The paper does not discuss this; it is this note's observation.
Limitations and open questions
- The authors' own (Section 8): the conversations are machine-generated and "may not fully reflect the nuances of real-world online conversations"; the web images lack the visual consistency of personal photos, so captions can replace them; English only; closed-source generators. They also flag that automatic metrics struggle with verbose LLM answers.
- Small and single-run. Ten conversations, one run per model, no intervals. Category sizes are uneven: open-domain has 96 questions.
- Partial-match F1. A correct paraphrase can score low and a wrong answer that shares words can score high; the authors constrain gold answers and model outputs toward the conversation's wording to limit this. Later users of the benchmark replaced F1 with LLM judges (see the Mem0 note).
- Adversarial category. Its scores move opposite to the others as context grows, so the overall score mixes two behaviors. Later evaluations, including Mem0's, dropped this category, Mem0 stating that "ground truth answers were unavailable" for it.
- Human baseline unstated. How 87.9 was measured is not described in the paper.
- Published critique. Zep's founders, in a post disputing Mem0's results (Chalef and Rasmussen, 6 May 2025, read in its June 2026 revision), argue that LoCoMo's conversations are short enough for modern context windows, that it lacks knowledge-update questions, and that it has data-quality problems: an unusable adversarial category, questions about image content absent from the BLIP captions, wrong speaker attributions and ambiguous questions. The post is by a vendor in a dispute over results on this benchmark, and it gives no counts for these problems.
Research questions
- How do memory methods rank on human-written long conversations? Collecting real multi-month conversations (with consent) and asking the same five question types would test whether results on generated dialogue transfer.
- Does the observation advantage depend on the generator's memory format? Building a version of LoCoMo with agents that stored raw turns or summaries, and re-running the three retrieval units, would show whether observations win because the text was written around them.
- How reliable are the gold answers? Having two annotators independently answer a sample of questions from the conversations, and measuring agreement per category, would give the error rate of the benchmark itself.
- How large is the adversarial trade-off at fixed model? Plotting adversarial and answerable scores for one model across many window sizes would separate the effect of context length from the effect of model choice in Table 2.
How it connects
- Mem0 evaluates its memory system on LoCoMo's ten conversations, without the adversarial category and with an LLM judge in place of F1. This note connects them.
- LongMemEval was built partly in response to benchmarks like this one: it criticizes LoCoMo for human-human (here, agent-agent) dialogue and for not testing knowledge updates or recall of what the assistant said (its Section 1). The LongMemEval paper makes that comparison.
- The components of agent memory describes a hierarchy of transcripts, notes and summaries; LoCoMo's three retrieval units (dialogue, observations, session summaries) are three of those layers, compared head to head. This note connects them.
- Retrieval-Augmented Generation and Passages and Chunking cover the choice of retrieval unit for documents; Table 3 here is that choice for a conversation. Dense Passage Retrieval covers the kind of dense retriever (DRAGON) used. This note connects them.
- Evaluating Retrieval-Augmented Systems separates retrieval errors from reader errors; LoCoMo's recall@k next to answer F1 allows the same split (summaries: high recall, low F1). This note connects them.
- GraphRAG pre-summarizes a document corpus; LoCoMo's summary retrieval unit is the conversational analogue, and it lost here to finer units. This note connects them. LAMA measures knowledge held in weights rather than in a store.
- The planned synthesis memory-for-language-model-agents will use this note for the benchmark that most memory systems report on.
References
- Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., and Fang, Y. (2024). Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870. https://doi.org/10.18653/v1/2024.acl-long.747 (arXiv:2402.17753, v1, which differs; see above) record · 3 pages
- Chalef, D., and Rasmussen, P. (2025, May 6; updated 2026, June 3). Lies, damn lies, & statistics: Is Mem0 really SOTA in agent memory? Zep blog. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/ record · 3 pages
- Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. (The agent architecture used to generate the conversations; entry as the arXiv version of this paper cites it; not read for this note.) record · 2 pages
- Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv:2305.14251. (The summarization metric; entry as the paper cites it; not read for this note.) record
- Lin, S.-C., Asai, A., Li, M., Oguz, B., Lin, J., Mehdad, Y., Yih, W., and Chen, X. (2023). How to train your DRAGON: Diverse augmentation towards generalizable dense retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6385–6400. (The retriever; entry as the paper cites it; not read for this note.) record