Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory

Chhikara and colleagues at Mem0 describe a memory layer that has an LLM extract candidate facts from each new message pair and then decide, against the most similar stored memories, to ADD, UPDATE, DELETE or leave them (NOOP), plus a graph variant that stores entity-relation triples in Neo4j; on LoCoMo without its adversarial questions, scored by a lenient LLM judge, Mem0 reached 66.88 and the graph variant 68.44 against 72.90 for passing the whole conversation, with far lower latency and context tokens, and the company's comparison with Zep was disputed in public by Zep, whose own re-run gave Zep 75.14.

Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, Deshraj Yadav (Mem0) · 2025-04-28 · original

Most memory systems for chat agents store something from the conversation and retrieve it later. This paper's system decides, every time a new exchange arrives, which facts in it are worth keeping and how each relates to what is already stored: new, a refinement of an existing memory, a contradiction of one, or nothing new. An LLM makes both decisions. A second version stores the facts as a graph of entities and labelled relations. On the LoCoMo benchmark, scored by an LLM judge, both versions scored higher overall than retrieval over raw chunks and than the other memory systems tested, but lower than simply giving the model the whole conversation (72.90); Mem0 did so at under a tenth of that baseline's p95 latency and about a fifteenth of its context tokens (Table 2; our arithmetic). The authors build and sell Mem0, and several of the systems they compare against are competitors' products. Zep, one of them, published a dispute of the comparison within days; it is reported below with its sources.

Version of record. The paper is arXiv 2504.19413, version 1 of 28 April 2025 (the date above), the only arXiv version. Crossref records a version in the ECAI 2025 proceedings (Frontiers in Artificial Intelligence and Applications, DOI 10.3233/FAIA251160, dated 21 October 2025) with the same title and authors; this note could not retrieve that copy and follows arXiv v1. Whether the proceedings version changes any number is unknown. Code is linked from mem0.ai/research.

Terms used in this note

  • Memory. In this paper, a short natural-language fact about a conversation's participants stored in a vector database (Mem0), or an entity-relation triple stored in a graph (Mem0g, written Mem0g below).
  • Message pair. The current message and the one before it, "typically" a user message and an assistant reply (Section 2.1). On LoCoMo both messages come from the two human-like speakers.
  • J (LLM-as-a-Judge score). The share of answers a judge model labels CORRECT against the gold answer, in percent (Section 3.2, Appendix A).
  • p50, p95 latency. The median and 95th-percentile time, for the memory search alone and for the full response including the answer (Section 3.2).

What is stored, how it is retrieved, what enters the context

Mem0: extraction (Section 2.1, Figure 2). When a message pair (mt−1, mt) arrives, an LLM is prompted with a running summary of the whole conversation (refreshed asynchronously by a separate module), the m = 10 previous messages, and the new pair, and returns a set of candidate facts "specifically from the new exchange".

Mem0: update (Section 2.1, Algorithm 1 in Appendix B). For each candidate fact, the s = 10 most similar stored memories are retrieved by embedding similarity. The LLM sees them with the fact, through a function-calling ("tool call") interface, and chooses one of four operations: ADD a new memory when "no semantically equivalent memory exists"; UPDATE an existing memory "with complementary information"; DELETE a memory "contradicted by new information"; or NOOP. The paper chooses the LLM over "a separate classifier" for this decision. In Algorithm 1, an UPDATE replaces the old memory only if the new fact carries more information. All LLM steps used GPT-4o-mini.

Mem0g: the graph variant (Section 2.2, Figure 3). Memories are a directed labelled graph: nodes are entities with a type, an embedding and a creation timestamp; edges are relation triples such as (Alice, lives_in, San_Francisco). Two LLM stages extract entities and then relations between them. For each new triple, the source and destination are embedded and matched to existing nodes above a similarity threshold t, creating nodes as needed. A conflict-detection step finds existing relations the new one may contradict, and an LLM "update resolver" marks them invalid "rather than physically removing them to enable temporal reasoning". The store is Neo4j, and the LLM is again GPT-4o-mini.

Retrieval. For Mem0, the paper's description of answering is the prompt (Appendix A): the memories of each speaker are listed, with timestamps. It does not state in the text how many memories are retrieved per question or how they are ranked; Table 2 reports 1,764 tokens of memory context per question for Mem0 and 3,616 for Mem0g. Mem0g retrieves two ways (Section 2.2): an entity-centric search that finds the query's entities, matches them to nodes and collects their incoming and outgoing relations into a subgraph, and a "semantic triplet" search that matches the embedded query against text encodings of every triple and keeps those above a threshold.

What enters the answering model's context (Appendix A). For Mem0: "Memories for user" for each of the two speakers, the question, and instructions to analyze the memories, to "prioritize the most recent memory" when memories contradict, to convert relative time expressions to dates using the memory timestamps, and to keep the answer under "5-6 words". Mem0g's prompt adds a block of "Relations" for each speaker and an instruction to analyze them.

The evaluation

Data (Section 3.1). The ten conversations of LoCoMo, which this paper describes as "approximately 600 dialogues and 26000 tokens on average" with about 200 questions each; LoCoMo's own paper gives 16,618.1 tokens per conversation (its Table 5). Mem0 counts its token figures with tiktoken's cl100k_base encoding (Section 3.2); LoCoMo does not name its tokenizer or say what its count includes (timestamps, speaker names, image captions), and neither paper explains the difference. The adversarial category was excluded "because ground truth answers were unavailable". By LoCoMo's Table 5, that leaves 1,540 questions (841 single-hop, 282 multi-hop, 321 temporal, 96 open-domain; our arithmetic from LoCoMo's counts).

The judge (Section 3.2, Appendix A). The paper describes it as "a separate, more capable LLM" and does not name the model. Its prompt, adapted from one released by the MemGPT authors (Packer et al., 2023), tells it the gold answer will usually be short and that "you should be generous with your grading": an answer is CORRECT "as long as it touches on the same topic as the gold answer", and for time questions as long as it refers to the same date or period. It returns CORRECT or WRONG. No validation of the judge against people is reported. "Due to the stochastic nature of J evaluations", each method was run 10 times and the mean and one standard deviation are reported; the text does not say whether the memories were rebuilt for each run or only the judging repeated. F1 and BLEU-1 are also reported. Temperature was 0 where applicable.

Baselines (Section 3.3).

  • Published LoCoMo systems: LoCoMo's own pipeline, ReadAgent, MemoryBank, MemGPT and A-Mem, with F1 and BLEU-1 taken from earlier published results "where gpt-4o-mini was used". Only A-Mem was re-run, to obtain J scores (A-Mem*).
  • LangMem (open source), with gpt-4o-mini.
  • RAG: each conversation cut into chunks of 128 to 8,192 tokens, embedded, and the top k = 1 or 2 chunks given as context.
  • Full context: the whole conversation, about 26,000 tokens, in the prompt.
  • OpenAI: ChatGPT's memory feature with gpt-4o-mini. Whole conversations were pasted into chat sessions with a prompt asking it to create memories with timestamps; all resulting memories were then given as context, "intentionally granting the OpenAI approach privileged access to all memories rather than only question-relevant ones". Its latency excludes the extraction step.
  • Zep, a commercial memory platform, used through its hosted service with timestamps preserved alongside the messages.

What it finds

By question type (Table 1, J, mean ± one standard deviation over 10 runs):

Method Single-hop Multi-hop Open-domain Temporal
Mem0 67.13 ± 0.65 51.15 ± 0.31 72.93 ± 0.11 55.51 ± 0.34
Mem0g 65.71 ± 0.45 47.19 ± 0.67 75.71 ± 0.21 58.13 ± 0.44
Zep 61.70 ± 0.32 41.35 ± 0.48 76.60 ± 0.13 49.31 ± 0.50
LangMem 62.23 ± 0.75 47.92 ± 0.47 71.12 ± 0.20 23.43 ± 0.39
OpenAI 63.79 ± 0.46 42.92 ± 0.63 62.29 ± 0.12 21.71 ± 0.20
A-Mem* 39.79 ± 0.38 18.85 ± 0.31 54.05 ± 0.22 49.91 ± 0.31

Mem0 led on single-hop and multi-hop, Mem0g on temporal, and Zep on open-domain. The graph did not help on single-hop or multi-hop questions; the authors suggest "potential inefficiencies or redundancies" in graph representations for integrative questions (Section 4.1). They attribute OpenAI's low temporal score to "missing timestamps in most generated memories" despite the prompt.

Overall, with cost (Table 2):

Method Context tokens Search p50 / p95 (s) Total p50 / p95 (s) Overall J
Full context 26,031 none 9.870 / 17.117 72.90 ± 0.19
Mem0g 3,616 0.476 / 0.657 1.091 / 2.590 68.44 ± 0.17
Mem0 1,764 0.148 / 0.200 0.708 / 1.440 66.88 ± 0.15
Zep 3,911 0.513 / 0.778 1.292 / 2.926 65.99 ± 0.16
Best RAG (k = 2, 256-token chunks) two chunks of 256 0.255 / 0.699 0.802 / 1.907 60.97 ± 0.20
LangMem 127 17.99 / 59.82 18.53 / 60.40 58.10 ± 0.21
OpenAI 4,437 none 0.466 / 0.889 52.90 ± 0.14
A-Mem 2,520 0.668 / 1.485 1.410 / 4.374 48.38 ± 0.15

The abstract's "26% relative improvements" over OpenAI is 66.88 against 52.90 (a ratio of 1.264, our arithmetic), and its "91% lower p95 latency" and "more than 90%" token saving are Mem0 against full context (1.440 against 17.117 seconds, 1,764 against 26,031 tokens: reductions of 91.6% and 93.2%, our arithmetic); Section 4.3 rounds the same latency reduction to "92%", and gives 85% for Mem0g (2.590 seconds). The conclusion's "5%, 11%, and 7% relative improvements in single-hop, temporal, and multi-hop" questions over the best other method can be reproduced from Table 1's J scores as follows (our arithmetic; the paper does not show the calculation): single-hop, Mem0 67.13 against OpenAI 63.79 (5.2%); multi-hop, Mem0 51.15 against LangMem 47.92 (6.7%); temporal, base Mem0 55.51 against A-Mem 49.91 (11.2%). Mem0g's temporal score, 58.13, is 16.5% above A-Mem, so the conclusion's temporal figure appears to be the base system's, not the higher of the two. The paper states that full context "still achieves the highest J score" and argues that memory systems offer "a more practical trade-off" (Section 4.3). Where latencies were measured from (machine, network, region) is not stated; the hosted services' latencies include network transit and Mem0's are described as benefiting from "infra improvements" (Section 4.4).

Store size and timing (Section 4.5). Mem0's memory for a conversation averaged about 7k tokens and Mem0g's about 14k; Zep's graph exceeded 600k tokens, which the authors attribute to a summary cached at every node and facts on edges. They also report that Zep's retrieval "often failed" right after memories were added and did better after "several hours", and that Mem0g's graph construction finishes "in under a minute even in worst-case scenarios". No measurements are given for these timing observations.

The prompts used

All of Mem0's behavior that the evaluation measures runs through prompts: the extraction prompt, the update decision, the graph extractors and resolver (not printed in the paper), the answer prompts (printed), and the judge prompt (printed). The answer prompt's instructions (prefer the most recent memory, convert relative dates, answer in a few words) apply to Mem0 and Mem0g; the paper does not print the answer prompts used for the baselines, so it cannot be checked whether every system answered under the same instructions. The judge prompt's leniency ("touches on the same topic") sets how strict "correct" is; a stricter judge could change both the absolute scores and the gaps.

Disputed comparisons

The comparison with Zep was disputed in public by Zep, and the paper's MemGPT baseline was questioned by Letta, the company of the MemGPT authors. All three parties sell competing memory products, and the exchanges are company blog posts and a GitHub issue, not peer-reviewed work. This note read each in full; it does not adjudicate.

  • Zep's post (Daniel Chalef and Preston Rasmussen, Zep blog, 6 May 2025; the copy read shows "Updated Jun 03, 2026"). It argues that Mem0 configured Zep wrongly: both LoCoMo speakers were assigned the user role in a structure meant for one user and one assistant; timestamps were appended to message text instead of Zep's created_at field; and Zep's searches were run "sequentially instead of in parallel", inflating latency. With its own implementation and code, Zep reports a J score of 75.14 ± 0.17 and a p95 search latency of 0.632 seconds, against Mem0's 65.99 and 0.778 for Zep. It also criticizes LoCoMo itself (see the LoCoMo note). A correction note in the post says an earlier version "erred" in computing Zep's score; the post's subtitle in the copy read still says Zep "outperforms it by 24%" while the corrected text says "by 10%".
  • Mem0's reply (GitHub issue 5 on getzep/zep-papers, opened 8 May 2025 by Deshraj Yadav, an author of this paper and Mem0's co-founder and CTO). It says Zep's original 84% counted correct answers from the excluded adversarial category in the numerator but not the denominator; that Zep changed the answer prompt from the one used for every other system in the paper, and the search template from the one in Zep's own earlier DMR benchmark against MemGPT; and that Zep reported a single run. Re-running Zep's code with only categories 1 to 4, the shared prompt and ten runs, Mem0 reports 58.44 ± 0.20 for Zep. On the configuration points it says it followed Zep's documentation and Zep's own earlier benchmark code; that created_at was not accepted by zep.memory.add in zep-cloud 2.10.1, the release current at the time, and was added in zep-cloud 2.12.1 on 1 May 2025; and that it ran all systems sequentially to imitate users.
  • Zep's response (same issue, Daniel Chalef, 12 May 2025). Zep accepted the calculation error, gave the corrected 75.14 ± 0.17 "over 10 runs", replied that it "performed 10 independent runs for Zep, not just one", stood by its other criticisms, said the prompt change clarified how Zep timestamps events, said created_at has existed in Zep's Message type since a zep-python change dated 8 May 2024, said the latency point concerned the two search queries that build Zep's context for each question being run in series rather than in parallel, not the order of users' messages, and asked Mem0 for its datasets. The issue was closed for inactivity on 19 May 2025.
  • Letta's post (Letta blog, 12 August 2025). Letta says its team "was unable to determine a way to backfill LoCoMo data into MemGPT/Letta without significant refactoring" and that "Mem0 did not respond to requests for clarification" on how the MemGPT numbers were computed. In this paper, MemGPT is one of the "established" LoCoMo baselines whose F1 and BLEU-1 are selected from results "where gpt-4o-mini was used" (Section 3.3); the paper does not name the publication those numbers come from. Letta reports 74.0% on LoCoMo for a Letta agent on GPT-4o-mini given only file tools (semantic search and grep over the conversation placed in a file), with tool rules that make it start by searching and stop by answering; the post does not say which judge scored it.

What the reader can take from this: on LoCoMo, Zep's score depends on who configured it (65.99 in this paper, 58.44 in Mem0's re-run, 75.14 in Zep's), and no independent re-run settling the matter was found for this note.

Limitations and open questions

  • One benchmark, one model. All results are on LoCoMo's ten conversations with GPT-4o-mini. LoCoMo's paper notes its dialogues are LLM-generated.
  • Lenient, unidentified, unvalidated judge. The judge model is not named, its prompt asks for generosity, and no human check is reported.
  • Variance of what? The ± values are standard deviations over 10 runs that may repeat only the judging; if so, they do not capture variation in memory construction.
  • Baselines not run in the same harness. The older systems' F1 and BLEU-1 come from other publications; OpenAI's run gave it every memory; Zep's configuration is disputed.
  • Below full context. On a benchmark whose conversations fit in the context window, the memory systems did not match giving the model everything; the benefits shown are cost and latency.
  • Deletions are not measured. The paper does not report how often ADD, UPDATE, DELETE and NOOP were chosen, or whether DELETE ever removed a correct memory. LoCoMo has no knowledge-update questions to test updating.
  • Interest. The authors are Mem0's team (the paper's contact address is research@mem0.ai) and evaluate their product against competitors' products; the arXiv version has no competing-interests statement.

Research questions

  1. How often does the update step delete or overwrite correct memories? Logging every operation on LongMemEval's knowledge-update questions, and checking each DELETE and UPDATE against the gold facts, would measure the cost of letting the LLM decide.
  2. Do the rankings hold under a strict judge? Re-scoring the same answers with a judge told to require the gold answer, and with a human-validated sample, would show how much of the gap between systems depends on leniency.
  3. What does an independent harness give for Mem0, Zep and a filesystem agent? Running all three with the same answer prompt, the same model, parallel and sequential search, and both teams' configurations, would settle the disputed numbers.
  4. Does the graph help when relations matter? Testing Mem0 and Mem0g on questions built to need two or more relation hops, rather than LoCoMo's categories, would test the authors' explanation of the multi-hop result.

How it connects

  • LoCoMo is the benchmark used; this paper drops its adversarial category and replaces its F1 with a judge. This note connects them.
  • LongMemEval found that storing extracted facts instead of the text loses information except on multi-session questions; Mem0 stores only extracted facts. Zep's post recommends LongMemEval over LoCoMo. This note connects them.
  • GraphRAG also has an LLM extract entities and relations into a graph, for documents and with communities; Mem0g does it incrementally for a conversation and resolves conflicts. Knowledge Graphs covers both kinds of store. This note connects them.
  • The components of agent memory reports that a fixed rule telling Claude to "record only failures" cut its gain from memory, and that newer models do better when left to judge what to store; Mem0 fixes the decision in an extraction prompt and four operations. This note connects them; neither cites the other.
  • Retrieval-Augmented Generation and Passages and Chunking cover the chunk retrieval used as a baseline here; Evaluating Retrieval-Augmented Systems covers model judges and their leniency. This note connects them. LAMA concerns facts stored in weights.
  • The planned synthesis memory-for-language-model-agents will draw on this note for extraction-and-update memory and for the state of comparisons between memory products.

References

  • Chhikara, P., Khant, D., Aryan, S., Singh, T., and Yadav, D. (2025). Mem0: Building production-ready AI agents with scalable long-term memory. arXiv:2504.19413 (v1, 28 April 2025). Also in ECAI 2025, Frontiers in Artificial Intelligence and Applications, https://doi.org/10.3233/FAIA251160 (per Crossref; not retrieved). record
  • Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., and Fang, Y. (2024). Evaluating very long-term conversational memory of LLM agents. In Proceedings of ACL 2024 (Volume 1: Long Papers), pages 13851–13870. record · 3 pages
  • Chalef, D., and Rasmussen, P. (2025, May 6; updated 2026, June 3). Lies, damn lies, & statistics: Is Mem0 really SOTA in agent memory? Zep blog. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/ record · 3 pages
  • Yadav, D. (2025, May 8). Revisiting Zep's 84% LoCoMo claim: Corrected evaluation & 58.44% accuracy. GitHub issue 5, getzep/zep-papers, with reply by D. Chalef (2025, May 12). https://github.com/getzep/zep-papers/issues/5 record
  • Letta (2025, August 12). Benchmarking AI agent memory: Is a filesystem all you need? Letta blog. https://www.letta.com/blog/benchmarking-ai-agent-memory record
  • Packer, C., Fang, V., Patil, S. G., Lin, K., Wooders, S., and Gonzalez, J. E. (2023). MemGPT: Towards LLMs as operating systems. (Source of the judge prompt the paper adapted; entry as the paper cites it; not read for this note.) record · 2 pages
  • Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., and Chalef, D. (2025). Zep: A temporal knowledge graph architecture for agent memory. arXiv:2501.13956. (The Zep baseline; entry as the paper cites it; not read for this note.) record