From Local to Global: A GraphRAG Approach to Query-Focused Summarization

Edge and colleagues at Microsoft build an index by having an LLM extract entities, relationships and claims from every chunk of a corpus, partition the resulting graph into nested Leiden communities and pre-write a summary of each community, then answer a corpus-wide question by mapping it over the summaries and reducing the partial answers; on two corpora of about one million tokens, an LLM judge (GPT-4, per the introduction; no model identifier given) comparing answers in pairs preferred GraphRAG over chunk-retrieval RAG on comprehensiveness (72 to 83%) and diversity (62 to 82%), with mixed results on empowerment, and a claim-counting check agreed with the judge on 69 to 78% of its decided comparisons.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, Jonathan Larson (Microsoft) · 2024-04-24 · original

Retrieval-augmented generation answers well when the answer sits in a few passages. It has no good answer to a question about a whole corpus, such as "What are the main themes in the dataset?" (abstract), because no passage holds the answer and the top-ranked chunks are a sample, not a summary. This paper builds a different index for such "global sensemaking" questions. An LLM reads every chunk once and extracts an entity graph; a community-detection algorithm splits the graph into nested groups of related entities; the LLM writes a report for each group in advance; and a question is answered by asking it of every report at one level of the hierarchy and summarizing the partial answers. On two corpora of about one million tokens, an LLM judge comparing answers in pairs preferred this to a chunk-retrieval baseline on comprehensiveness and diversity, with win rates of 62% to 83%, and gave mixed results on "empowerment", the criterion about helping the reader make informed judgments.

Version of record. The paper exists only as an arXiv preprint, 2404.16130. Version 1 (24 April 2024, the date above) was titled "From Local to Global: A Graph RAG Approach to Query-Focused Summarization". Version 2 (19 February 2025, marked "Preprint. Under review.") renamed the method GraphRAG, renamed the baseline from "naïve RAG" to "vector RAG", added significance tests (Appendix G) and a second experiment that checks the judge with claim counts (Sections 4.2 and 5.2), and dropped a limitation that v1 stated: the need "to validate our sensemaking questions and target metrics with end users". The win-rate ranges that the text of the first experiment quotes are the same in both versions. This note follows v2 and reports the v1 differences where they matter. The method is released as open-source software (github.com/microsoft/graphrag, named in Section 1); this note describes the paper's method, not the later software.

Terms used in this note

  • Global sensemaking question. The paper's term for a question that needs "global understanding of the entire dataset" rather than a fact from a few records (Section 1). The paper frames answering it as query-focused summarization (QFS).
  • Vector RAG (SS, "semantic search"). The baseline: retrieve text chunks by embedding similarity and add them to the context until a token limit is reached (Section 4.1.2).
  • Community. A group of entities more densely connected to each other than to the rest of the graph, found by the Leiden algorithm (Traag et al., 2019). Applied recursively, it gives a hierarchy; levels C0 (root, fewest communities) to C3 (lowest level used, most communities).
  • Community summary (community report). An LLM-written report of one community's entities, relationships and claims, written at indexing time.
  • Win rate. In one pairwise comparison on one question, the preferred answer scores 100, the other 0, and a tie 50 each; scores are averaged over five judge runs and over the 125 questions (Table 6 caption).

What the method stores, retrieves and puts in context

What is stored (indexing, Section 3.1). Documents are cut into chunks (600 tokens with 100-token overlaps in the experiments, Section 4.1.1). For each chunk an LLM is prompted, with few-shot examples that can be tailored to the domain, to list entities (name, type, description) and relationships between "clearly related" entities (source, target, description, a numeric strength), and optionally claims, "important factual statements about entities" such as dates and events (Section 3.1.2; prompt in Appendix E.1). A "self-reflection" step asks the LLM whether it missed entities, forcing a yes/no with a logit bias of 100, and if so tells it that "MANY entities were missed in the last extraction" to prompt another pass (Appendix A.2). Instances are merged by exact string match on entity names (Section 3.1.3); the descriptions gathered for each node and edge are summarized by the LLM into one description, and the number of duplicate relationship instances becomes the edge weight. The graph is partitioned with hierarchical Leiden (graspologic library), recursing until communities can no longer be split (Sections 3.1.4 and 4.1.3). Each community gets a report: for a leaf community, element descriptions are added in decreasing order of the combined degree of each edge's endpoints until the context limit; for a higher community whose elements do not fit, sub-community reports replace their elements' descriptions (Section 3.1.5). The report prompt asks for a title, a summary, an "impact severity rating" from 0 to 10, and 5 to 10 findings, each grounded with record identifiers (Appendix E.2).

So the store is three layers of LLM-written text: element descriptions attached to graph nodes and edges, the graph structure itself, and pre-written community reports at every level.

How it is retrieved (query time, Section 3.1.6). There is no ranking against the query. At the chosen level, all community reports are shuffled and packed into chunks of a fixed token size. Each chunk is given to the LLM with the question (map step), which writes a partial answer and a helpfulness score from 0 to 100; answers scored 0 are dropped. The partial answers are sorted by score and added to a new context until the limit (reduce step), and the LLM writes the final answer.

What enters the final context. Partial answers, not source text: the reduce step sees only the highest-scored partial answers, each written from community reports, which were themselves written from LLM-extracted descriptions. The answer prompts ask for citations to report identifiers ("[Data: Reports (...)]") and say "Do not include information where the supporting evidence for it is not provided" (Appendix E.4). A context window of 8k tokens was used throughout for summaries, partial answers and final answers (Section 4.1.3).

The evaluation

Corpora (Section 4.1.1). Podcast transcripts of Behind the Tech with Kevin Scott, 1,669 chunks (about 1 million tokens); and a news benchmark (MultiHop-RAG; Tang and Yang, 2024) of articles from September 2013 to December 2023, 3,197 chunks (about 1.7 million tokens). The indexes had 8,564 nodes and 20,691 edges (podcasts) and 15,754 nodes and 19,520 edges (news) (Section 5.1).

Questions (Section 3.2, Algorithm 1). An LLM was given a short description of each corpus, asked to describe K = 5 hypothetical users, N = 5 tasks per user and M = 5 questions per user-task pair that "require understanding of the entire corpus" and "do not require retrieval of specific low-level facts": 125 questions per corpus. The questions were generated from the description, not from the corpus text (Section 2.3). They are not released with the paper; Table 1 shows examples ("Which episodes deal primarily with tech policy and government regulation?").

Conditions (Section 4.1.2). Six: GraphRAG at community levels C0 to C3; TS, the same map-reduce applied to shuffled source chunks instead of reports; and SS, vector RAG. The answer prompts and the 8k window were the same across conditions. The 8k size was chosen in a preliminary test of 8k, 16k, 32k and 64k windows "to determine the optimum context size for our baseline condition (SS)"; 8k won on comprehensiveness (average win rate 58.1%) and was comparable on the other criteria (Appendix C).

Judges: how they were set up. This is the part of the paper a later synthesis most needs to read exactly.

  • Who judged. "An LLM evaluator" (Section 3.3). The paper does not give the judge's model identifier. Section 1 of v2 says the results hold "when using GPT-4 as the LLM", and the indexing used a gpt-4-turbo endpoint (Section 4.1.3); neither version states separately which model judged or whether it was the model that wrote the answers.
  • What it saw. The question, two answers labelled "Answer 1" and "Answer 2", and one criterion's text (prompt in Appendix F.1). The conditions' names were not shown, but the answers' citations differ by condition: the answer prompts ask GraphRAG answers to cite "[Data: Reports (...)]" and use "Sources" in place of "Reports" for the SS and TS conditions (Appendix E.4), so a judge could tell a community-report answer from the others by its citations. The paper does not discuss this; it is this note's observation. The paper also does not say whether the order of the two answers was randomized or swapped; in its one worked example GraphRAG is Answer 1 (Table 5).
  • What it returned. A winner (1 or 2) or 0 if the answers are "fundamentally similar and the differences are immaterial", with a short reason, as JSON.
  • The criteria (Appendix F.2). Comprehensiveness: "How much detail does the answer provide to cover all the aspects and details of the question?" Diversity: "How varied and rich is the answer in providing different perspectives and insights on the question?", whose full text adds that "A diverse answer should also provide different sources and evidence to support the answer". Empowerment: "How well does the answer help the reader understand and make informed judgements about the topic without being misled or making fallacious assumptions", judged on "explaining and providing reasoning and sources behind the claims". Directness, a "control criterion" for concision, which the authors expected to oppose comprehensiveness and diversity (Section 3.3).
  • Repetition and statistics. Each comparison was run five times and averaged (Figure 2 caption). Pairs of conditions were compared with Wilcoxon signed-rank tests with Holm-Bonferroni correction, after Shapiro-Wilk tests rejected normality (Appendix G). See Multiple Comparisons and the Bonferroni Correction for the correction.
  • Validation against people. None is reported. The paper cites earlier work in which LLM evaluations were "competitive with human evaluations" (Section 2.4). Its own check of the judge is Experiment 2, below, which uses another LLM-based method.

What it finds

Experiment 1, judge win rates (Figure 2, 125 questions per comparison, five runs each; p-values from Section 5.1 and Table 6):

  • Against vector RAG. All global conditions (C0 to C3 and TS) beat SS on comprehensiveness, 72% to 83% on podcasts and 72% to 80% on news (p < .001), and on diversity, 75% to 82% on podcasts (p < .001) and 62% to 71% on news (p < .01).
  • Empowerment. "Mixed results" both against SS and between GraphRAG and TS (Section 5.1). On podcasts, for example, SS beat C0 57% to 43% and TS beat SS 58% to 42% (Figure 2). The authors had an LLM read the judge's reasons and report that "specific examples, quotes, and citations" were judged key, which they suggest the extraction prompts may lose.
  • Directness. SS won every comparison (54% to 65%), as the control was designed to show.
  • Against source-text summarization (TS). Community reports gave "a small but consistent improvement" except at the root level: the best were 57% comprehensiveness (C2, podcasts, p < .001) and 64% (C3, news, p < .001), and 57% (p = .036) and 60% (p < .001) on diversity.
  • Cost. Per query, the root level C0 used 26,657 tokens of context on podcasts and 39,770 on news, against 1,014,611 and 1,707,694 for TS (Table 2): over 97% fewer, with comprehensiveness and diversity win rates over SS of 72% and 62% retained. C3 used 26% to 33% fewer tokens than TS. Indexing the podcast corpus took 281 minutes on one virtual machine against a public gpt-4-turbo endpoint (Section 4.1.3). The paper reports no dollar cost and no token count for indexing.

Experiment 2, claim counts (Sections 4.2 and 5.2, v2 only). Claimify (Metropolitansky and Larson, 2025, two of this paper's authors), an LLM-based method, extracted 47,075 unique factual claims from the answers, about 31 per answer. Comprehensiveness was measured as claims per answer and diversity as the number of clusters of claims (agglomerative clustering, complete linkage, distance 1 − ROUGE-L, thresholds 0.5 to 0.8). Every global condition and TS had more claims than SS on both corpora (Table 3; for example 34.18 for C0 against 25.23 for SS on news), significant at p < .05. On diversity, all global conditions beat SS at every threshold on podcasts, but on news only C0 did (Table 4). No significant difference was found among the global conditions or between them and TS. Where the five judge runs gave a majority winner (33% of comparisons for comprehensiveness, 39% for diversity), the judge's winner matched the claim-count winner in 78% and 69% to 70% of cases, which the authors call "moderately strong alignment".

The prompts used

The index, the answers and the evaluation are all prompt-defined, and the paper prints its prompts (Appendices E and F): the extraction prompt with its tuple format and one Federal Reserve example; the community report prompt, framed as helping "a human analyst to perform general information discovery", with its impact severity rating and grounding rules; the map and reduce answer prompts; and the judge prompt with four criterion texts. Two features of the wording bear on the results. The diversity and empowerment criteria both ask about sources and evidence, and the answer prompts for every condition require cited sources. The question generator was instructed to avoid questions needing "specific low-level facts", which is the kind vector RAG is built for. The paper does not vary any prompt's wording.

Limitations and open questions

  • The authors' own (Section 6.1): two corpora of about one million tokens; generalization to other domains and uses is untested; fabrication rates were not compared and would "strengthen the current analysis". Version 1 also called for validating the questions and criteria with end users.
  • No reference answers. Questions have no gold answers, so correctness is never scored; comprehensiveness and diversity are judged, not checked against the corpus.
  • One judge, unvalidated. The judge model is not identified precisely, its answer order is not reported, the answers it compares carry condition-specific citation labels ("Reports" or "Sources"), and its agreement with people is not measured. The claim-count check is itself LLM-based and comes from the same group.
  • Questions shaped toward the method. The questions were generated to be global, and the baseline's strength (specific facts) was excluded by instruction. The result is a claim about this class of question.
  • Single exact-match entity merging. The paper says softer matching is possible and that duplicates tend to fall into the same community (Section 3.1.3); it does not measure the effect.

Later work. Han et al. (arXiv:2502.11371, version 4 dated 7 November 2026, published at KDD 2026) re-ran community-based GraphRAG against RAG on four public query-based summarization datasets with human-written reference summaries (Section 5). Scored by ROUGE-2 and BERTScore against the references, community GraphRAG with global search "generally underperforms RAG". Repeating this paper's pairwise judging with GPT-4o on two of those datasets, they found that swapping the order of the two summaries produced "substantially different, and in some cases opposite, judgments" (Section 5.3); averaged over both orders, with GPT-4o judging on QMSum and ODSum-story, GraphRAG-Global's win rate was 0.91 and 0.79 on diversity and 0.29 and 0.16 on comprehensiveness (Table 18); a GPT-5.4 judge "reduces but does not eliminate" the order effect (Section 5.3). They recommend evaluating both orders and averaging. Their datasets differ from this paper's (whose questions, they note, are unreleased) and target specific roles or events rather than global themes, so the two studies do not test the same claim; they agree that global community summaries raise diversity. This note read Han et al.'s Sections 1 to 3, 5 and Appendix D, not its question-answering results in full.

Research questions

  1. Do the win rates survive order swapping? Re-judging this paper's answer pairs in both orders, as Han et al. recommend, would show how much of the 72% to 83% comprehensiveness margin is position.
  2. Do people prefer the same answers? Having analysts rate a sample of the pairs on the same four criteria, blind to condition, would validate (or not) the judge on this task.
  3. How often are global answers wrong? Checking the claims Claimify extracts from each condition's answers against the source corpus would give the fabrication comparison the authors call for.
  4. Where does the index lose detail? Tracing which source facts reach the element descriptions, the community reports and the final answer would test the authors' explanation of the empowerment results.
  5. What does indexing cost against querying? Recording the tokens spent building the graph and reports, against the tokens saved per query relative to TS, would give the break-even number of questions per corpus.

How it connects

  • Knowledge Graphs covers GraphRAG as one way to build a knowledge graph from text and use it for retrieval; this note gives the paper's evaluation in detail for that article.
  • Retrieval-Augmented Generation is the setting this paper departs from. Its "vector RAG" baseline is the retrieve-and-read pipeline of that unit, and Passages and Chunking and Assembling the Context cover the chunking and context-packing choices that the baseline here fixes at 600-token chunks and an 8k window. This note connects them.
  • Evaluating Retrieval-Augmented Systems treats model judges and their agreement with people; this paper's judges are a case with no reference answers and no human validation. This note connects them.
  • The components of agent memory reports wikis that Claude built from a document corpus helping on questions that span documents; a community report is a pre-written page of the same kind. This note connects them; neither cites the other.
  • LoCoMo and Mem0 store LLM-extracted facts or triples about conversations rather than documents; Mem0's graph variant extracts entities and relations much as GraphRAG does, without communities. LAMA tests the opposite store, the model's weights. This note connects them.
  • The planned synthesis memory-for-language-model-agents will draw on this note for its account of pre-summarized stores.

References

  • Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., and Larson, J. (2024, revised 2025). From local to global: A GraphRAG approach to query-focused summarization. arXiv:2404.16130 (v1, 24 April 2024; v2, 19 February 2025). record · 2 pages
  • Han, H., Ma, L., Wang, Y., Shomer, H., Guo, K., Lei, Y., Qi, Z., Hua, Z., Long, B., Liu, H., Aggarwal, C., and Tang, J. (2026). RAG vs. GraphRAG: A systematic evaluation and key insights. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '26). https://doi.org/10.1145/3770855.3817575 (arXiv:2502.11371, v4) record
  • Traag, V. A., Waltman, L., and van Eck, N. J. (2019). From Louvain to Leiden: Guaranteeing well-connected communities. Scientific Reports, 9(1). (The community-detection algorithm; entry as the paper cites it; not read for this note.) record
  • Tang, Y., and Yang, Y. (2024). MultiHop-RAG: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv:2401.15391. (Source of the news corpus; entry as the paper cites it; not read for this note.) record
  • Metropolitansky, D., and Larson, J. (2025). Towards effective extraction and evaluation of factual claims. (The paper's reference for Claimify, given without venue; described here only as the paper describes it.) record