Evaluating Retrieval-Augmented Systems

How the quality of a retrieval-augmented system is measured, and how its failures are assigned to retrieval or to the reader. An exact accounting of every question by whether the answer was retrieved, whether the reader was right, and whether the reader knew the answer without retrieval, computed on recorded runs of Claude Opus 5.5 and Claude Haiku 5.5; answer correctness measured four ways, by substring, by two model judges and by reading judgments made by an instance of Claude Opus 5.5 (not a person), with a stratified estimate of each method's agreement with that reading; KILT's provenance-aware scores, RAGAS and eRAG; model judges of relevance as the literature reports them; and this site's own judging records, read for what the choice of judge does to a measured score. Four worked problems cover the accounting, a provenance-aware score, the estimate of a measure's agreement with the reading and the effect of one judge's grades on one query.

Asked "what is the song in Red Dead Redemption", Claude Opus 5.5 answers "Far Away" by José González when given no passages at all. Given BM25's top five passages, which describe the game, its sequel and its development but not the song, it answers that the search results name no specific song, and is scored wrong. Given the top twenty, among which the answering passage sits eighth, it answers "Far Away" again. When a retrieval-augmented system gives a wrong answer, was the retriever at fault, the reader, or the measure that called it wrong?

Evaluating a retrieval-augmented system means measuring three things and keeping them apart: whether retrieval put the needed evidence in front of the reader, whether the reader produced a correct answer from what it was given, and whether the measure of correctness is itself right. End-to-end accuracy mixes all three. The example above is a retrieval miss that became an error only because the reader, told to use "only the provided search results", declined to answer from memory.

Here is the route. First we account for every question exactly by three yes-or-no facts, which separates retrieval errors from reader errors and shows what retrieval added and what it took away. Then we turn to the measure of correctness and compare four of them on the same 450 answers. Next come scores that also ask for evidence, KILT's provenance-aware scores, and reference-free scores, RAGAS and eRAG, with the evaluation of the TREC 2024 RAG track. Then we take model judges of relevance, first as the literature reports them and then in this site's own judging records. The article assumes Open-Domain Question Answering for top-$k$ accuracy and exact match, Assembling the Context for the runs and the decomposition of accuracy it proves, and Relevance Judgments for kappa. The numbers are computed by tools/experiments/rag/evaluating-retrieval-augmented-systems.py.

Where the error is: an exact accounting

Assembling the Context proves that a reader's accuracy is a weighted sum, $P(C_k) = R(k)\,a_1(k) + (1 - R(k))\,a_0(k)$, of its accuracy when the answer is in the top $k$ passages and when it is not. Evaluation needs one more fact per question: whether the reader knew the answer without retrieval. Three yes-or-no facts per question give eight cells, and every question falls in exactly one.

For a question $q$, a retriever and a reader, let

  • $A = 1$ if a gold answer string occurs in the top $k$ passages (the event behind top-$k$ accuracy),
  • $Y = 1$ if the reader, given those passages, answers correctly,
  • $B = 1$ if the same reader answers correctly closed-book, with no passages.

The cells $(A, Y, B)$ partition the questions, so their counts add to the number of questions and any accuracy is a sum of cells. Four groups of cells carry names:

  • Retrieval failures are the questions with $A = 0$: the evidence was not in the context.
  • Reader failures are the questions with $A = 1$ and $Y = 0$: the evidence was there and the answer was still wrong.
  • Retrieval gains are the questions with $Y = 1$ and $B = 0$: right with retrieval, wrong without.
  • Retrieval losses are the questions with $Y = 0$ and $B = 1$: wrong with retrieval, right without.

Proposition 1. The end-to-end accuracy minus the closed-book accuracy equals (retrieval gains minus retrieval losses) divided by the number of questions. The same difference is what McNemar's test tests, and the two discordant counts of that test are exactly the gains and the losses.

Proof. End-to-end accuracy counts the cells with $Y = 1$ and closed-book accuracy the cells with $B = 1$. The cells with $Y = B$ appear in both counts and cancel. What remains is the number of cells with $Y = 1, B = 0$ minus the number with $Y = 0, B = 1$, which are the gains and the losses. McNemar's test compares two paired binary outcomes through exactly these discordant pairs. $\blacksquare$

The proposition makes a familiar point exact: a retrieval-augmented reader can be better than the same model without retrieval only by the margin of questions retrieval rescues over those it spoils.

The accounting on the recorded runs. The runs are those of Assembling the Context: NaturalQuestions-Open questions (CC BY-SA 3.0), BM25 over the 50,769 Wikipedia passages of the released examples of Liu et al. (2024) (each question's answering paragraph and its 19 hard distractors are in this corpus by construction, so retrieval failures here are failures against deliberately confusable passages; that article describes the corpus), Liu et al.'s prompt with a short-answer instruction, Opus 5.5 on 150 questions and Haiku 5.5 on 300, recorded on 18 November 2026. Correctness here is Liu et al.'s substring rule; the next section measures how often that rule departs from a reading of the responses. For the top five passages:

Cell: answer retrieved $A$, reader right $Y$, closed-book right $B$ Opus 5.5 Haiku 5.5
$A=1, Y=1, B=1$ 51 74
$A=1, Y=1, B=0$ (retrieval gain) 9 46
$A=1, Y=0, B=1$ (reader failure and retrieval loss) 1 3
$A=1, Y=0, B=0$ (reader failure) 10 21
$A=0, Y=1, B=1$ (answered from memory) 34 24
$A=0, Y=1, B=0$ 0 3
$A=0, Y=0, B=1$ (retrieval failure and retrieval loss) 18 62
$A=0, Y=0, B=0$ (retrieval failure) 27 67
Total 150 300

Read across the two columns.

  • Retrieval failures dominate. BM25 put an answer string in the top five for 71 of Opus's 150 questions and 144 of Haiku's 300. Of the errors, 45 of Opus's 56 and 129 of Haiku's 153 are on questions whose answer was never retrieved.
  • Opus 5.5 lost more than it gained. It gained 9 questions and lost 19, so its accuracy with five passages fell 10 questions below its closed-book accuracy, 94 against 104. Of the 19 losses, 18 are retrieval failures. A reading of those 18 responses (by an instance of Claude Opus 5.5, recorded in data/hand-losses.tsv) finds 3 explicit declines like the opening example's, 2 adjacent answers that also say the sources lack the one asked for, 11 confident wrong answers taken from or prompted by the passages (a coach's record for a team's, a song named like the show instead of its theme, 1967 for 1968), 1 answer to the question as literally worded that the annotation reads differently, and 1 that gives the right year without the annotated date, which the reading counts as correct. The 19th loss, whose passages held the answer, names only a town. On these counts, retrieval failures cost Opus mostly by misleading it, not by silencing it. It also answered 34 questions from memory when retrieval failed.
  • Haiku 5.5 lost more than it gained, though not detectably. It gained 49 and lost 65 ($p = 0.160$, Problem 1); its accuracy with five passages, 147, is below its closed-book 163. Its losses are more often declines: of the 62 losses with the answer not retrieved, 38 have a first line containing a phrase of refusal ("do not", "don't", "not mention", "no specific", "cannot" and similar; a word match, not a reading), against 5 of Opus's 18.
  • Reader failures with the answer present are the smaller share. Opus was wrong on 11 of the 71 questions whose answer it was shown, Haiku on 24 of 144. Some of these the reading judges correct, as the next section shows.

With twenty passages the picture changes: retrieval failures fall to 28 of Opus's questions and 52 of Haiku's, and both readers gain more than they lose, 18 against 8 for Opus and 72 against 32 for Haiku. The oracle condition, the answering passage alone, isolates the reader completely: $A = 1$ for every question by construction, so its 17 wrong answers for Opus and 31 for Haiku are reader failures or measurement errors, with no retrieval failure possible.

The accounting has a limit worth naming. A question in the cell $A = 1, Y = 1, B = 1$ may have been answered from memory with the passage ignored; the cells record outcomes, not how the reader used its input. Salemi and Zamani (2024) measure usefulness more directly with eRAG: each retrieved document is given to the reader alone, and the reader's downstream score with that document becomes the document's relevance label, aggregated over the ranking by a standard metric. They report that this agreed better with end-to-end performance than relevance labels did, with gains in Kendall's $\tau$ of 0.168 to 0.494 over the baselines they compared, and at much lower cost than end-to-end evaluation.

Measuring the answer

Every cell above rests on a judgment of correctness. The measures in common use disagree, and on these runs the disagreement is larger than most of the differences the measures are used to detect.

Four measures. We compared, on Opus 5.5's 450 responses in three conditions (closed-book, the answering passage tenth of twenty, and BM25's top five):

  • Substring, Liu et al.'s rule: an annotated answer, normalized, appears in the first line of the response. Exact match, defined in Open-Domain Question Answering, is stricter still.
  • Haiku judge and Opus judge: the model is shown the question, the annotated answers ("any one is acceptable") and the response, and asked whether the response answers with one of them "or with an answer that means the same thing", judging "only the answer the response gives, not its explanation". Low effort. A reply was read as its leading digit (after any Markdown emphasis, or after a leading "Answer:" or "Verdict:"), or as a final line consisting of the digit, emphasis allowed; a reply without a verdict within 512 tokens was asked again with room for 2,048, after which every response had a verdict.
  • Reading: the response read against the annotated answers under the same rule, by the writer of this article, an instance of Claude Opus 5.5 working as an agent, without seeing the automatic verdicts. It is a careful reading, not an independent human judgment, and it shares a model with the Opus judge.

The sample. On 371 of the 450 responses the three automatic measures agree; on 79 they do not. All 79 were read, and 40 were drawn at random (seed 20261118) from the 371. The estimate of each measure's agreement with the reading weights the two strata by their sizes, as in Attribution and Citation.

Measure Agrees with reading on the 79 split On 40 of the 371 unanimous Estimated agreement on all 450
Substring 8 39 0.822
Haiku judge 66 39 0.951
Opus judge 73 39 0.966

On the split responses the substring rule called 64 responses wrong that the reading accepted, and 7 right that the reading rejected. The first kind are paraphrases and fuller answers: "health (hit) points" against the annotation "hit points or health points", "about 2.4 billion years ago" against "around 2.45 billion years ago", "the Normans" against "Norman invaders". The second kind lead with a wrong answer and mention the annotated one later, which the substring rule cannot tell from a right answer when both fit on the first line. The model judges' disagreements with the reading were fewer and on contestable cases: a question that asks which Telugu film a film remakes (the response said the original was a Tamil film and named its Telugu remake, the annotated answer; in two of the three conditions both judges rejected it), an answer naming a group that the annotation describes more broadly (the Spanish, for "European colonization", which Haiku rejected), and an answer naming a later group of settlers as the first (which Opus accepted).

Condition (Opus 5.5, 150 questions) Substring Haiku judge Opus judge Reading
Closed-book 104 122 128 128
Answer tenth of twenty 118 132 134 135
BM25 top five 94 108 111 111

The reading column uses the reading on the 79 split responses and on the 40 sampled unanimous ones, and the common verdict on the other 331. Under every measure the order of the three conditions is the same, and the closed-book accuracy stays above the BM25 accuracy, so the conclusions of the accounting above hold. But the levels move by 17 to 24 questions in 150 between the substring rule and the reading, more than the 10-question difference between closed-book and BM25 that the accounting explained. A comparison of two systems scored by one measure is safer than a comparison of either with an absolute standard, and neither is safe when the systems differ in how they word their answers.

The annotations themselves set a ceiling. NaturalQuestions-Open answers were annotated against Wikipedia at one time, and some questions have moved: one asks who the current president is, and another asks for "the new" release of a series. A response correct today can be wrong by the annotation and the reverse.

Scores that ask for evidence, and scores without references

KILT. Petroni et al. (2021) built KILT, a benchmark of knowledge-intensive tasks over one Wikipedia snapshot, with provenance: for each answer, the set of Wikipedia pages that supports it. Retrieval is scored at the page level by R-precision, $r/R$, where $R$ is the number of pages in a provenance set and $r$ the number of them among the top $R$ retrieved, taking the best provenance set; for most KILT datasets $R = 1$, and R-precision is precision at rank 1. The downstream scores are given KILT versions: accuracy, exact match, ROUGE-L and F1 earn points for an example only if its R-precision is 1, so a system scores only when it both answers and ranks complete evidence at the top. In the authors' words, the choice emphasizes "that systems must be able to explain their output with proper evidence, not simply answer."

On our runs the natural provenance is the passage annotated by NaturalQuestions, which is in the corpus by construction (the corpus is built from Liu et al.'s examples, as described in Assembling the Context), so R-precision here measures how often BM25 ranks it above its own hard distractors and the passages of other questions. BM25 ranked it first for 22 of Opus's 150 questions and 51 of Haiku's 300, so R-precision is 0.147 and 0.170. With the top five passages Opus answered correctly on 94 questions, but only 19 of those also had the annotated passage first; its KILT accuracy is $19/150 = 0.127$ against its accuracy of $94/150 = 0.627$. The gap is the share of correct answers that rest on memory or on another passage, and KILT's scores refuse to count them.

Without references. Es et al. (2024) proposed RAGAS, three scores computed by prompting a model, with no gold answers: faithfulness, the share of statements extracted from the answer that the model judges inferable from the retrieved context, $F = |V|/|S|$; answer relevance, the mean cosine similarity between the question and questions a model generates from the answer; and context relevance, the share of context sentences that a model extracts as needed for the question. On their WikiEval set of 50 questions, each RAGAS score agreed with human pairwise preferences in 95, 78 and 70 percent of comparisons, ahead of two simpler prompting baselines. Faithfulness is the reference-free relative of the citation recall of Attribution and Citation: it asks whether the answer is supported by the context, without asking which passage supports which claim.

The TREC 2024 RAG track. The two initial reports on the evaluation of the TREC 2024 Retrieval-Augmented Generation track, both of November 2024, combine two automatic methods with human calibration. Pradeep et al. (2024) report an initial nugget evaluation: a nugget is an atomic fact an answer should contain, an idea that goes back to the TREC question answering track of 2003, and their AutoNuggetizer framework uses language models both to create nuggets and to assign them to system answers. On 21 topics and 45 runs they report a strong correlation between the fully automatic evaluation and a mostly manual one. For relevance, Upadhyay et al. (2024) compared NIST's manual judgments with three alternatives using the UMBRELA model assessor on 77 runs from 19 teams, and report that system rankings by automatic judgments correlated highly with those by manual judgments, that having people post-edit or filter the model's judgments did not appear to raise the correlation, and that human assessors appeared stricter than UMBRELA.

Model judges of relevance

The model judges of the last two sections are a case of a general practice: asking a language model to make the judgments that evaluation needs. Relevance Judgments introduces it briefly. The published results are mixed in a specific way.

Thomas et al. (2024), working with labels from real searchers at Bing, report that a carefully selected model and prompt labeled relevance as accurately as human labelers. On TREC-Robust documents their 32 prompt variants reached Cohen's $\kappa$ of 0.20 to 0.64 against the TREC assessors (their Table 1). Among prompt features, asking for aspects (topicality and trustworthiness) raised $\kappa$ by 0.21 on average, while asking for several judges lowered it by 0.13. Paraphrasing a single good prompt 42 ways moved its mean $\kappa$ between 0.50 and 0.72, "a wide range from a single prompt design". Faggioli et al. (2023) set out the arguments for and against, including circularity: a model that judges relevance could also rank by it, and "if the model is used to judge relevance both for annotation and for retrieval, its evaluation would be overinflated", while systems built on other principles would be penalized. Zheng et al. (2023), judging chat answers rather than documents, describe position bias (favoring the first of two answers), verbosity bias and possible self-enhancement bias, and report that strong judges agreed with human preferences over 80 percent of the time, about as often as humans agreed with each other.

The runs of this unit cannot settle whether a model judge is trustworthy, for a reason the circularity argument names. On answers, the Opus judge's estimated agreement with the reading was 0.966 and the Haiku judge's 0.951. On citation support it was 0.935 and 0.853 (Attribution and Citation). The judge that agrees best with the reading is the one most like the reader, which is the circularity concern in miniature: agreement with a check that shares the judge's model cannot establish the judge's independence.

This site's judging records

This site's search is evaluated with graded relevance judgments, and most of the grades come from model judges. The records are in tools/search/eval/: the grades in qrels.tsv, one row per judgment set in judgment-sets.tsv, and the account of the judging in model-judging-2026-11-14.md. Pages to grade were pooled from the top 20 results of the rankers named in the pool file's ranks column, and graded from 0 to 3 against a written guide. The judges were calibrated on 150 query-page pairs, 25 topical queries with six pages each: the owner graded them in a randomized checklist with ranks hidden, a model judge (Claude Sonnet 5.5) graded them reading every page in full, and a second (Claude Sonnet 5.5) graded them by skimming each page's summary, headings and the sections bearing on the need. The records report, for the owner against the full-read judge, exact agreement 0.71 and quadratic-weighted $\kappa$ 0.88; owner against skim, 0.68 and 0.85; full read against skim, 0.85 and 0.94. The skim protocol then graded all 1,199 pairs pooled at the time, and later judgment sets graded the pairs added by new pages and new rankers. Measures of Agreement analyzes these 150 grades in full: the weights, the standard errors, and a sign test showing that the owner grades higher than the full-read judge more often than lower (30 disagreements to 13).

Two further readings of the same records belong here, because they bear on what a model judge does to an evaluation.

Agreement on relevant or not. Many uses of judgments need only a binary decision. Counting grades 2 and 3 as relevant, the owner and the full-read judge agree on 129 of the 150 pairs, 0.860, with Cohen's $\kappa$ 0.717; the owner and the skim judge on 0.840, $\kappa$ 0.676. In 14 of the 21 disagreements of the first pair, the owner called a page relevant that the model did not. The direction is the opposite of Upadhyay et al.'s finding for TREC, where the human assessors appeared the stricter; here the owner is the more lenient.

What the judge does to the score. The site's search is scored by nDCG@10. Over the 25 calibration queries, the live ranker's top 10 (as recorded in the pool file) has mean nDCG@10 0.766 when the 150 calibration pairs carry the owner's grades, 0.763 with the full-read judge's and 0.751 with the skim judge's; every other judged page keeps its model grade. The three means lie within 0.015 of one another. Per query the effect is larger: replacing the full-read grades by the owner's raises nDCG@10 for 13 queries and lowers it for 10, by up to 0.126 (Problem 4 works the largest case). The judges disagree on single grades about three times in ten, but a mean over 25 queries absorbs most of it, which is why the evaluation records concluded that skim judging was adequate with a person spot-checking. These are measured agreements on 150 pairs and one ranker; they do not say how a model judge would treat a ranker built on a model like itself, which is the circularity concern again.

What is proved here, what is cited, and what the experiment shows

Proved here. The eight-cell accounting partitions the questions, and the difference between end-to-end and closed-book accuracy equals retrieval gains minus retrieval losses, the discordant pairs of McNemar's test (Proposition 1); the stratified estimate of a measure's agreement with a reading. The decomposition of accuracy by retrieval success is proved in Assembling the Context.

Cited. The questions, passages, prompt and substring rule of the runs (Liu et al., 2024); KILT's provenance, R-precision and KILT scores (Petroni et al., 2021); RAGAS's three scores and WikiEval agreement (Es et al., 2024); eRAG and its reported correlation gains (Salemi and Zamani, 2024); the TREC 2024 RAG track's nugget evaluation and its relevance-assessment study (Pradeep et al., 2024; Upadhyay et al., 2024); model relevance labeling and its prompt sensitivity (Thomas et al., 2024); circularity (Faggioli et al., 2023); position, verbosity and self-enhancement biases (Zheng et al., 2023).

Observed. On runs recorded on 18 November 2026: the eight-cell accounts of Opus 5.5 and Haiku 5.5 with BM25; the agreement of four correctness measures with a reading on 450 Opus responses, the reading made by an instance of Opus 5.5; the provenance-aware accuracy with the annotated passage as provenance. On the site's records as committed (commit 817b13a): binary agreement of the calibration judges and the effect of the judge on nDCG@10 over 25 queries. All are observations of these runs and records, not general findings.

Worked problems

The problems are posed on the recorded runs and the site's records; the unit has no course assignments. The script recomputes every number.

Problem 1 (Haiku's account). For Haiku 5.5 with BM25's top five, use the table to find the closed-book accuracy, the accuracy with retrieval, the retrieval gains and losses, and check Proposition 1.

Solution. Closed-book right ($B = 1$): $74 + 3 + 24 + 62 = 163$. Right with retrieval ($Y = 1$): $74 + 46 + 24 + 3 = 147$. Gains ($Y = 1, B = 0$): $46 + 3 = 49$. Losses ($Y = 0, B = 1$): $3 + 62 = 65$. Then $147 - 163 = -16 = 49 - 65$, as the proposition requires. In McNemar's test the discordant counts are 49 and 65, and the exact two-sided $p$-value is 0.160: the run cannot distinguish the two accuracies, although 62 questions that Haiku answered closed-book were lost to retrieval failures.

Problem 2 (a provenance-aware score). For Opus 5.5 with the top twenty passages, 114 answers were correct by substring, the annotated passage was ranked first for 22 questions, and on 20 of those the answer was correct. Compute R-precision and the KILT version of accuracy, and explain why KILT accuracy can change with $k$ only through the reader's answers on the questions whose provenance is ranked first.

Solution. R-precision with $R = 1$ is the share of questions whose provenance is ranked first: $22/150 = 0.147$. KILT accuracy counts a correct answer only when R-precision is 1: $20/150 = 0.133$, against an accuracy of $114/150 = 0.760$. R-precision depends on the ranking alone, not on how many passages the reader sees; KILT accuracy can differ across $k$ only through the reader's answers on the 22 questions with provenance first (19 correct with five passages, 20 with one and with twenty).

Problem 3 (how far is the substring rule from the reading?). From the sample above, estimate the substring rule's agreement with the reading over all 450 responses.

Solution. The 79 split responses were read in full; the rule agreed on 8. The 371 unanimous responses were sampled at 40; the rule (which agrees with both judges there) agreed on 39, an estimated $371 \times \tfrac{39}{40} = 361.725$. The estimate is $(8 + 361.725)/450 = 0.822$. Almost all of the rule's disagreements are refusals of correct paraphrases: 64 of the 71 split responses on which it differs from the reading.

Problem 4 (one query, two judges). For the site's calibration query T38 (shown by number only), the live ranker's top 10 pages carry grades 3, 3, 2, 3, 2, 2, 2, 1, 2, 1 when the calibration pages take the owner's grades, and 2, 3, 2, 2, 2, 1, 2, 1, 2, 1 when they take the full-read judge's. Among all of the query's judged pages, the ten highest grades are 3, 3, 3 and seven 2s under the owner's grades, and 3 and nine 2s under the judge's. Compute nDCG@10 both ways, with gain $2^g - 1$ and discount $1/\log_2(r + 1)$.

Solution. The gains are 7 for grade 3, 3 for grade 2 and 1 for grade 1. With the owner's grades, $\mathrm{DCG} = \tfrac{7}{1} + \tfrac{7}{1.585} + \tfrac{3}{2} + \tfrac{7}{2.322} + \tfrac{3}{2.585} + \tfrac{3}{2.807} + \tfrac{3}{3} + \tfrac{1}{3.170} + \tfrac{3}{3.322} + \tfrac{1}{3.459} = 20.668$, and the ideal ordering gives $\mathrm{IDCG} = 22.154$, so nDCG@10 $= 0.933$. With the judge's grades, $\mathrm{DCG} = 14.233$ and $\mathrm{IDCG} = 17.631$, so nDCG@10 $= 0.807$. The judge's lower grades lower both the score and its ideal; the score falls more because the downgraded pages sit high in the ranking, where the discount is smallest. One query moves by 0.126, and the mean over 25 queries by 0.0025 (0.7658 against 0.7633).

What we have learned

A retrieval-augmented answer can be wrong because the evidence was not retrieved, because the reader failed to use it, or because the measure misjudged a correct answer, and the three can be separated. The eight-cell accounting assigns every question to retrieval or to the reader and shows what retrieval added and what it removed: on our runs, retrieval failures dominated, and Opus 5.5 lost more questions to them than it gained, mostly by giving wrong answers drawn from passages that lacked the right one (11 of its 18 retrieval-failure losses, by a reading), while Haiku 5.5's losses were more often declines. The substring rule disagreed with the reading judgments on about one response in six, mostly by refusing paraphrases, while model judges disagreed on about one in twenty to thirty; the closest agreement came from the judge most like the reader. Provenance-aware scores and reference-free scores each answer a narrower question, and the initial reports on the TREC 2024 RAG track combine automatic nuggets and model relevance judgments with human calibration. On this site, the choice of judge moved single queries by up to an eighth of the scale and the mean by less than two hundredths.

This closes the unit on retrieval-augmented generation. The collection continues with the engines underneath: answering a query without scoring every posting, keeping the index small and keeping it current, in Efficient Query Processing.

References

  • Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150–158. record
  • Faggioli, G., Dietz, L., Clarke, C. L. A., et al. (2023). Perspectives on large language models for relevance judgment. In Proceedings of ICTIR 2023, 39–50. https://doi.org/10.1145/3578337.3605136 record · 2 pages
  • Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638 record · 3 pages
  • Petroni, F., Piktus, A., Fan, A., et al. (2021). KILT: A benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2523–2544. record
  • Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., and Lin, J. (2024). Initial nugget evaluation results for the TREC 2024 RAG Track with the AutoNuggetizer framework. arXiv:2411.09607. record
  • Salemi, A., and Zamani, H. (2024). Evaluating retrieval quality in retrieval-augmented generation. In Proceedings of SIGIR 2024, 2395–2400. https://doi.org/10.1145/3626772.3657957 record · 2 pages
  • Thomas, P., Spielman, S., Craswell, N., and Mitra, B. (2024). Large language models can accurately predict searcher preferences. In Proceedings of SIGIR 2024, 1930–1940. https://doi.org/10.1145/3626772.3657707 record · 2 pages
  • Upadhyay, S., Pradeep, R., Thakur, N., Campos, D., Craswell, N., Soboroff, I., Dang, H. T., and Lin, J. (2024). A large-scale study of relevance assessments with large language models: An initial look. arXiv:2411.08275. record
  • Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track). record · 2 pages