Asked who wrote "Put Your Hand in the Hand", Claude Opus 5.5 read five retrieved passages and answered in seven sentences, each ending in bracketed numbers that point to the passages. The first sentence reads: "'Put Your Hand in the Hand' was written by Canadian songwriter Gene MacLellan [2]." The others go on to the song's first recording, its sales and its cover versions. The answer is correct. But passage [2] says that the song was "composed by Gene MacLellan and first recorded by Canadian singer Anne Murray"; it calls Murray Canadian and says nothing of MacLellan's nationality. Is the first sentence supported by what it cites?
Attribution is the relation between a generated statement and a source when the source supports the statement: a reader who accepted the source, and nothing else, would accept the statement. A citation is the system's claim that this relation holds for a particular statement and a particular source. Citation evaluation asks how often that claim is true, and how often a statement that needs a source has none. The first sentence of the answer above is a case where it is not quite true, and the reason is a single word.
Here is the route. First we state what "supported" means, following the standard of Rashkin et al. (2023). Then we define citation recall and citation precision as Gao et al. (2023) do, and prove the properties that follow from the definitions, including what the precision rule cannot see. Next we compute both measures on the answer above, by reading and with two model judges, and find that the judges reach the same totals from different statements. Then we report a recorded run: two models answering 50 questions with citations, every citation judged by two models, and the judges checked against reading judgments. Finally we list what automatic checks miss. The article assumes Assembling the Context for the setting and Relevance Judgments for the kappa statistic. The numbers are computed by tools/experiments/rag/attribution-and-citation.py from runs recorded by citation_runs.py.
What "supported" means
Rashkin et al. (2023) define attributable to identified sources (AIS) for a statement that can be interpreted on its own, a standalone proposition $s$ at a time $t$. The pair $(s, t)$ is attributable to a set of parts $P$ of a corpus when "a generic hearer will, with a chosen level of confidence, affirm the following statement: 'According to $P$, $s$', where $s$ is interpreted relative to time $t$" (their Definition 2). The generic hearer is an idealization; in practice, they note, judgments come from raters who differ in background knowledge.
Two features of the definition matter here. First, the statement must be a standalone proposition. A sentence such as "He discovered X-rays" is not one, because "he" needs the surrounding text; Rashkin et al. evaluate the sentence's explicature, the proposition it expresses in its context ("Wilhelm Röntgen discovered X-rays"). Second, the hearer affirms the statement according to $P$. Facts the hearer happens to know do not count: a sentence can be true and still not attributable to its source.
The first sentence of the opening answer fails on the second point. It is true that MacLellan was Canadian, but passage [2] does not say so, and a hearer who knew only passage [2] could not affirm it. Whether that one adjective should sink the whole sentence is a question of how strictly "affirm" is read, and different judges read it differently, as we shall see.
Liu, Zhang and Liang (2023) graded support in three levels. A citation gives full support when all of the information in the statement is supported by it, partial support when some of it is, and no support otherwise. They judged a statement by the AIS standard, within the context of the query and the response. Applying these definitions by hand to four commercial generative search engines, they found that on average 51.5 percent of generated sentences were fully supported by their citations and 74.5 percent of citations supported their sentences.
Citation recall and citation precision
Gao et al. (2023) built a benchmark, ALCE, for answers with citations and defined two measures that can be computed automatically. An answer is split into statements $s_1, \dots, s_n$, one per sentence. Statement $s_i$ cites a set $C_i$ of passages, written as bracketed numbers. A judge $\phi(\text{premise}, \text{hypothesis})$ returns 1 if the premise entails the hypothesis and 0 otherwise. Gao et al. used an NLI (natural language inference) model, TRUE, a T5-11B model fine-tuned on entailment datasets; here $\phi$ can be any judge, including a person.
Citation recall. Statement $s_i$ has recall 1 if and only if it cites something and its cited passages together entail it:
$$\mathrm{rec}(s_i) = \begin{cases} 1 & \text{if } C_i \neq \emptyset \text{ and } \phi(\mathrm{concat}(C_i), s_i) = 1, \\ 0 & \text{otherwise.} \end{cases}$$
The answer's citation recall is the mean of $\mathrm{rec}(s_i)$ over its statements, and a system's is the mean over its answers.
Citation precision. A citation $c \in C_i$ is irrelevant if it cannot support $s_i$ by itself and the other citations can do without it:
$$c \text{ is irrelevant} \iff \phi(c, s_i) = 0 \ \text{ and } \ \phi(\mathrm{concat}(C_i \setminus \{c\}), s_i) = 1.$$
Citation $c$ has precision 1 if and only if $\mathrm{rec}(s_i) = 1$ and $c$ is not irrelevant. The answer's citation precision is the mean over all its citations (an answer with no citations scores 0, as in Gao et al.'s released evaluation code; no answer in our run lacked citations), and a system's is the mean over answers. Gao et al. state the intuition: a citation is irrelevant if "(a) $c_{i,j}$ itself cannot support $s_i$ and (b) removing $c_{i,j}$ does not affect the rest of the citations to support $s_i$". They deliberately do not require a minimal set of citations, because "human writing often cites redundant sources to enhance credibility".
When $C_i = \{c\}$ has one element, condition (b) asks whether the empty set entails $s_i$. We read an empty premise as entailing nothing, as the formula's use of $\mathrm{concat}$ suggests, so a lone citation is never irrelevant and its precision equals the statement's recall. The scripts use this reading.
What the definitions imply
The properties below follow from the definitions. Two of them need an assumption about the judge: $\phi$ is monotone if adding passages to a premise never turns an entailment into a non-entailment, that is, $\phi(A, s) = 1$ implies $\phi(A \cup B, s) = 1$. Logical entailment is monotone. A model judge need not be, and a person reading a longer premise can be distracted, so we state where the assumption is used.
Proposition 1. For any judge $\phi$:
- An unsupported statement earns nothing. If $\mathrm{rec}(s_i) = 0$, every citation of $s_i$ has precision 0.
- A lone citation is scored by its statement. If $|C_i| = 1$, the citation's precision equals $\mathrm{rec}(s_i)$.
- Redundant support is not penalized. If every citation of $s_i$ supports it alone, $\phi(c, s_i) = 1$ for all $c \in C_i$, then no citation of $s_i$ is irrelevant, and all have precision $\mathrm{rec}(s_i)$.
- Jointly needed citations are all kept. If no proper subset of $C_i$ entails $s_i$, no citation is irrelevant.
- A partial supporter beside a full supporter is irrelevant. If $\phi(c_1, s_i) = 1$ and $\phi(c_2, s_i) = 0$ for $C_i = \{c_1, c_2\}$, then $c_2$ is irrelevant, however much of $s_i$ it supports.
If $\phi$ is monotone, also:
- Padding keeps recall and costs precision. Adding to a supported statement $s_i$ a citation $c'$ with $\phi(c', s_i) = 0$ leaves $\mathrm{rec}(s_i) = 1$, makes $c'$ irrelevant, and raises no other citation's score; so the answer's citation precision falls, unless it was already 0.
Proof. (1) Precision 1 requires $\mathrm{rec}(s_i) = 1$. (2) With $C_i = \{c\}$, condition (b) concerns the empty premise, which entails nothing, so $c$ is never irrelevant; its precision is then 1 exactly when $\mathrm{rec}(s_i) = 1$. (3) Condition (a) fails for every $c$, so none is irrelevant. (4) For each $c$, $C_i \setminus \{c\}$ is a proper subset, which by hypothesis does not entail $s_i$, so condition (b) fails. (5) For $c_2$: condition (a) holds since $\phi(c_2, s_i) = 0$, and $C_i \setminus \{c_2\} = \{c_1\}$ entails $s_i$, so (b) holds. (6) $\mathrm{concat}(C_i)$ entails $s_i$, so by monotonicity $\mathrm{concat}(C_i \cup \{c'\})$ does, and $\mathrm{rec}(s_i)$ stays 1. Condition (a) holds for $c'$ by assumption, and (b) holds because $C_i$, which is what remains when $c'$ is removed, entails $s_i$. No other citation $c$ of $s_i$ gains: the statement's recall was already 1, and if $c$ was irrelevant, then $\phi(c, s_i) = 0$ and $C_i \setminus \{c\}$ entailed $s_i$, so by monotonicity $(C_i \setminus \{c\}) \cup \{c'\}$ does too, and $c$ stays irrelevant. (A citation that scored 1 can become irrelevant, if $c'$ completes what the others lacked.) Citations of other statements are untouched. The answer thus gains a citation scoring 0 while no score rises, so its precision, a mean, falls unless it was 0. $\blacksquare$
Item 5 is the blind spot. A passage that supports half of a sentence, cited beside one that supports all of it, scores exactly like a passage on another subject. Gao et al. name the limitation themselves: their NLI judge "cannot detect the case of 'partially support'", which made their automatic precision lower than human precision. Their human evaluation gave a citation precision 1 when the sentence was fully supported by its citations and the citation at least partially supported it. Liu et al. (2023) count a partially supporting citation only when the citations together fully support the statement and no single citation does. In the case of item 5 their rule also scores the partial supporter 0, since another citation fully supports the statement alone. The rules differ when the full support comes only from several citations together. Gao et al. give that case in their Appendix E: $s_3$ cites [2][4][5], [2] supports part of $s_3$ that [4][5] also cover, and [4][5] together entail it. ALCE marks [2] irrelevant; Liu et al.'s rule counts it.
Item 6 says that recall and precision pull in opposite directions under padding, which is why both are reported. A system that cites every passage for every statement maximizes recall under a monotone judge and pays for it in precision; Liu et al. make the same point about citing every web page.
The opening answer, worked in full
The answer cites four of its five passages. Passage [1] is a chunk of the Wikipedia article on the song that ends mid-sentence with "covered in the 1970s by a number of other performers, including Elvis"; passage [4] is the next chunk, beginning "Presley (who also covered MacLellan's 'Snowbird'), Randy Stonehill, Frankie Laine, Donny Hathaway, Joan Baez". Passage [2] is the gold paragraph quoted above, and passage [5] describes a different song, "Put Your Hand in Mine". The seven statements and their citations:
| Statement (shortened) | Cites | Read | Haiku judge | Opus judge |
|---|---|---|---|---|
| $s_1$ written by Canadian songwriter Gene MacLellan | [2] | 0 | 1 | 0 |
| $s_2$ gospel pop song, first recorded by Anne Murray | [2] | 1 | 1 | 1 |
| $s_3$ 22nd best-seller of 1971 | [1] | 1 | 1 | 1 |
| $s_4$ covered in the 1970s by Presley, Laine, Hathaway, Baez | [1][4] | 1 | 1 | 1 |
| $s_5$ Bing Crosby's 1972 album | [4] | 1 | 1 | 1 |
| $s_6$ Ron Hynes's tribute "Godspeed" | [1] | 1 | 1 | 1 |
| $s_7$ not to be confused with "Put Your Hand in Mine" by Ewing and Wayne | [5] | 1 | 0 | 0 |
The "Read" column holds reading judgments made by an instance of Claude Opus 5.5 (the writer of this article), with the rule and the caveat stated in the section on checking the judges below; they are not a person's judgments. The judges' columns are the recorded verdicts of Haiku 5.5 and Opus 5.5 given the premise and the statement alone.
Statement $s_4$ needs both passages: [1] lacks the names after Presley, and [4] lacks the fact that they covered the song in the 1970s. All three judges agree that $\phi([1], s_4) = \phi([4], s_4) = 0$ and $\phi([1][4], s_4) = 1$. By item 4 of Proposition 1 neither citation is irrelevant.
With the reading judgments. Recall: $s_1$ fails, the other six pass, $6/7 = 0.857$. Precision: the answer has eight citations. The citation of $s_1$ scores 0 by item 1; the two citations of $s_4$ score 1 by item 4; the remaining five score their statements' recall by item 2, all 1. Precision is $7/8 = 0.875$.
With the Haiku judge. It accepts $s_1$ and rejects $s_7$: recall $6/7$, precision $7/8$, the same numbers from a different pair of statements.
With the Opus judge. It rejects both: recall $5/7 = 0.714$, precision $6/8 = 0.750$.
The disagreement on $s_7$ is about what the sentence claims. Passage [5] gives the writers and singer of "Put Your Hand in Mine". The sentence adds advice, that the two songs should not be confused. Read as a claim about the other song, it is supported; read as a claim that the two are distinct songs that a reader might confuse, the premise says nothing about the first song at all. Both model judges took the second reading. A measure computed from one judge's verdicts inherits that judge's reading of every sentence, and here two readings give recall from 0.714 to 0.857 for the same answer.
A recorded run: 50 cited answers, two answerers, two judges
The observations in this section are of one dated run, recorded on 18 November 2026 with every prompt and response, on the questions and passages of Assembling the Context. They describe these models on these questions and are not general findings. Anthropic's documentation says to omit temperature, top_p and top_k for both models (for Opus 5.5 they return an error; Anthropic, 2026a, 2026b), so a rerun would be a new observation.
Answers. The first 50 of the questions drawn there, all NaturalQuestions-Open questions, each with BM25's top five passages from the 50,769-passage corpus described in Assembling the Context: every passage of the question set of Liu et al. (2024), so that each question's answering paragraph and its 19 hard distractors are in it by construction. The questions are NQ-Open's (CC BY-SA 3.0) and the passages Wikipedia text (CC BY-SA), as released with the code of Liu et al. (2024) under the MIT licence; the passage quoted in the worked example is one of them. The prompt is Gao et al.'s VANILLA instruction, verbatim (their Table 23): "Write an accurate, engaging, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, use [1][2][3]. Cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite a minimum sufficient subset of the documents." The passages follow in the format Document [n](Title: t) text, then the question. Gao et al. also prepend two worked demonstrations; we did not. Opus 5.5 and Haiku 5.5 each answered all 50, with low effort.
Statements. Each answer is split into sentences at a full stop, question mark or exclamation mark followed by a capital letter, a digit or a quotation mark, with line breaks also ending a sentence and with citation markers after a full stop kept with their sentence. An initialism such as "U.S." or a title such as "Dr." does not end a sentence. A heading, a bold line or a list line is kept as a statement of its own. The bracketed numbers of a sentence are its citations; the statement given to a judge has the markers removed.
Judges. For every statement with a citation, the checks that the definitions need: $\phi(\mathrm{concat}(C_i), s_i)$, and for statements with two or more citations $\phi(c, s_i)$ and $\phi(\mathrm{concat}(C_i \setminus \{c\}), s_i)$ for each citation. Each check shows the judge the premise passages and the statement, with this instruction: "Does the premise entail the hypothesis? The hypothesis is entailed if a reader who accepts everything in the premise, and nothing else, would affirm it as true. Reply with the single digit 1 if it is entailed and 0 if it is not." Haiku 5.5 and Opus 5.5 each judged every check, with low effort. A reply was read as its leading digit (after any Markdown emphasis, or after a leading "Answer:" or "Verdict:"), or as a final line consisting of the digit, emphasis allowed; replies that gave neither within 512 tokens were asked again with room for 2,048, after which every check had a verdict.
Results.
| Opus answers | Haiku answers | |
|---|---|---|
| Statements | 382 | 195 |
| Statements with a citation | 271 | 143 |
| Statements with two or more citations | 15 | 16 |
| Citations | 286 | 159 |
| Checks judged | 331 | 207 |
| Citation recall, Haiku judge | 0.625 | 0.571 |
| Citation recall, Opus judge | 0.633 | 0.611 |
| Citation precision, Haiku judge | 0.842 | 0.775 |
| Citation precision, Opus judge | 0.846 | 0.820 |
| Judges agree (checks) | 290 of 331 | 188 of 207 |
| Cohen's $\kappa$ between the judges | 0.669 | 0.772 |
Recall and precision are Gao et al.'s: means over each answer's statements or citations, then over the 50 answers.
Three observations follow.
- Uncited sentences dominate the recall. Opus 5.5 wrote 111 statements with no citation at all, 29 percent of its statements, and every one has recall 0 by definition. Sorted into disjoint groups in this order: 10 are bold lines used as headings and 8 are lead-ins ending in a colon, which the splitter keeps as statements (Gao et al. split with a sentence tokenizer, which would join a heading to the following text); 23 more are an answer's first sentence, which often states the answer before cited sentences explain it; 12 more contain a word about the sources ("provided", "search results", "sources", "documents" or "passages", a word match, not a reading), usually to say what the sources do not contain; the other 58 are scattered through the answers. Of the 271 statements that did cite, the Opus judge found 237 supported; leaving out the 18 headings and lead-ins, 237 of 364 statements are supported, 0.651 counted over statements rather than averaged over answers.
- Precision is high because most statements cite one passage. By item 2 of Proposition 1, a lone citation scores its statement's recall, so precision is close to recall among cited statements. Only 15 of Opus's statements cited two or more passages, the only ones on which a citation can be irrelevant.
- The two judges agree on the totals more than on the checks. For Opus's answers the judges' recall differs by 0.008, but they disagree on 41 of the 331 checks, and their kappa, 0.669, falls just below the 0.67 that the textbook scale quoted in Relevance Judgments takes as the lower limit of fair agreement. Different statements fail under each judge, as in the worked example.
The run cost $0.32 for the 100 answers and $0.99 for the 1,076 judgments (538 checks, each judged by both models), at batch prices.
The judges checked against reading
A model judge is a measuring instrument, and its readings need checking against an independent standard. Here the standard is a set of reading judgments: the writer of this article read the premise and statement of each sampled check, without the judges' verdicts, and labeled it under one stated rule. The reader is itself an instance of Claude Opus 5.5, working with the passages open rather than through the interface the judges used. Its judgments are therefore not independent of the Opus judge in the way a person's would be, and agreement between the two may partly reflect a shared way of reading. That is a limit on what this check can show, stated before its results.
The rule. A statement is entailed if every factual element of it is stated in the premise or necessarily follows from it; an attribute the premise does not give, such as a nationality, a first name or a role, makes it not entailed; a pronoun or a heading fragment is read with its plain antecedent.
The sample. Of the 331 checks on Opus's answers, the two judges disagreed on 41. All 41 were read, and 40 more were drawn at random (seed 20261118) from the 290 on which the judges agreed. Reading the whole disagreement stratum and a sample of the agreement stratum is a stratified sample: the strata are read at different rates, so the results are reported per stratum and combined with weights.
| Disagreements (41, all read) | Agreements (40 of 290 read) | Estimated agreement over all 331 checks | |
|---|---|---|---|
| Haiku judge agrees with the reading | 7 | 38 | 0.853 |
| Opus judge agrees with the reading | 34 | 38 | 0.935 |
On the 40 random checks both judges match the reading on 38, necessarily together, since the judges agree there. On the disagreements the reading sides with Opus 34 times and with Haiku 7 times. Of the 34 disagreements on which Haiku differed from the reading, it accepted a statement the reading rejected 18 times and rejected one the reading accepted 16 times; it differs in both directions. The estimate in the last column weights each stratum by its size: for the Opus judge, $(34 + 290 \times \tfrac{38}{40})/331 = 0.935$.
The cases behind the disagreements are of a few kinds: an added attribute (a first name, "French", "holiday album", a nationality), a premise cut off mid-sentence because passages are 100-word chunks, two consecutive chunks that support a statement only together, and statements whose antecedent sits in the previous sentence. Each is a place where "would a reader affirm it" has more than one defensible answer.
What automatic checks miss
The definitions are exact; the checks they rest on are not. The run above shows each of the following.
- Segmentation. Citation recall is a property of statements, and a statement is whatever the splitter makes of a sentence. An earlier version of our splitter cut "the U.S. Armed Forces" after "U.S.", producing a fragment with no citation and recall 0. Liu et al. (2023) also note that one sentence can hold several claims and that a marker's scope is ambiguous.
- Context. A judge shown "It went on to become the 22nd best-seller of 1971" must guess what "it" is. AIS judges the explicature in context; an NLI-style check sees the sentence alone.
- Truncated passages. A 100-word chunk often begins or ends mid-sentence, and a statement drawn from two consecutive chunks is supported by neither alone, as $s_4$ above shows.
- Strictness. Whether "Canadian songwriter" is supported by a passage that calls only the singer Canadian is a judgment about how literal "according to $P$" is. The two model judges differed on exactly such cases.
- Support is not correctness. A statement can be fully supported by a cited passage that is itself out of date or wrong. In a related run of Evaluating Retrieval-Augmented Systems, Opus answered a question about a state senate's majority leader from passages describing an earlier holder of the office, and said so. Gao et al. measure correctness separately for this reason, and report that a closed-book model with citations added afterwards (their POSTCITE) reached strong correctness but poor citation quality.
Gao et al. checked their automatic metrics against human judgments and report Cohen's kappa of 0.698 for citation recall and 0.525 for citation precision between human and automatic evaluation, and accuracies of 85.1 and 77.6 percent when the human annotations are taken as gold labels. These are the judge's agreement with people on their data; ours, above, is a model's agreement with a model's reading on ours.
What is proved here, what is cited, and what the experiment shows
Proved here. The six properties of citation recall and precision in Proposition 1, the last under the stated monotonicity assumption; the computation of both measures on the worked example under three judges; the stratified estimate of a judge's agreement.
Cited. The AIS definition, explicatures and the generic hearer (Rashkin et al., 2023); full and partial support, the four engines' 51.5 and 74.5 percent, and the precision rule for partial support (Liu, Zhang and Liang, 2023); the ALCE measures, their NLI judge, the VANILLA instruction, the treatment of partial support, the human-evaluation agreement and the POSTCITE result (Gao et al., 2023); the questions and passages of the run (Liu et al., 2024); the models' sampling parameters (Anthropic, 2026a, 2026b).
Observed in one recorded run. The recall, precision and judge agreement of Opus 5.5 and Haiku 5.5 answers on 50 NaturalQuestions-Open questions with BM25 passages, judged by Haiku 5.5 and Opus 5.5 on 18 November 2026, and the agreement of each judge with reading judgments made by an instance of Opus 5.5. With 41 disagreements and 40 sampled agreements, the estimates have wide uncertainty, and the reader's independence from the Opus judge is limited.
Worked problems
The problems are posed on Gao et al.'s published example and on the recorded run; the unit has no course assignments. The script recomputes every number.
Problem 1 (the published example). Gao et al.'s Figure 3 shows an answer of three statements: $s_1$ cites [1][2], $s_2$ cites [3], and $s_3$ cites [2][4][5]. In the figure the judge finds that [1][2] entails $s_1$, [3] does not entail $s_2$ and [2][4][5] entails $s_3$; that [2] alone does not entail $s_3$ while [4][5] does; and it marks [1], [2] in $s_1$ and [4], [5] in $s_3$ as not irrelevant. Compute recall and precision.
Solution. Recall: $s_1$ and $s_3$ pass and $s_2$ fails, so recall is $2/3 = 0.667$. Precision: $s_2$'s citation [3] scores 0 because its statement fails (item 1). In $s_3$, citation [2] is irrelevant: it fails alone and [4][5] succeed without it, so it scores 0. The other four citations belong to supported statements and are not irrelevant, so each scores 1. Over the six citations that is $4/6 = 0.667$, the 66 percent the figure shows for each measure.
Problem 2 (padding). Suppose Opus had added to each of its seven statements in the worked example a citation of passage [3], which describes the other song. Assume that [3] is inert: adding it to any premise never changes a verdict, and it supports no statement alone. Using the reading judgments, what are recall and precision?
Solution. Because [3] is inert, every verdict on a premise that includes it equals the verdict without it. Recall: each statement's cited set now includes [3], with the same verdict as before, so recall is unchanged at $6/7 = 0.857$. Precision now counts 15 citations. The seven new citations of [3]: on the six supported statements each fails alone while the original citations still entail the statement, so each is irrelevant and scores 0 (item 6); on $s_1$ the statement is unsupported, so it scores 0 (item 1). The eight original citations: removing one now leaves the other original citations plus [3], which has the same verdict as the other original citations alone, so each original citation is irrelevant exactly when it was before, and the seven that scored 1 still do. Precision is $7/15 = 0.467$. Recall cannot see the padding; precision falls from 0.875 to 0.467.
Problem 3 (agreement of the judges). On the 331 checks of Opus's answers, both judges said 1 on 228, only Haiku said 1 on 22, only Opus on 19, and both said 0 on 62. Compute the observed agreement and Cohen's kappa.
Solution. Observed agreement $p_o = (228 + 62)/331 = 290/331 = 0.876$. Haiku said 1 on $250/331$ checks and Opus on $247/331$, so chance agreement is $p_e = \tfrac{250}{331}\cdot\tfrac{247}{331} + \tfrac{81}{331}\cdot\tfrac{84}{331} = \tfrac{61{,}750 + 6{,}804}{109{,}561} = 0.626$, and $\kappa = (p_o - p_e)/(1 - p_e) = 0.669$. The judges agree on nearly nine checks in ten, but most checks are easy acceptances, so much of that agreement would occur by chance; Measures of Agreement proves why a common category inflates raw agreement.
Problem 4 (estimating a judge from a stratified sample). Using the sample above, estimate the Haiku judge's agreement with the reading over all 331 checks, and say why reading only a random sample of 81 checks would have been a worse design.
Solution. The disagreement stratum, 41 checks, was read in full, and Haiku matched 7. The agreement stratum, 290 checks, was sampled at 40, of which Haiku matched 38, an estimated $290 \times \tfrac{38}{40} = 275.5$. The estimate is $(7 + 275.5)/331 = 0.853$. A simple random sample of 81 from 331 would have contained about $81 \times 41/331 \approx 10$ disagreements, and the judges differ only there; reading all of them puts the effort where the two judges' accuracies can differ.
What we have learned
A citation is a claim that a statement is attributable to a source, and the AIS standard says what that means: a hearer who accepted the source alone would affirm the statement. Citation recall asks whether each statement is supported by what it cites, and citation precision whether each citation is needed. The definitions imply that lone citations are scored by their statements, that redundant and jointly needed citations are both kept, that padding lowers precision without touching recall, and that a partial supporter beside a full one cannot be told from noise. In the recorded run, the measures were set mostly by sentences with no citation and by how literally a judge reads "according to". Two model judges gave nearly equal totals from different statements, and the one closer to the reading judgments was the one most like the reader, which is the next article's question in general form.
How far can a model judge be trusted, for citations, for answers and for relevance, and how are the errors of retrieval separated from the errors of the reader? That is the subject of Evaluating Retrieval-Augmented Systems.
References
- Anthropic (2026a). Migrating to Claude Opus 5.5. Claude Platform documentation. https://platform.claude.com/docs/en/models/opus-5-5/migration-guide (accessed 18 November 2026). record · 2 pages
- Anthropic (2026b). Claude Haiku 5.5. Claude Platform documentation. https://platform.claude.com/docs/en/models/haiku-5-5/overview (accessed 18 November 2026). record · 2 pages
- Gao, T., Yen, H., Yu, J., and Chen, D. (2023). Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 6465–6488. record
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638 record · 3 pages
- Liu, N. F., Zhang, T., and Liang, P. (2023). Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, 7001–7025. record
- Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., and Reitter, D. (2023). Measuring attribution in natural language generation models. Computational Linguistics, 49(4), 777–840. https://doi.org/10.1162/coli_a_00486 record