Measures of Agreement: Cohen's Kappa

How far two classifications of the same units agree beyond what chance would produce. Cohen's kappa for any number of categories, with its maximum given the margins proved; weighted kappa for ordered categories, and the proof that quadratic weights make it the concordance correlation coefficient; the large-sample variance derived by the delta method and checked by the bootstrap; kappa written in terms of prevalence and bias indices, which proves the two paradoxes of high agreement with low kappa; and why agreement and bias are separate questions, answered by kappa and by McNemar's test. Worked on repeated blood pressure readings of 4,782 NHANES adults and on the relevance grades given by the owner of this site and by two model judges, with worked problems on a paradox, the maximum kappa, a standard error by hand and kappa from means and variances.

The NHANES examination took up to three blood pressure readings at one sitting, and McNemar's Test for Paired Proportions showed that among 4,782 adults the first systolic reading was classified high slightly more often than the third. Placed in four bands, below 120, 120 to 129, 130 to 139, and 140 or more mm Hg, the two readings put the same adult in the same band 78.0% of the time. Two readings that had nothing to do with each other would also agree some of the time, because many adults fall in the lowest band on either reading. How much of the 78.0% is agreement that chance would not have produced, and how sure can we be of the answer?

Cohen's kappa (Cohen, 1960) measures it: the observed agreement minus the agreement expected by chance, divided by the most that the difference could be. Relevance Judgments defines it for two assessors who call documents relevant or not, derives the chance-agreement formula, compares Cohen's version with the version that pools the two assessors' rates, and shows how a rare category lowers kappa at the same raw agreement. This article goes further. (In the rest of this collection $\kappa$ denotes the kurtosis $\mu_4/\sigma^4$; in this article it denotes Cohen's coefficient.)

First we define kappa for any number of categories and prove the largest value it can reach once the two margins are fixed. Then we add weights for ordered categories and prove that quadratic weights turn kappa into a correlation coefficient. Next we derive the standard error, so that an observed kappa gets an interval. Then we write kappa in terms of prevalence and bias, which proves the two known paradoxes, and separate agreement from bias. Finally we apply all of it to the relevance grades behind this site's search evaluation.

Agreement beyond chance with several categories

Cross-classify the 4,782 adults by the band of the first reading (rows) and of the third (columns):

First \ Third <120 120–129 130–139 ≥140 Total
<120 1,807 184 7 0 1,998
120–129 228 667 164 2 1,061
130–139 6 200 408 103 717
≥140 3 18 138 847 1,006
Total 2,044 1,069 717 952 4,782

Write $n_{ij}$ for the counts, $p_{ij} = n_{ij}/n$ for the shares, and $p_{i+}$ and $p_{+j}$ for the row and column shares, the two classifications' distributions over the $k = 4$ categories. The observed agreement is the share on the diagonal,

$$p_o = \sum_i p_{ii} = \frac{1{,}807 + 667 + 408 + 847}{4{,}782} = \frac{3{,}729}{4{,}782} = 0.780.$$

The chance model. Suppose the two readings were independent, each keeping its own distribution over the bands. Then the probability that both fall in band $i$ is $p_{i+}p_{+i}$, and the chance agreement is

$$p_e = \sum_i p_{i+}p_{+i} = 0.293.$$

Under that model the expected diagonal counts are $np_{i+}p_{+i}$: 854.0, 237.2, 107.5 and 200.3, against 1,807, 667, 408 and 847 observed. Kappa is the share of the possible improvement on chance that the readings achieved:

$$\kappa = \frac{p_o - p_e}{1 - p_e} = \frac{0.780 - 0.293}{1 - 0.293} = 0.689.$$

As a population quantity, kappa is the same expression with the cell probabilities $\pi_{ij}$ in place of the shares, and the sample value estimates it. Kappa is 0 when the observed agreement equals the chance agreement and 1 when every unit is on the diagonal. For fixed margins, $p_e$ is fixed and kappa is smallest when the diagonal is as small as the margins allow; if they allow it to be empty (which needs $p_{i+} + p_{+i} \le 1$ for every $i$, since row $i$ and column $i$ share only the cell $(i, i)$), the minimum is $-p_e/(1 - p_e)$, the lower end of the range Shankar and Bangdiwala (2014) state. It can be as low as $-1$: two categories with $p_{12} = p_{21} = \tfrac12$ give $p_o = 0$, $p_e = \tfrac12$ and $\kappa = -1$.

How high can kappa go once the margins are fixed? Kappa reaches 1 only if every unit is on the diagonal, which requires the two margins to be identical. When they differ, perfect agreement is impossible, and kappa has a smaller ceiling.

Theorem 1 (maximum kappa). For fixed margins $p_{i+}$ and $p_{+i}$,

$$\kappa \le \kappa_{\max} = \frac{p_{\max} - p_e}{1 - p_e}, \qquad p_{\max} = \sum_i \min(p_{i+}, p_{+i}).$$

For two categories the bound is attained.

Proof. A diagonal cell is part of its row and of its column, so $p_{ii} \le p_{i+}$ and $p_{ii} \le p_{+i}$, hence $p_{ii} \le \min(p_{i+}, p_{+i})$. Summing, $p_o \le p_{\max}$. The chance agreement $p_e$ depends only on the margins, and kappa is increasing in $p_o$ when $p_e < 1$, so $\kappa \le \kappa_{\max}$. For two categories, suppose $p_{1+} \le p_{+1}$ (the other case is the same with rows and columns exchanged) and set $p_{11} = p_{1+}$, $p_{12} = 0$, $p_{21} = p_{+1} - p_{1+}$, $p_{22} = 1 - p_{+1}$. All four are nonnegative and the margins are the given ones. The diagonal sum is $p_{1+} + 1 - p_{+1} = \min(p_{1+}, p_{+1}) + \min(p_{2+}, p_{+2})$, since $p_{2+} = 1 - p_{1+} \ge 1 - p_{+1} = p_{+2}$. So $p_o = p_{\max}$. $\blacksquare$

For the bands, $p_{\max} = 0.989$ and $\kappa_{\max} = 0.984$: the two readings' distributions over the bands are so close that the ceiling barely binds, and the observed 0.689 is 0.700 of it. Reporting $\kappa/\kappa_{\max}$ separates two reasons for a low kappa: the classifications disagree about individual units, or their margins differ so much that they could not agree.

Weighted kappa for ordered categories

Unweighted kappa counts a first reading of 120 to 129 and a third of 130 to 139 as a disagreement, exactly as serious as below 120 and 140 or more. For ordered categories that is wasteful. Cohen (1968) generalized kappa so that each cell receives a degree of credit, "ratio-scaled degrees of disagreement (or agreement)" in his abstract's words.

Let $w_{ij}$ be agreement weights, with $w_{ii} = 1$ and $0 \le w_{ij} \le 1$. Weighted observed and chance agreement are

$$p_o(w) = \sum_{i,j} w_{ij}\,p_{ij}, \qquad p_e(w) = \sum_{i,j} w_{ij}\,p_{i+}p_{+j},$$

and weighted kappa is $\kappa_w = [p_o(w) - p_e(w)]/[1 - p_e(w)]$. Unweighted kappa is the case $w_{ij} = 0$ off the diagonal. Two choices for $k$ ordered categories are standard:

  • linear weights, $w_{ij} = 1 - |i - j|/(k - 1)$;
  • quadratic weights, $w_{ij} = 1 - (i - j)^2/(k - 1)^2$.

The disagreement form. With disagreement weights $v_{ij} = 1 - w_{ij}$,

$$\kappa_w = 1 - \frac{\sum_{i,j} v_{ij}\,p_{ij}}{\sum_{i,j} v_{ij}\,p_{i+}p_{+j}}.$$

Proof. Since $\sum_{i,j} p_{ij} = 1$ and $\sum_{i,j} p_{i+}p_{+j} = 1$, we have $1 - p_o(w) = \sum v_{ij}p_{ij}$ and $1 - p_e(w) = \sum v_{ij}p_{i+}p_{+j}$. Then $\kappa_w = \{[1 - p_e(w)] - [1 - p_o(w)]\}/[1 - p_e(w)] = 1 - [1 - p_o(w)]/[1 - p_e(w)]$. $\blacksquare$

In words: one minus the ratio of the observed mean disagreement to the mean disagreement expected under independence. This is the form in which the site's search evaluation computes it (tools/search/eval/metrics.py), and the script confirms that the two forms agree exactly for every weighting used here.

For the bands, linear weights give $p_o(w) = 0.924$, $p_e(w) = 0.576$ and $\kappa_w = 0.820$, and quadratic weights give $p_o(w) = 0.973$, $p_e(w) = 0.695$ and $\kappa_w = 0.910$. The weighted values are higher because almost all disagreements here are between adjacent bands, which the weights forgive in part.

Example (which weights?). The three numbers 0.689, 0.820 and 0.910 describe one table. None is wrong, and none can be compared with a kappa computed under different weights. A report of "kappa" for ordered categories needs its weights stated, as the site's evaluation records do ("quadratic-weighted kappa").

Quadratic weights and the concordance correlation

Quadratic weights have a property that the others lack: they turn kappa into a statement about means, variances and a covariance. Score the categories $0, 1, \ldots, k - 1$, and let $x_t$ and $y_t$ be the scores of unit $t$ under the two classifications. Write $\bar x$, $\bar y$ for the means and

$$s_x^2 = \frac1n\sum_t (x_t - \bar x)^2, \quad s_y^2 = \frac1n\sum_t (y_t - \bar y)^2, \quad s_{xy} = \frac1n\sum_t (x_t - \bar x)(y_t - \bar y)$$

for the variances and covariance, with divisor $n$.

Theorem 2. With quadratic weights,

$$\kappa_w = \frac{2s_{xy}}{s_x^2 + s_y^2 + (\bar x - \bar y)^2}.$$

Proof. Use the disagreement form. Quadratic disagreement weights are $v_{ij} = (i - j)^2/(k - 1)^2$, and the constant $(k - 1)^2$ cancels between numerator and denominator, so we may use $(i - j)^2$. The numerator is the observed mean squared difference of scores,

$$\sum_{i,j} (i - j)^2 p_{ij} = \frac1n\sum_t (x_t - y_t)^2 = s_x^2 + s_y^2 - 2s_{xy} + (\bar x - \bar y)^2,$$

by writing $x_t - y_t = (x_t - \bar x) - (y_t - \bar y) + (\bar x - \bar y)$, squaring and averaging; the cross terms with $\bar x - \bar y$ average to zero. The denominator is the mean squared difference when an $x$ drawn from the row distribution is paired with an independent $y$ from the column distribution. The same expansion applies, but now the covariance is zero:

$$\sum_{i,j} (i - j)^2 p_{i+}p_{+j} = s_x^2 + s_y^2 + (\bar x - \bar y)^2.$$

So $\kappa_w = 1 - [s_x^2 + s_y^2 - 2s_{xy} + (\bar x - \bar y)^2]/[s_x^2 + s_y^2 + (\bar x - \bar y)^2] = 2s_{xy}/[s_x^2 + s_y^2 + (\bar x - \bar y)^2]$. $\blacksquare$

The right-hand side is the sample form of Lin's (1989) concordance correlation coefficient, which he described as "the correlation between the two readings that fall on the 45 degree line through the origin" (his abstract). Compare it with Pearson's correlation $r = s_{xy}/(s_xs_y)$. Since $s_x^2 + s_y^2 \ge 2s_xs_y$, the concordance coefficient is at most $r$ in absolute value (when $s_x s_y > 0$, so that $r$ is defined), with equality, when $s_{xy} \ne 0$, only when the two variances are equal and the means coincide. Pearson's $r$ asks whether the points lie on some line; the concordance coefficient asks whether they lie on the line $y = x$. Fleiss and Cohen (1973) showed the related equivalence of weighted kappa with an intraclass correlation coefficient, as the title of their paper states.

For the bands (scores 0 to 3), the means are 1.153 and 1.121, the variances 1.386 and 1.359, and the covariance 1.250. The squared difference of means is only 0.00104, so the quadratic kappa of 0.910 is almost Pearson's $r$, 0.911. The script confirms the identity in exact rational arithmetic.

Example (what quadratic kappa cannot see). Because it depends on the table only through two means, two variances and a covariance, quadratic kappa can be blind to changes that matter. Warrens (2012) showed that for an odd number of categories, if one rater uses categories 1 and $k$ equally often, 2 and $k - 1$ equally often, and so on, then quadratic kappa "does not depend on the value of the center cell", the count of exact agreements on the middle category. The script checks this on the $3 \times 3$ table

1 2 3
1 18 6 2
2 5 30 4
3 3 5 18

whose first and third rows both total 26. Its quadratic kappa is 0.608, and it stays 0.608 exactly when 10, 100 or 1,000 agreements are added to the center cell. Unweighted kappa rises from 0.577 to 0.694 when 100 are added. Warrens recommends linear weights when a single index is wanted for an ordinal scale.

Proof of the center-cell property, from Theorem 2. Score the categories $0, \ldots, k - 1$ with $k$ odd, and let $c = (k - 1)/2$ be the center score. If the row rater uses scores $i$ and $k - 1 - i$ equally often for every $i$, the row scores are symmetric about $c$, so $\bar x = c$; adding units at the center cell $(c, c)$ keeps this true. Multiply the numerator and denominator of Theorem 2 by $n$. The numerator becomes $2\sum_t (x_t - c)(y_t - \bar y) = 2\sum_t (x_t - c)y_t$, since $\sum_t (x_t - c) = 0$. Using $\sum_t (y_t - \bar y)^2 + n(c - \bar y)^2 = \sum_t (y_t - c)^2$, the denominator becomes $\sum_t (x_t - c)^2 + \sum_t (y_t - c)^2$. A unit at $(c, c)$ adds $0$ to both sums, because each of its terms has the factor $x_t - c = 0$ or $y_t - c = 0$. So both are unchanged, and so is $\kappa_w$. $\blacksquare$

Kappa from a sample: the standard error

An observed kappa is an estimate, and it needs a standard error. Here is the plan. Kappa is a smooth function of the cell shares; the shares are multinomial, with known covariances; the delta method carries the covariances through the function.

The formal version. Write $\hat\kappa_w = f(\hat{\mathbf p})$, where $\hat{\mathbf p}$ is the vector of the $k^2$ cell shares. The Chi-Square Goodness-of-Fit Test proves that multinomial shares have covariance matrix $(\operatorname{diag}\boldsymbol\pi - \boldsymbol\pi\boldsymbol\pi^{\mathsf T})/n$. The Delta Method proves that a differentiable function of an asymptotically normal vector is asymptotically normal, with variance $\nabla f^{\mathsf T}\Sigma\,\nabla f$. With $g_{ij} = \partial f/\partial\pi_{ij}$, that is

$$\operatorname{Var}(\hat\kappa_w) \approx \frac1n\Big[\sum_{i,j}\pi_{ij}g_{ij}^2 - \Big(\sum_{i,j}\pi_{ij}g_{ij}\Big)^2\Big].$$

It remains to find the gradient. Write $P_o = \sum w_{ij}\pi_{ij}$ and $P_e = \sum w_{ij}\pi_{i+}\pi_{+j}$, so $\kappa_w = (P_o - P_e)/(1 - P_e)$. Then $\partial P_o/\partial\pi_{ij} = w_{ij}$. The cell $\pi_{ij}$ enters $\pi_{i+}$ and $\pi_{+j}$, so

$$\frac{\partial P_e}{\partial\pi_{ij}} = \sum_l w_{il}\pi_{+l} + \sum_l w_{lj}\pi_{l+} = \bar w_{i\cdot} + \bar w_{\cdot j},$$

which defines $\bar w_{i\cdot}$ and $\bar w_{\cdot j}$. By the quotient rule, and using $(1 - P_o)/(1 - P_e) = 1 - \kappa_w$,

$$g_{ij} = \frac{w_{ij}(1 - P_e) - (\bar w_{i\cdot} + \bar w_{\cdot j})(1 - P_o)}{(1 - P_e)^2} = \frac{w_{ij} - (\bar w_{i\cdot} + \bar w_{\cdot j})(1 - \kappa_w)}{1 - P_e}.$$

For the mean of the gradient, note that $\sum_{i,j}\pi_{ij}\bar w_{i\cdot} = \sum_i \pi_{i+}\bar w_{i\cdot} = P_e$, and likewise for $\bar w_{\cdot j}$. So $(1 - P_e)\sum\pi_{ij}g_{ij} = P_o - 2P_e(1 - \kappa_w) = \kappa_w - P_e(1 - \kappa_w)$, using $P_o = \kappa_w(1 - P_e) + P_e$. Putting the pieces together,

$$\operatorname{Var}(\hat\kappa_w) \approx \frac{\sum_{i,j}\pi_{ij}\big[w_{ij} - (\bar w_{i\cdot} + \bar w_{\cdot j})(1 - \kappa_w)\big]^2 - \big[\kappa_w - P_e(1 - \kappa_w)\big]^2}{n(1 - P_e)^2}.$$

For unweighted kappa, $\bar w_{i\cdot} = \pi_{+i}$ and $\bar w_{\cdot j} = \pi_{j+}$. The standard error is estimated by putting the sample shares in place of the probabilities. Fleiss, Cohen and Everitt (1969) gave large-sample standard errors for kappa and weighted kappa, and their abstract explains that the expressions in the original papers were incorrect, "having been derived from the contradictory assumptions of fixed marginal totals and binomial variation of cell frequencies". The derivation above assumes neither: the margins vary with the sample, as they do when $n$ units are drawn at random and both classifications recorded. The script checks the closed form against a numerical gradient for every table in this article.

Back to the readings. The standard errors and 95% intervals are:

Weights $\hat\kappa_w$ SE 95% interval
None 0.689 0.0082 0.673 to 0.705
Linear 0.820 0.0053 0.810 to 0.831
Quadratic 0.910 0.0033 0.904 to 0.917

A check by resampling. The bootstrap resamples the 4,782 pairs with replacement and recomputes kappa. Over 30 bootstraps of 1,000 resamples each (seeds 20,000,000 to 20,000,029), the standard deviation of unweighted kappa had median 0.0082 (interquartile range 0.0081 to 0.0083), matching the delta-method value. The theory of the bootstrap is the subject of Resampling Methods.

Under independence $\kappa = 0$, so a test of independence such as the chi-square test of The Chi-Square Test of Independence also tests whether there is any agreement beyond chance. The converse fails: $\kappa$ can be 0 in a table that is not independent. With three categories, shares $\tfrac19$ on the diagonal, $\tfrac29$ in cells $(1, 2)$, $(2, 3)$ and $(3, 1)$, and 0 elsewhere, every margin is $\tfrac13$, so $p_o = p_e = \tfrac13$ and $\kappa = 0$, yet the cells with share 0 show that the classifications are dependent. For two readings of one person's blood pressure neither hypothesis is interesting; the interval is the informative summary.

Prevalence, bias and the two paradoxes

Feinstein and Cicchetti (1990) named two paradoxes of kappa. Shankar and Bangdiwala (2014) state them: first, "lower kappa values despite high observed agreement" when the marginals are imbalanced; second, "higher kappa values for asymmetrical than for symmetrical imbalanced marginal totals". Relevance Judgments shows the first by example. Writing kappa in two indices makes both exact.

For a $2 \times 2$ table with shares $p_{11}, p_{12}, p_{21}, p_{22}$, Byrt, Bishop and Carlin (1993) proposed reporting, beside kappa, a prevalence index and a bias index, defined as (Shankar and Bangdiwala, 2014)

$$\text{PI} = p_{11} - p_{22}, \qquad \text{BI} = p_{12} - p_{21},$$

and the prevalence-adjusted bias-adjusted kappa $\text{PABAK} = 2p_o - 1$. The prevalence index is large in absolute value when one category dominates the agreements. The bias index is zero exactly when the two margins are equal.

Theorem 3. For a $2 \times 2$ table,

$$p_e = \frac{1 + \text{PI}^2 - \text{BI}^2}{2}, \qquad \kappa = \frac{\text{PABAK} - \text{PI}^2 + \text{BI}^2}{1 - \text{PI}^2 + \text{BI}^2}.$$

Proof. Since $p_{11} + p_{22} = p_o$ and $p_{12} + p_{21} = 1 - p_o$, the four shares are

$$p_{11} = \tfrac12(p_o + \text{PI}), \quad p_{22} = \tfrac12(p_o - \text{PI}), \quad p_{12} = \tfrac12(1 - p_o + \text{BI}), \quad p_{21} = \tfrac12(1 - p_o - \text{BI}).$$

So the margins are $p_{1+} = p_{11} + p_{12} = \tfrac12(1 + \text{PI} + \text{BI})$, $p_{+1} = \tfrac12(1 + \text{PI} - \text{BI})$, $p_{2+} = \tfrac12(1 - \text{PI} - \text{BI})$ and $p_{+2} = \tfrac12(1 - \text{PI} + \text{BI})$. Then

$$p_e = p_{1+}p_{+1} + p_{2+}p_{+2} = \tfrac14\big[(1 + \text{PI})^2 - \text{BI}^2 + (1 - \text{PI})^2 - \text{BI}^2\big] = \tfrac12\big(1 + \text{PI}^2 - \text{BI}^2\big).$$

Substituting, $p_o - p_e = \tfrac12(2p_o - 1 - \text{PI}^2 + \text{BI}^2)$ and $1 - p_e = \tfrac12(1 - \text{PI}^2 + \text{BI}^2)$; the halves cancel and $2p_o - 1 = \text{PABAK}$. $\blacksquare$

Corollary (the paradoxes). Hold $p_o$ fixed, with $p_o < 1$.

  1. For fixed BI, kappa decreases as $|\text{PI}|$ increases. Write $A = \text{PABAK} + \text{BI}^2$ and $B = 1 + \text{BI}^2$, so $\kappa = (A - y)/(B - y)$ with $y = \text{PI}^2$. Since $\text{PABAK} < 1$, $A < B$. The derivative in $y$ is $(A - B)/(B - y)^2 < 0$.
  2. For fixed PI, kappa increases as $|\text{BI}|$ increases. Now $\kappa = (A' + x)/(B' + x)$ with $x = \text{BI}^2$, $A' = \text{PABAK} - \text{PI}^2 < B' = 1 - \text{PI}^2$, and the derivative is $(B' - A')/(B' + x)^2 > 0$.

The first is the paradox of high agreement with low kappa: a dominant category makes chance agreement high, and kappa can then be small or negative at high observed agreement. The second is the paradox that a bias between the raters, which ought to count against their agreement, raises kappa: unequal margins lower the chance agreement. Shankar and Bangdiwala's scenarios, recomputed by the script, show both:

Scenario $p_o$ PI BI $\kappa$
2: 80, 10, 5, 5 0.85 0.75 0.05 0.318
3: 90, 5, 5, 0 0.90 0.90 0.00 −0.053
4: 45, 15, 25, 15 0.60 0.30 −0.10 0.130
5: 25, 35, 5, 35 0.60 −0.10 0.30 0.259
9: 85, 5, 5, 5 0.90 0.80 0.00 0.444
10: 70, 10, 0, 20 0.90 0.50 0.10 0.737

(The cells are $n_{11}, n_{12}, n_{21}, n_{22}$ out of 100.) Scenario 3 agrees on 90 of 100 units and has a negative kappa. Scenarios 9 and 10 have the same observed agreement, and the one with a bias has the higher kappa; most of the difference, by the corollary, comes from its smaller prevalence index.

PABAK depends on $p_o$ alone, so it removes both effects, and with them the correction for chance that was kappa's reason for existing. Byrt and colleagues recommended reporting the bias and prevalence indices beside kappa, according to Shankar and Bangdiwala (2014); the theorem shows that kappa, PI, BI and $p_o$ together say everything a $2 \times 2$ table says about agreement except its sample size.

The readings at 130 mm Hg. The $2 \times 2$ table of McNemar's Test for Paired Proportions, 1,496, 227, 173 and 2,886, has $p_o = 0.916$, $p_e = 0.542$ and $\kappa = 0.817$. Its prevalence index is $-0.291$, since more adults agree on "not high" than on "high", and its bias index is 0.0113. PABAK is 0.833.

Agreement and bias are different questions

Kappa and McNemar's test both summarize a paired table, and they answer different questions. McNemar's test asks whether the two margins differ, which depends only on $n_{12} - n_{21}$. Kappa asks how often the two classifications coincide beyond chance, which depends mostly on the diagonal. The bias index connects them: $n \cdot \text{BI} = n_{12} - n_{21}$, McNemar's numerator, 54 for the readings.

The readings show that the two can point in different directions. Agreement is high, $\kappa = 0.817$, and McNemar's test rejects equal margins at $p = 0.0069$. Both are correct. The readings agree on most adults, and among those on whom they disagree, the disagreement runs more often in one direction. By Theorem 3, the bias raises kappa slightly, although a reader would naturally count it as a defect of agreement. A report on two methods or two raters needs both numbers, or the table itself.

The same holds in the other direction. Two raters with identical margins have $\text{BI} = 0$ and McNemar's statistic 0, whatever their agreement; a kappa near zero is possible with perfectly matched margins. Neither statistic substitutes for the other.

The site's search is evaluated on informational queries whose results are graded for relevance on a four-point scale, from 0 (not relevant) to 3 (highly relevant), following a written judging guide (tools/search/eval/judging-guide.md). The pages to grade for each query were pooled from the top results of the rankers named in the pool file's ranks column. Most grades come from a model judge, and the model judges were calibrated against the site's owner on 150 query–page pairs: 25 queries with six pages each. The owner graded the 150 pairs in a randomized checklist with the ranks hidden. Two model judges (Claude Sonnet 5.5) graded the same pairs, one reading every page in full and one skimming each page's summary, headings and the sections bearing on the need. The evaluation records report exact agreement and quadratic-weighted kappa for each pair of judges. The grades are in tools/search/eval/qrels.tsv; this article uses only the grades, not the queries.

The owner (rows) against the full-read judge (columns):

Owner \ Full read 0 1 2 3 Total
0 27 4 0 0 31
1 5 20 7 0 32
2 1 13 21 2 37
3 0 0 11 39 50
Total 33 37 39 41 150

The script reproduces the recorded values and adds the others:

Judges Exact agreement Unweighted $\kappa$ Linear Quadratic SE (quadratic)
Owner vs full read 0.71 0.62 0.77 0.88 0.021
Owner vs skim 0.68 0.57 0.73 0.85 0.026
Full read vs skim 0.85 0.80 0.88 0.94 0.013

Over 30 bootstraps of 2,000 resamples of the 150 owner–full pairs, the standard deviation of quadratic kappa had median 0.021 (interquartile range 0.021 to 0.021 at three places), matching the delta-method value. The unweighted kappa of the same pair, 0.62, has a standard error of 0.049.

Three observations follow from the theory above.

  • The weights decide the number. The same 150 grades give 0.62, 0.77 or 0.88. Only one pair of grades differs by two steps, so nearly all disagreements are adjacent, and quadratic weights forgive most of them.
  • Quadratic kappa is a concordance correlation. For the owner and the full-read judge, Pearson's correlation of the grades is 0.884 and quadratic kappa 0.879. The gap is the difference of the mean grades, which Problem 4 works out.
  • The judges differ in level, and kappa does not show it. In 30 of the 43 disagreements the owner's grade was the higher one, and in 13 the full-read judge's. If neither judge tended to grade higher, each disagreement would go either way with probability one half, and the sign test of The Sign Test and the Wilcoxon Signed-Rank Test gives a two-sided p-value of 0.0137. The column totals show it too: the owner gave 50 threes and the full-read judge 41. This is the evaluation records' finding that "the model judges are the stricter", now with a test.

These numbers describe agreement on these 150 pairs. They say nothing about how any ranker performs, and they do not measure how often a grade is "right": kappa treats both judges alike, and neither as the standard.

Other coefficients

Kappa has many relatives, and two are worth naming here because of the paradoxes. Gwet's AC1 replaces the chance agreement by one computed from the average of the two raters' category shares, in the form that Shankar and Bangdiwala (2014) give, so that it is less affected by imbalanced margins (Gwet, 2008). Fleiss (1971) extended kappa to agreement among many raters. Landis and Koch (1977) developed kappa-type statistics within a general methodology for observer agreement; the descriptive labels they proposed for ranges of kappa, and the different labels used in information retrieval, are discussed in Relevance Judgments.

What is proved, what is cited, what the experiment shows

Proved here. The maximum kappa for fixed margins, attained for two categories (Theorem 1); the disagreement form of weighted kappa; quadratic weighted kappa as the concordance correlation coefficient (Theorem 2); the delta-method variance of weighted kappa, from the multinomial covariance and the delta method of the linked pages; the center-cell property of quadratic kappa; the lower bound of kappa for margins that allow an empty diagonal; kappa in terms of prevalence and bias indices, and the monotonicity behind the two paradoxes (Theorem 3 and its corollary); the relation of the bias index to McNemar's numerator.

Cited. The multinomial covariance (The Chi-Square Goodness-of-Fit Test) and the delta method for several variables (The Delta Method); the paradoxes as stated by Feinstein and Cicchetti (1990) through Shankar and Bangdiwala (2014), and their scenarios; the definitions of the prevalence and bias indices and of PABAK (Byrt, Bishop and Carlin, 1993, as given by Shankar and Bangdiwala, 2014); Warrens's (2012) statement of the center-cell result (proved here from Theorem 2 and checked on one example); Fleiss, Cohen and Everitt (1969) on earlier standard errors; Fleiss and Cohen (1973) on the intraclass correlation.

Computed. All kappas, indices and standard errors are exact or closed-form computations on the tables shown. The bootstrap standard errors are seeded resampling experiments on these two data sets.

Worked problems

These problems were written for this article, on the tables above and on scenario 3 of Shankar and Bangdiwala (2014). The script tools/experiments/statistics/measures-of-agreement.py recomputes every number.

Problem 1 (ninety percent agreement and a negative kappa). Two raters classify 100 units: 90 both positive, 5 positive by the first only, 5 by the second only, none both negative. Compute kappa directly and from Theorem 3.

Solution. $\kappa = -0.0526$ by both routes.

Directly: $p_o = 0.90$, both margins are $(0.95, 0.05)$, so $p_e = 0.95^2 + 0.05^2 = 0.905$, and $\kappa = (0.90 - 0.905)/(1 - 0.905) = -0.005/0.095 = -0.0526$. From the indices: $\text{PI} = 0.90 - 0 = 0.90$, $\text{BI} = 0$ and $\text{PABAK} = 2 \times 0.90 - 1 = 0.80$, so

$$\kappa = \frac{0.80 - 0.81}{1 - 0.81} = \frac{-0.01}{0.19} = -0.0526.$$

The raters agree on 90% of units and still agree less often than two independent raters with the same margins would, 90.5%. Nothing is wrong with the arithmetic: with a category this dominant, chance alone produces high agreement, and these raters never agree on the rare category.

Problem 2 (the ceiling for the readings). For the readings classified at 130 mm Hg, find $\kappa_{\max}$ and the ratio $\kappa/\kappa_{\max}$.

Solution. $\kappa_{\max} = 0.975$, and $\kappa/\kappa_{\max} = 0.838$.

The margins are $p_{1+} = 1{,}723/4{,}782$ and $p_{+1} = 1{,}669/4{,}782$. By Theorem 1, $p_{\max} = \min(p_{1+}, p_{+1}) + \min(p_{2+}, p_{+2}) = (1{,}669 + 3{,}059)/4{,}782 = 0.9887$. With $p_e = 0.5422$,

$$\kappa_{\max} = \frac{0.9887 - 0.5422}{1 - 0.5422} = 0.975,$$

and $\kappa = (0.9164 - 0.5422)/(1 - 0.5422) = 0.8173$, a ratio of 0.838. The bias between the readings costs little: the ceiling is close to 1, and most of the gap from perfect agreement is disagreement about individuals.

Problem 3 (a standard error by hand). Compute the delta-method standard error of $\hat\kappa$ for the same $2 \times 2$ table.

Solution. $\operatorname{SE}(\hat\kappa) = 0.0087$.

Write $h_{ij} = (1 - P_e)g_{ij} = w_{ij} - (\bar w_{i\cdot} + \bar w_{\cdot j})(1 - \kappa)$, which for unweighted kappa is $h_{ij} = 1\{i = j\} - (p_{+i} + p_{j+})(1 - \kappa)$. The shares are $p_{1+} = 0.3603$ and $p_{+1} = 0.3490$, and $1 - \kappa = 0.1827$. Then

  • $h_{11} = 1 - (p_{+1} + p_{1+})(1 - \kappa) = 0.8704$, and $h_{22} = 1 - (p_{+2} + p_{2+})(1 - \kappa) = 0.7642$;
  • $h_{12} = -(p_{+1} + p_{2+})(1 - \kappa) = -0.1806$, and $h_{21} = -(p_{+2} + p_{1+})(1 - \kappa) = -0.1848$.

Weighting the squares by the cell shares, $\sum p_{ij}h_{ij}^2 = 0.5922$. The subtracted term is $\kappa - P_e(1 - \kappa) = 0.7182$, whose square is 0.5159, so the numerator is 0.0764 (computed from the unrounded values; the rounded ones give 0.0763), and

$$\operatorname{Var}(\hat\kappa) \approx \frac{0.0764}{4{,}782 \times (1 - 0.5422)^2}, \qquad \operatorname{SE} = 0.0087.$$

The 95% interval is $0.817 \pm 0.017$.

Problem 4 (kappa from means and variances). For the owner and the full-read judge, compute quadratic kappa from Theorem 2, and find what it would be if the two mean grades were equal.

Solution. $\kappa_w = 0.879$; with equal means it would be 0.884.

With grades 0 to 3 as scores, the owner's mean grade is 1.707 and the full-read judge's 1.587; the variances are 1.287 and 1.229 and the covariance 1.112. The squared difference of the means is $0.12^2 = 0.0144$. By Theorem 2,

$$\kappa_w = \frac{2 \times 1.112}{1.287 + 1.229 + 0.0144} = 0.879,$$

and without the shift term, $2 \times 1.112/(1.287 + 1.229) = 0.884$, which happens to equal Pearson's $r$ to three places here because the two variances are nearly equal. The difference in mean grade costs 0.005 of quadratic kappa, although the sign test detects the difference in level at $p = 0.0137$. Here quadratic kappa is governed by whether the judges order the pages alike; a shift in level of 0.12 grade costs little because it enters squared, beside variances of about 1.3. A larger shift would cost more: by Theorem 2 a shift of a whole grade with the same variances and covariance would give $2 \times 1.112/(1.287 + 1.229 + 1) = 0.633$.

Summary

Quantity Formula
Kappa $\kappa = (p_o - p_e)/(1 - p_e)$, $p_o = \sum p_{ii}$, $p_e = \sum p_{i+}p_{+i}$
Maximum for fixed margins $\kappa_{\max} = (\sum_i \min(p_{i+}, p_{+i}) - p_e)/(1 - p_e)$
Weighted kappa $\kappa_w = 1 - \sum v_{ij}p_{ij}/\sum v_{ij}p_{i+}p_{+j}$
Quadratic weights $\kappa_w = 2s_{xy}/(s_x^2 + s_y^2 + (\bar x - \bar y)^2)$
Variance $\big\{\sum \pi_{ij}[w_{ij} - (\bar w_{i\cdot} + \bar w_{\cdot j})(1 - \kappa_w)]^2 - [\kappa_w - P_e(1 - \kappa_w)]^2\big\}/[n(1 - P_e)^2]$
Two categories $\kappa = (\text{PABAK} - \text{PI}^2 + \text{BI}^2)/(1 - \text{PI}^2 + \text{BI}^2)$

What we have learned

Kappa measures agreement as the share of the possible improvement on chance that two classifications achieve, where chance means independence with the observed margins. Its ceiling is set by the margins, and with ordered categories its value depends on the weights, so a kappa needs its weights and, for two categories, its prevalence and bias indices beside it. Quadratic weights make kappa a concordance correlation, which explains both its stability and its blind spots. A delta-method standard error, confirmed by the bootstrap, turns an observed kappa into an interval. On the NHANES readings the four bands give 0.689 unweighted and 0.910 with quadratic weights; on the relevance grades behind this site's search, the owner and the full-read judge reach 0.88 quadratic-weighted kappa while the two judges differ in level more than chance allows: the owner's grade is the higher one in 30 of the 43 disagreements. Agreement and bias are separate questions: kappa answers the first and McNemar's test the second.

This article closes the unit Categorical Data Analysis. The quadratic case connected agreement to correlation, and correlation, with the least-squares line, opens the next unit, Linear Models, in Correlation: Pearson, Spearman and Kendall.

References