In the MT-bench study of Zheng et al. (2023), expert judges compared the answers of six chat models two at a time, on 80 questions, and recorded which answer was better or that the two were tied. Of the 3,355 votes, 780 were ties. Among the decisive votes, GPT-3.5 won 0.708 of its comparisons and Claude-v1 won 0.696. Does that make GPT-3.5 the stronger model in the judges' eyes? It does not. GPT-3.5 met the three weakest models in 0.67 of its comparisons and Claude-v1 in 0.59, so GPT-3.5 had the easier schedule. Once each comparison is credited according to the opponent, the order of the two reverses, and the difference that remains is a small fraction of its uncertainty.
A paired comparison model gives the probability that one item is preferred to another as a function of a rating attached to each. The Bradley–Terry model is the standard one: model $i$ beats model $j$ with probability $\pi_i/(\pi_i + \pi_j)$, where $\pi_i > 0$ is the strength of $i$. Hunter (2004) traces it to Zermelo (1929) and to Bradley and Terry (1952). Written with $\beta_i = \log \pi_i$, it is a logistic regression, and the ratings that chess players know as Elo ratings are the same $\beta_i$ multiplied by $400/\ln 10$.
First we see why raw win rates mislead, and write the model, deriving it from a latent quality with random noise. Then we take its likelihood apart: the score equations, the condition under which a best rating exists, and Zermelo's algorithm for finding it. Next come standard errors, what dependence between votes does to them, two treatments of ties, and a test of fit. Then we show that Elo's rule is a stochastic gradient step on the same likelihood, and measure what the order of the games does to it. Finally we change the judge to GPT-4, and derive pairwise games from a benchmark graded right or wrong. The article assumes Logistic Regression and Maximum Likelihood Estimation; the large-sample theory it uses is in Asymptotics of the Maximum Likelihood Estimator.
Why win rates mislead
The 2,575 decisive votes form a table of wins. Write $w_{ij}$ for the number of times model $i$ was preferred to model $j$, and $n_{ij} = w_{ij} + w_{ji}$ for the number of decisive comparisons between them. Model $i$'s total wins are $W_i = \sum_j w_{ij}$ and its total comparisons $n_i = \sum_j n_{ij}$. The votes were not spread evenly: GPT-3.5 took part in 1,081 decisive comparisons and Claude-v1 in 761, and the pairs met between 103 and 249 times.
| Model | Comparisons $n_i$ | Wins $W_i$ | Win share $W_i/n_i$ | Average pairwise win rate |
|---|---|---|---|---|
| GPT-4 | 807 | 643 | 0.797 | 0.796 |
| GPT-3.5 | 1,081 | 765 | 0.708 | 0.678 |
| Claude-v1 | 761 | 530 | 0.696 | 0.689 |
| Vicuna-13B | 812 | 384 | 0.473 | 0.466 |
| Alpaca-13B | 788 | 177 | 0.225 | 0.269 |
| LLaMA-13B | 901 | 76 | 0.084 | 0.103 |
The win share is a weighted average of head-to-head win rates, with weights set by the schedule:
$$\frac{W_i}{n_i} = \sum_{j \ne i} \frac{n_{ij}}{n_i}\cdot\frac{w_{ij}}{n_{ij}}.$$
GPT-3.5 played 726 of its 1,081 comparisons against the three 13B models, and Claude-v1 452 of its 761, so GPT-3.5's share leans more on easy opponents. The last column, the average of the five head-to-head rates with equal weights, is the "average win rate" of Zheng et al. It removes the schedule's weights and puts Claude-v1 ahead. But it is still a property of the pool: add a seventh model that both beat easily and both averages rise, by amounts that depend on how often each met it. And each head-to-head rate is an estimate from its own pair's votes only, although the other pairs carry information about it too. Claude-v1 and GPT-3.5 met in 164 decisive comparisons, of which GPT-3.5 won 85. What both lost to GPT-4, and how badly each beat the weaker models, also bears on which is stronger.
A model with one number per competitor uses all of that evidence at once, and its ratings do not depend on the schedule in the way win shares do.
The Bradley–Terry model
Definition. Models $1, \dots, k$ have real ratings $\beta_1, \dots, \beta_k$. In a comparison of $i$ with $j$, $i$ is preferred with probability
$$p_{ij} = P(i \text{ beats } j) = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}} = \sigma(\beta_i - \beta_j), \qquad \sigma(x) = \frac{1}{1 + e^{-x}},$$
and different comparisons are independent. With strengths $\pi_i = e^{\beta_i}$ this is the ratio form $\pi_i/(\pi_i + \pi_j)$ of Hunter's equation (1). The function $\sigma$ is the logistic function of Logistic Regression, and $p_{ji} = 1 - p_{ij} = \sigma(\beta_j - \beta_i)$.
Only differences of ratings enter, so adding the same constant to every $\beta_i$ changes nothing. The ratings are fixed by a convention: here they are centered so that $\sum_i \beta_i = 0$. The odds that $i$ beats $j$ are $p_{ij}/p_{ji} = e^{\beta_i - \beta_j}$, so the rating difference is a log-odds, and odds multiply along chains: the odds of $i$ over $l$ are the odds of $i$ over $j$ times those of $j$ over $l$. That is a real restriction on the data, and the section on fit tests it.
Where the logistic form comes from. Here is one way the model arises. Suppose the judge perceives each answer's quality as $U_i = \beta_i + \varepsilon_i$ and prefers the larger, where the noise terms are independent with the standard Gumbel distribution function $F(x) = \exp(-e^{-x})$. Then $P(U_i > U_j) = \sigma(\beta_i - \beta_j)$.
Proof. Let $x = \beta_i - \beta_j$. The event $U_i > U_j$ is $\varepsilon_j < \varepsilon_i + x$. Conditioning on $\varepsilon_i = g$, whose density is $f(g) = e^{-g}\exp(-e^{-g})$, the probability is $F(g + x) = \exp(-e^{-g}e^{-x})$. So
$$P(U_i > U_j) = \int_{-\infty}^{\infty} e^{-g}\exp\!\left(-e^{-g}\right)\exp\!\left(-e^{-g}e^{-x}\right)dg = \int_0^\infty \exp\!\left(-u\,(1 + e^{-x})\right)du = \frac{1}{1 + e^{-x}},$$
with the substitution $u = e^{-g}$, $du = -e^{-g}\,dg$, which maps $g$ from $-\infty$ to $\infty$ onto $u$ from $\infty$ to 0. $\blacksquare$
The derivation is a motivation, not an assumption the fit needs: the model is the formula for $p_{ij}$. Standard normal noise instead of Gumbel noise gives $\Phi((\beta_i - \beta_j)/\sqrt{2})$ in place of $\sigma(\beta_i - \beta_j)$, because the difference of two independent standard normal variables is normal with variance 2. The two curves have different scales, but once the scales are matched they nearly coincide: the script finds that $\Phi(x/1.702)$ is never more than 0.0095 away from $\sigma(x)$, for any $x$. Data would need to pin a win probability down to better than about one percentage point to tell the two models apart.
Stochastic transitivity. If $p_{ij} \ge \tfrac12$ and $p_{jl} \ge \tfrac12$, then $p_{il} \ge \max(p_{ij}, p_{jl})$. Proof. The two conditions say $\beta_i \ge \beta_j \ge \beta_l$. Then $\beta_i - \beta_l = (\beta_i - \beta_j) + (\beta_j - \beta_l)$ is at least each of the two differences, because both are nonnegative, and $\sigma$ is increasing. $\blacksquare$ So the model cannot describe a cycle in which $i$ usually beats $j$, $j$ usually beats $l$ and $l$ usually beats $i$.
The Elo scale. Elo ratings are $r_i = c\,\beta_i$ plus a constant, with $c = 400/\ln 10 = 173.72$, because then
$$p_{ij} = \sigma\!\left(\frac{r_i - r_j}{c}\right) = \frac{1}{1 + e^{-(r_i - r_j)\ln 10/400}} = \frac{1}{1 + 10^{-(r_i - r_j)/400}},$$
which is the expected score of Boubdir et al. (2023, equation 1) and the win fraction of Bai et al. (2022, equation 2.1). A 100-point advantage is a win probability of $1/(1 + 10^{-1/4}) = 0.64$, the figure quoted in the note on A General Language Assistant as a Laboratory for Alignment. Bai et al. also give a conversion in which a difference of Elo scores is about 174 times the corresponding difference of preference-model scores, which are on a log-odds scale; the factor is $c$ rounded.
A logistic regression without an intercept
Each decisive vote between $i$ and $j$ can be written as one row of a logistic regression. Code the outcome $y = 1$ if the first-listed model won, and the covariates as a vector $x$ with $+1$ in position $i$, $-1$ in position $j$ and 0 elsewhere. Then $x \cdot \beta = \beta_i - \beta_j$ and $P(y = 1) = \sigma(x \cdot \beta)$: a logistic regression with one column per model and no intercept. The columns sum to zero in every row, which is the regression's version of the constant that cannot be identified; dropping one column fixes that model's rating at 0, and is the reference convention used for standard errors below.
An intercept would mean an advantage for whichever answer is listed first. Zheng et al. (2023, Section 3.3) found such position bias in language model judges, and Hunter (2004, equation 5) gives the corresponding "home-field advantage" form of the model. The same regression with real-valued features in place of the $\pm 1$ indicators is the pair model of RankNet, which scores documents by a network of their features rather than by one free parameter each.
The likelihood and its score equations
Treat the decisive votes as independent comparisons (a later section measures what that costs). The log-likelihood is
$$\ell(\beta) = \sum_{i \ne j} w_{ij}\log\sigma(\beta_i - \beta_j).$$
The derivative of $\log\sigma(x)$ is $1 - \sigma(x)$, because $\sigma'(x) = \sigma(x)(1 - \sigma(x))$. The rating $\beta_i$ appears in the terms with $w_{ij}$, with derivative $w_{ij}(1 - p_{ij})$, and in the terms with $w_{ji}$, with derivative $-w_{ji}(1 - p_{ji}) = -w_{ji}\,p_{ij}$. Adding over $j$,
$$\frac{\partial \ell}{\partial \beta_i} = \sum_{j} w_{ij} - \sum_j (w_{ij} + w_{ji})\,p_{ij} = W_i - \sum_j n_{ij}\,p_{ij}.$$
Theorem 1 (score equations). At a maximum likelihood estimate $\hat\beta$, every model's observed wins equal its expected wins: $W_i = \sum_j n_{ij}\,\hat p_{ij}$ for every $i$, where $\hat p_{ij} = \sigma(\hat\beta_i - \hat\beta_j)$.
Proof. At an interior maximum of a differentiable function every partial derivative is zero, and the derivative was just computed. $\blacksquare$
The theorem says what the ratings do with the schedule. Expected wins are computed against the opponents a model actually met, so a model that met weak opponents must win more to earn the same rating.
The second derivatives are $\partial^2\ell/\partial\beta_i\,\partial\beta_l = n_{il}\,p_{il}(1 - p_{il})$ for $l \ne i$, and $-\sum_{j} n_{ij}\,p_{ij}(1 - p_{ij})$ for $l = i$. So the matrix of second derivatives is $-L(\beta)$, where $L$ has off-diagonal entries $-c_{il}$ and diagonal entries $\sum_j c_{ij}$, with weights $c_{ij} = n_{ij}\,p_{ij}(1 - p_{ij})$. A matrix of this form is the Laplacian of the graph whose edges join the compared pairs. For any vector $v$,
$$v^{\mathsf T}Lv = \sum_{i < j} c_{ij}\,(v_i - v_j)^2 \ge 0,$$
which can be checked by expanding the square: each pair contributes $c_{ij}v_i^2 + c_{ij}v_j^2$ to the diagonal part and $-2c_{ij}v_iv_j$ to the rest. The log-likelihood is therefore concave. The quadratic form is zero exactly when $v_i = v_j$ for every pair that was compared, which, along paths of compared pairs, means that $v$ is constant on each connected group of models. When the comparison graph is connected, the only flat direction is the constant vector, the direction the centering convention removes.
When does a best rating exist?
Concavity does not guarantee a maximum. A model that won every comparison it took part in can always be made more likely by raising its rating further, and its rating has no finite estimate. Ford (1957) gave the condition that rules out this and every similar failure; Hunter (2004) states it as his Assumption 1.
Theorem 2 (existence and uniqueness). The maximum likelihood estimate exists and is unique (with $\sum_i \beta_i = 0$) if and only if, for every partition of the models into two nonempty groups $S$ and $S^c$, some member of $S^c$ has beaten some member of $S$ at least once.
Applied to both $S$ and its complement, the condition says that each group has won at least once against the other. Hunter notes the graph form: draw an arrow from $i$ to $j$ whenever $i$ has beaten $j$; the condition holds exactly when every model can be reached from every other along arrows.
Proof that the condition is necessary. Suppose a group $S$ never lost to $S^c$. Raise every rating in $S$ by $t > 0$. Comparisons within a group are unchanged. Every comparison between the groups was won by a member of $S$, and its term $\log\sigma(\beta_i - \beta_j)$ with $i \in S$ increases with $t$. If there was at least one such comparison, $\ell$ strictly increases for all $t$, so no finite maximum exists. If there was none, $\ell$ is unchanged for all $t$, so a maximum, if one exists, is not unique even after centering (shifting $S$ up and $S^c$ down by amounts that keep the sum zero changes the ratings but not $\ell$).
Proof that the condition is sufficient. Every term of $\ell$ is the logarithm of a probability, so $\ell \le 0$, and $\ell$ is continuous. Take any centered $\beta$ and order its values; let $D$ be its range, the largest rating minus the smallest. The $k - 1$ gaps between consecutive values add up to $D$, so at least one gap is at least $D/(k - 1)$. Let $S$ be the models above that gap. By the condition, some model $j \notin S$ beat some model $i \in S$, and the term for that win is
$$\log\sigma(\beta_j - \beta_i) \le \log\sigma\!\left(-\frac{D}{k - 1}\right),$$
because $\beta_i - \beta_j$ is at least the gap. As $D \to \infty$ the right side tends to $-\infty$, and since every other term is at most 0, so does $\ell$. Now fix any centered $\beta^{(0)}$. The set of centered $\beta$ with $\ell(\beta) \ge \ell(\beta^{(0)})$ is closed, and by the bound it lies within a bounded range $D$, and so (being centered) within a bounded set. A continuous function on a closed bounded set attains its maximum, which gives existence. The condition also makes the comparison graph connected, so by the previous section $\ell$ is strictly concave along every centered direction, and a strictly concave function has at most one maximizer. $\blacksquare$
Example (the MT-bench votes satisfy the condition). Every pair of the six models has decisive votes in both directions; even LLaMA-13B beat GPT-4 12 times. Every arrow is present, so the estimate exists and is unique.
Example (a finite rating is not evidence that an estimate exists). Elo's rule, below, assigns a finite rating to a model that has never lost, after any finite number of games. That number reflects only how many games were played and the step size, not a finite strength; by Theorem 2 the likelihood has no maximum. Worked problem 3 shows the case.
Computing the estimate: Zermelo's iteration
The score equations have no closed-form solution beyond two models. Zermelo (1929) gave an iteration, which in the strengths $\pi_i = e^{\beta_i}$ reads
$$\pi_i^{(t+1)} = W_i \left(\sum_{j \ne i}\frac{n_{ij}}{\pi_i^{(t)} + \pi_j^{(t)}}\right)^{-1},$$
followed by rescaling (any common factor leaves the probabilities unchanged). Hunter (2004, equation 3) shows it is an instance of an MM (minorize–maximize) algorithm, after Lange, Hunter and Yang (2000). The idea: at the current point, replace the log-likelihood by a simpler function that lies below it everywhere and touches it there; maximizing the simpler function cannot lower the log-likelihood.
Theorem 3 (Zermelo's iteration never lowers the likelihood). Suppose every model has won at least once (this holds under the condition of Theorem 2: take $S$ to be all models but one). Then $\ell(\pi^{(t+1)}) \ge \ell(\pi^{(t)})$.
Proof. In the strengths, $\ell(\pi) = \sum_{i \ne j} w_{ij}\left[\log\pi_i - \log(\pi_i + \pi_j)\right]$. Since the logarithm is concave, it lies below its tangent line: $\log x \le \log y + (x - y)/y$ for positive $x, y$, with equality at $x = y$. Rearranged, $-\log x \ge 1 - \log y - x/y$ (Hunter's inequality 9). Apply it with $x = \pi_i + \pi_j$ and $y = \pi_i^{(t)} + \pi_j^{(t)}$ to each term:
$$\ell(\pi) \ge Q(\pi) = \sum_{i \ne j} w_{ij}\left[\log\pi_i + 1 - \log\left(\pi_i^{(t)} + \pi_j^{(t)}\right) - \frac{\pi_i + \pi_j}{\pi_i^{(t)} + \pi_j^{(t)}}\right],$$
with equality at $\pi = \pi^{(t)}$. Collect the terms of $Q$ that contain $\pi_i$: $W_i\log\pi_i$ from the first part, and $-\pi_i\sum_j n_{ij}/(\pi_i^{(t)} + \pi_j^{(t)})$ from the fraction, since $\pi_i$ appears in the fraction of both $w_{ij}$ and $w_{ji}$. So $Q$ is a sum of separate functions of each $\pi_i$, each of the form $W_i\log\pi_i - a_i\pi_i$, which, since $W_i > 0$, is maximized at $\pi_i = W_i/a_i$: that is the update. Therefore $\ell(\pi^{(t+1)}) \ge Q(\pi^{(t+1)}) \ge Q(\pi^{(t)}) = \ell(\pi^{(t)})$. $\blacksquare$
Hunter (2004, Section 4) proves that under the condition of Theorem 2 the iteration converges to the unique estimate from any starting point; that convergence result is cited, not proved here. Newton's method on the logistic regression, as in Logistic Regression, reaches the same point in fewer, more expensive steps; the script runs both and checks that they agree to $10^{-9}$.
Example (Zermelo's iteration on MT-bench). Starting from equal strengths, every decisive vote has probability one half, and the log-likelihood is $2{,}575\log\tfrac12 = -1{,}784.9$. One step raises it to $-1{,}193.6$, and after 45 steps no rating moves by more than $10^{-6}$; the maximum is $-1{,}118.8$.
The ratings
| Model | $\hat\beta_i$ | Elo scale $c\,\hat\beta_i$ | Wins $W_i$ | Expected wins $\sum_j n_{ij}\hat p_{ij}$ |
|---|---|---|---|---|
| GPT-4 | 1.43 | 249 | 643 | 643.0 |
| Claude-v1 | 0.90 | 156 | 530 | 530.0 |
| GPT-3.5 | 0.86 | 150 | 765 | 765.0 |
| Vicuna-13B | −0.08 | −13 | 384 | 384.0 |
| Alpaca-13B | −1.06 | −183 | 177 | 177.0 |
| LLaMA-13B | −2.06 | −358 | 76 | 76.0 |
The last two columns are Theorem 1 on the data. Claude-v1 now sits above GPT-3.5, by 0.04 on the log-odds scale: the model gives Claude-v1 a probability of 0.509 of being preferred to GPT-3.5, although in their own 164 meetings Claude-v1 won only 0.482. The fit uses the indirect evidence: Claude-v1 did better against GPT-4 and the other models, relative to what their ratings predict, than GPT-3.5 did. The model also says GPT-4 is preferred to Claude-v1 with probability 0.630 and to LLaMA-13B with probability 0.970.
Whether 0.04 means anything is the next question.
How sure are we?
The information matrix. For a logistic regression the expected and observed information coincide, and here both are the Laplacian $L(\hat\beta)$ of the previous sections. It is singular, because of the unidentified constant. Fixing one model's rating at 0 (the reference) and deleting its row and column leaves an invertible matrix whose inverse estimates the covariance of the other ratings. The asymptotic normality of a maximum likelihood estimate with several parameters, which justifies this, is stated and cited in Asymptotics of the Maximum Likelihood Estimator. A difference $\hat\beta_i - \hat\beta_j$ does not depend on which model is the reference, and neither does its estimated variance; the script checks this for every choice of reference, and also against the pseudo-inverse of the full matrix.
For neighbors in the ranking:
| Difference | Estimate | Standard error | $z$ | 95% interval |
|---|---|---|---|---|
| GPT-4 − Claude-v1 | 0.53 | 0.115 | 4.63 | 0.31 to 0.76 |
| Claude-v1 − GPT-3.5 | 0.04 | 0.105 | 0.35 | −0.17 to 0.24 |
| GPT-3.5 − Vicuna-13B | 0.94 | 0.100 | 9.35 | 0.74 to 1.14 |
| Vicuna-13B − Alpaca-13B | 0.98 | 0.117 | 8.40 | 0.75 to 1.21 |
| Alpaca-13B − LLaMA-13B | 1.00 | 0.143 | 7.04 | 0.72 to 1.28 |
The order of Claude-v1 and GPT-3.5 is not determined by these votes ($p = 0.73$). Every other neighbor is separated by more than four standard errors.
Example (the intervals cover under the model). The script simulated the experiment 1,000 times for each of 30 seeds: the same schedule of decisive comparisons, outcomes drawn independently from the fitted probabilities, the model refitted each time. The 95% interval for Claude-v1 − GPT-3.5 covered the true difference in a median of 0.950 of the simulations (interquartile range over the seeds 0.947 to 0.953), and the interval for GPT-4 − Claude-v1 in 0.948 (0.945 to 0.954). With comparisons that really are independent, the standard errors are right for these two differences at this sample size.
The votes are not independent. They come in clusters. Each of the 80 questions has two turns, the same pair of answers to a turn was judged by up to seven judges, and a judge typically cast dozens of votes: the identifiers in the data are 58 experts with 2,668 votes and 7 of the authors with 687. Votes on the same pair of answers are related. Zheng et al. (2023, Section 4.2 and Table 5) report an agreement of 81% between two different human judges on the same first-turn question, counting non-tied votes only (82% on second turns). Agreement alone does not show dependence, since independent votes on a lopsided pair agree often too; what shows it is the excess over the model. In the data there are 1,550 pairs of decisive votes by different judges on the same question, turn and pair of answers. They agree in 82.7% of cases, against 71.8% expected if each vote were an independent Bradley–Terry outcome at the fitted ratings. Several votes on one pair of answers therefore carry less information than as many independent comparisons would. The design links votes in a second way. Zheng et al. (2023, Section 4.2 and Appendix C.1) report that when a judge's vote differed from GPT-4's, the judge was shown GPT-4's judgment and asked whether it was reasonable, and that judges were willing to change their choices in 34% of such cases.
The questions are the units sampled from a population, as in the framing of Adding Error Bars to Evals, so the natural repair is to resample questions. A cluster bootstrap draws 80 questions with replacement, keeps every vote on each drawn question, and refits. Clustered Standard Errors derives the variance this targets, and The Bootstrap for Dependent Data the cluster bootstrap itself. With 1,000 resamples for each of 30 seeds, the median bootstrap standard errors are 1.38 to 1.80 times the model's:
| Difference | Model standard error | Cluster bootstrap standard error | Ratio |
|---|---|---|---|
| GPT-4 − Claude-v1 | 0.115 | 0.195 | 1.69 |
| Claude-v1 − GPT-3.5 | 0.105 | 0.188 | 1.80 |
| GPT-3.5 − Vicuna-13B | 0.100 | 0.148 | 1.48 |
| Vicuna-13B − Alpaca-13B | 0.117 | 0.188 | 1.62 |
| Alpaca-13B − LLaMA-13B | 0.143 | 0.197 | 1.38 |
The 30 seeds agree closely: for Claude-v1 − GPT-3.5 the interquartile range of the bootstrap standard error is 0.187 to 0.190. The conclusions survive, since GPT-4 − Claude-v1 is still $0.53/0.195 \approx 2.7$ standard errors from zero, but the intervals are nearly twice as wide as the model claims. For the Chatbot Arena ratings, Chiang et al. (2024, Section 5) report intervals from robust "sandwich" variances, which remain valid when the model's assumptions fail, and from a bootstrap; Bootstrap Confidence Intervals discusses which bootstrap interval to report.
Ties
Of the 3,355 votes, 780 (23%) were ties, and the fit above dropped them. There are three common choices.
Dropping ties. Under the model of ties below, dropping them is justified: the decisive votes then follow the Bradley–Terry model exactly, and the fit above is the maximum of a conditional likelihood.
Counting a tie as half a win to each side. This adds $t_{ij}/2$ to both $w_{ij}$ and $w_{ji}$, where $t_{ij}$ is the number of ties between $i$ and $j$, and fits as before. It is not the likelihood of any model in which ties occur: it multiplies $p_{ij}$ for each win, $1 - p_{ij}$ for each loss and $\sqrt{p_{ij}(1 - p_{ij})}$ for each tie, and these three numbers add up to more than 1 whenever $0 < p_{ij} < 1$ (because $\sqrt{p(1-p)} > 0$), so they cannot be the probabilities of the three outcomes. Nor can any constant multiple of them, because a likelihood is fixed only up to a factor that does not depend on the ratings, and the factor that would turn them into probabilities is $1/(1 + \sqrt{p_{ij}(1 - p_{ij})})$, which changes with $p_{ij}$. Rescaled by that factor they are Davidson's model below with $\nu = 1$, so half wins maximize Davidson's likelihood at $\nu = 1$ multiplied by $\prod_{i<j}\big(1 + \sqrt{p_{ij}(1 - p_{ij})}\big)^{n_{ij}}$, where $n_{ij}$ counts all comparisons of $i$ and $j$, ties included; that extra factor depends on the ratings. It also pulls every head-to-head share toward one half, so the ratings shrink: GPT-4 falls from 1.43 to 1.02 and LLaMA-13B rises from −2.06 to −1.45. The order of the six models is unchanged on these data.
Davidson's model. Davidson (1970), as Hunter (2004, equation 7) states it, makes the three outcomes proportional to $\pi_i$, $\pi_j$ and $\nu\sqrt{\pi_i\pi_j}$:
$$P(i \text{ beats } j) = \frac{\pi_i}{\pi_i + \pi_j + \nu\sqrt{\pi_i\pi_j}}, \qquad P(\text{tie}) = \frac{\nu\sqrt{\pi_i\pi_j}}{\pi_i + \pi_j + \nu\sqrt{\pi_i\pi_j}},$$
with one extra parameter $\nu > 0$. Ties are most likely between models of equal strength, where the tie probability is $\nu/(2 + \nu)$, and rarer as the strengths separate.
Lemma (decisive outcomes under Davidson's model). Given that a comparison was not a tie, $i$ wins with probability $\pi_i/(\pi_i + \pi_j)$.
Proof. The conditional probability is $P(i \text{ beats } j)$ divided by $P(i \text{ beats } j) + P(j \text{ beats } i)$; the common denominator $\pi_i + \pi_j + \nu\sqrt{\pi_i\pi_j}$ cancels and leaves $\pi_i/(\pi_i + \pi_j)$. $\blacksquare$
So the Bradley–Terry fit to the decisive votes estimates Davidson's strengths, ignoring the information that the ties carry about how far apart the models are. Fitted by maximum likelihood to all 3,355 votes, Davidson's model gives ratings 1.50, 0.95, 0.88, −0.08, −1.11 and −2.14 (in the order of the table above), close to the 1.43 to −2.06 of the decisive votes alone, with $\nu = 0.86$. The fitted tie probability for two equal models is 0.30. For Claude-v1 against GPT-3.5 it predicts 0.30 against an observed 0.34, and for GPT-4 against LLaMA-13B 0.12 against an observed 0.11. Rao and Kupper (1967) proposed another model, with a threshold below which a judge declares a tie (Hunter's equation 6); the script fits only Davidson's.
Does the model fit?
With six models there are 15 pairs. A saturated model gives every pair its own win probability, estimated by $w_{ij}/n_{ij}$; the Bradley–Terry model fits the same 15 probabilities with 5 free ratings. The likelihood ratio statistic between them, the deviance,
$$G^2 = 2\sum_{i \ne j} w_{ij}\log\frac{w_{ij}}{n_{ij}\,\hat p_{ij}},$$
has, if the model holds and votes are independent, approximately a chi-squared distribution with $15 - 5 = 10$ degrees of freedom; Likelihood Ratio, Wald and Score Tests proves the one-parameter case of that theorem and cites the general one. On MT-bench $G^2 = 22.8$ ($p = 0.01$), and Pearson's version, the sum over pairs of squared standardized differences between $w_{ij}$ and $n_{ij}\hat p_{ij}$, is $X^2 = 24.1$ ($p = 0.01$). The largest standardized residual, 2.77 in absolute value, belongs to GPT-4 against LLaMA-13B: GPT-4 won 0.936 of those comparisons, against 0.970 fitted. LLaMA-13B beat GPT-4 more often than its overall rating allows.
Example (the misfit is weaker than the test says). Both statistics assume independent votes. The cluster bootstrap showed variances of rating differences inflated by factors of about $1.38^2 = 1.9$ to $1.80^2 = 3.2$. If the residuals are inflated similarly, which this article does not prove, $G^2$ divided by 2 is about 11 on 10 degrees of freedom, no evidence of misfit. The honest summary is that these votes do not clearly reject the model.
Transitivity can be checked directly. Among the raw head-to-head shares there are 0 triples in which $i$ beats $j$ and $j$ beats $l$ more often than not while $l$ beats $i$ more often than not, so the votes contain no cycle of the kind the model forbids.
Elo: the same model, one game at a time
Elo's system updates ratings after each game instead of fitting them all at once. In the notation of Boubdir et al. (2023, equations 1 and 2), after a game between $i$ and $j$ with outcome $S = 1$ if $i$ won and 0 if it lost,
$$r_i \leftarrow r_i + K\,(S - E_i), \qquad r_j \leftarrow r_j - K\,(S - E_i), \qquad E_i = \frac{1}{1 + 10^{(r_j - r_i)/400}},$$
where $K$, the K-factor, sets the step. The total of the ratings never changes.
Theorem 4 (Elo's rule is a stochastic gradient step). For one game, let $\ell_g(r) = S\log E_i + (1 - S)\log(1 - E_i)$ be its log-likelihood under the Bradley–Terry model on the Elo scale. Then $\partial\ell_g/\partial r_i = (S - E_i)/c$ and $\partial\ell_g/\partial r_j = -(S - E_i)/c$, so the Elo update is $r \leftarrow r + Kc\,\nabla\ell_g(r)$: one step of gradient ascent on that game's log-likelihood, with step size $Kc$.
Proof. $E_i = \sigma((r_i - r_j)/c)$, as shown in the section on the Elo scale. With $x = (r_i - r_j)/c$, $\ell_g = S\log\sigma(x) + (1 - S)\log(1 - \sigma(x))$ has derivative in $x$ equal to $S(1 - \sigma(x)) - (1 - S)\sigma(x) = S - \sigma(x)$, and $\partial x/\partial r_i = 1/c$, $\partial x/\partial r_j = -1/c$. $\blacksquare$
Corollary. Average the Elo update over all games in a fixed set of data, each chosen with equal probability. At the maximum likelihood ratings, $r = c\hat\beta$, the average update of every model is zero.
Proof. Summed over the games, model $i$'s updates are $K$ times its wins minus $K$ times its expected wins, $K\big(W_i - \sum_j n_{ij}\hat p_{ij}\big)$, which is zero by Theorem 1. $\blacksquare$
A zero average step for every model is exactly the score equations of Theorem 1, whose solutions are the maximum likelihood ratings, unique up to a common shift by Theorem 2 (concavity makes every solution a maximum). Elo's fixed total picks one of them: with every model started at 1,000, the only ratings with zero average step are $1{,}000 + c\hat\beta$, the batch estimate shifted to that total, and that is the point the ratings are drawn toward. They do not settle there, and the corollary does not say that their average position equals it: with a constant step, the ratings fluctuate around that point. With a constant $K$ every game moves the ratings by an amount that does not shrink, so the final ratings depend on the order in which the games arrive, and the last games count most. In chess that is intended, because players' strengths change; Boubdir et al. (2023) note that the system was designed for players in dynamic games and ask whether it suits models whose skill is fixed. A frozen model's answers do not improve between votes, and the order of the votes is an accident of data collection.
Example (30 orders of the same votes). The script ran Elo through the 2,575 decisive votes in 30 random orders, starting every model at 1,000, for three K-factors, and compared the centered final ratings with the batch estimate on the Elo scale (GPT-4 249, LLaMA-13B −358):
| K-factor | GPT-4: median over orders | Width of its interquartile range | Largest deviation from the batch estimate (median) | Orders with GPT-3.5 above Claude-v1 |
|---|---|---|---|---|
| 4 | 228 | 8 | 37 | 14 of 30 |
| 16 | 250 | 26 | 44 | 10 of 30 |
| 32 | 256 | 33 | 70 | 10 of 30 |
The two failures pull in opposite directions. A small K-factor has little noise, but after 2,575 votes the ratings have not yet travelled from their common start: GPT-4 at 228 and LLaMA-13B at −321 are both short of the batch estimate. A large K-factor gets there, but each order leaves the ratings somewhere else, by tens of points. The order of Claude-v1 and GPT-3.5 flips in a third to a half of the orders, which the standard errors above say is all these votes can support. Averaging the final ratings over the 30 orders reduces the noise (for K = 16 the averages are 254 for GPT-4 and −358 for LLaMA-13B); Boubdir et al. (2023, Section 7) recommend averaging over many permutations of the games, ideally 100 or more. Chiang et al. (2024, Section 4) report that Chatbot Arena moved from Elo scores to Bradley–Terry coefficients "because the BT coefficients are better for the purpose of statistical estimation". For a fixed set of votes about fixed models, the batch fit is the estimate Elo approximates.
Who judges matters
The same 80 questions and answers were also judged by GPT-4 itself, which compared each pair of answers in both orders. The dataset holds 2,400 of its votes, of which 600 are ties, 380 of them labeled inconsistent: GPT-4 preferred a different answer when the positions were swapped, and Zheng et al. count such a pair as a tie. Fitting the decisive votes gives ratings 3.01, 1.90, 0.94, −0.12, −2.02 and −3.70.
The order is the human judges' order. The scale is not. The distance from GPT-4 to LLaMA-13B is 6.71 under GPT-4's judgment and 3.49 under the experts', a factor of 1.92, and the gap between Claude-v1 and GPT-3.5 is 0.97 instead of 0.04. A rating is a statement about a population of models, questions and judges together. A more decisive judge spreads the same models further apart, and two leaderboards built with different judges are on different scales even when they agree on the order. Zheng et al. (2023) also examine whether a model judge favors its own answers, one of the biases listed in their abstract, which is a separate reason not to read GPT-4's rating of GPT-4 as the experts'.
Ratings from a benchmark graded right or wrong
A benchmark scored by correctness can be turned into pairwise games: on each question, model $i$ beats model $j$ if $i$ answered correctly and $j$ did not, and the question is a tie otherwise. The unit's dataset, described in Standard Errors for Benchmark Scores, is enough to try it: the answers of 28 models to the 14,042 test questions of MMLU (Hendrycks et al., 2021), as published by the HELM project (Liang et al., 2022).
Theorem 5 (net wins rank as accuracy). If every model answers every question, the net wins of model $i$ in derived games are $W_i - L_i = k\,C_i - \sum_l C_l$, where $C_i$ is its number of correct answers and $L_i$ its losses. So net wins put the models in exactly the order of their accuracy.
Proof. On one question, let $s$ models be correct. A correct model wins against the $k - s$ incorrect ones and loses to none; an incorrect model loses to the $s$ correct ones. So model $i$'s net wins on the question are $k - s$ if it is correct and $-s$ if it is not, which is $k\cdot[\text{$i$ correct}] - s$ in both cases. Summing over questions gives $k\,C_i - \sum_l C_l$. $\blacksquare$
The Bradley–Terry ratings fitted to the same games do not follow accuracy exactly. Across the 28 models they swap three adjacent pairs. Claude Instant 1.2 (HELM's claude-instant-1.2) answered 9,599 questions correctly (0.6836) and GPT-3.5 Turbo (gpt-3.5-turbo-0613) 9,582 (0.6824), but the fitted ratings put GPT-3.5 Turbo ahead by 0.011. The fit weighs each pair's games by how many questions the pair disagrees on, and these vary from pair to pair, so the expected wins of Theorem 1 are not a fixed function of accuracy.
With only these two models the fit has a closed form. Write $b$ for the questions Claude Instant got right and GPT-3.5 Turbo wrong, and $c$ for the reverse: $b = 1{,}474$ and $c = 1{,}457$. The likelihood is that of a binomial count $b$ out of $b + c$, so $\hat\beta_1 - \hat\beta_2 = \log(b/c) = 0.012$, with standard error $\sqrt{1/b + 1/c} = 0.037$ (worked problem 4) and $z = 0.31$. That is McNemar's comparison of two paired proportions, which uses only the discordant questions; McNemar's own statistic $(b - c)/\sqrt{b + c}$ is also 0.31. McNemar's Test for Paired Proportions proves why only the discordant pairs carry information, and Paired Comparisons of Models applies it to models.
With all 28 models in one fit, the model-based standard error of the same difference is 0.010, about a quarter (0.27) of the two-model value, as though the other 26 models added independent evidence. They do not: the derived games share questions, and a question that 14 of the 28 models answer correctly produces games for $14 \times 14 = 196$ of the 378 pairs at once. A cluster bootstrap over questions (200 resamples for each of 30 seeds) gives 0.033, 3.4 times larger. Turning graded answers into pairwise games therefore adds nothing that a paired comparison of accuracies lacks, and its model-based standard errors are wrong. Pairwise models earn their place where the data are pairwise, as with human or model judges.
What is proved, what is cited, what the experiments show
Proved here. The logistic form from Gumbel noise; strong stochastic transitivity of the model; the conversion to the Elo scale; the score equations and the Laplacian form of the information; the necessity and sufficiency of Ford's condition for a unique maximum; the monotone increase of the likelihood under Zermelo's iteration; the Bradley–Terry form of decisive outcomes under Davidson's model of ties; Elo's update as a stochastic gradient step and the zero average update at the maximum likelihood ratings; the identity between net wins and accuracy for derived games.
Cited. The model's history (Zermelo, 1929; Bradley and Terry, 1952) and Ford's condition as stated by Hunter (2004); the convergence of Zermelo's iteration from any start (Hunter, 2004); Davidson's (1970) and Rao and Kupper's (1967) models as stated by Hunter; the asymptotic normality of a several-parameter maximum likelihood estimate and the chi-squared limit of the deviance, through the linked articles; the description of the MT-bench data, its judges and GPT-4's role (Zheng et al., 2023); the Elo formulas (Boubdir et al., 2023; Bai et al., 2022); Chatbot Arena's choice of Bradley–Terry coefficients (Chiang et al., 2024).
What the experiments show. The coverage of the Wald interval under independent simulated votes, the cluster bootstrap standard errors, the behavior of Elo over 30 orders, and the comparison of judges are observations on these data. They show what can happen with votes clustered by question and with constant K-factors; they are not general rates.
Worked problems
These problems were written for this article: problems 1 to 3 are small exact cases, and problem 4 uses the MMLU answers. The script tools/experiments/statistics/rating-models-from-pairwise-comparisons.py recomputes every number.
Problem 1 (one step of Zermelo's iteration). Three models A, B and C were compared in pairs. A beat B 3 times and lost once; A beat C twice and lost once; B beat C 3 times and lost once. Starting from equal strengths, carry out one step of Zermelo's iteration, check that the log-likelihood rose, and check the score equations at the final estimate.
Solution. After one step the strengths are proportional to 1.429, 1.000 and 0.571, and the log-likelihood rises from −7.625 to −6.772. The wins are $W_A = 3 + 2 = 5$, $W_B = 1 + 3 = 4$ and $W_C = 1 + 1 = 2$. The numbers of decisive comparisons are $n_{AB} = 4$, $n_{AC} = 3$ and $n_{BC} = 4$. With all strengths 1, each denominator term is $n_{ij}/2$, so
$$\pi_A = \frac{5}{(4 + 3)/2} = \frac{10}{7} = 1.429, \qquad \pi_B = \frac{4}{(4 + 4)/2} = 1.000, \qquad \pi_C = \frac{2}{(3 + 4)/2} = \frac{4}{7} = 0.571.$$
At the start every one of the 11 games has probability one half, so $\ell = 11\log\tfrac12 = -7.625$. After the step, $p_{AB} = 1.429/2.429 = 0.588$, $p_{AC} = 1.429/2.000 = 0.714$ and $p_{BC} = 1.000/1.571 = 0.636$, and
$$\ell = 3\log 0.588 + \log 0.412 + 2\log 0.714 + \log 0.286 + 3\log 0.636 + \log 0.364 = -6.772.$$
Iterating to convergence gives centered ratings 0.658, 0.000 and −0.658. The symmetry can be seen without computing. Reverse every outcome and exchange the names A and C: "A beat B 3 times" becomes "B beat C 3 times", "A beat C twice" becomes "A beat C twice" again, and so on, so the data are unchanged. The same operation maps ratings $(\beta_A, \beta_B, \beta_C)$ to $(-\beta_C, -\beta_B, -\beta_A)$, which therefore have the same likelihood. Since the estimate is unique (Theorem 2 applies: every split has wins both ways), it equals its own image, so $\beta_B = 0$ and $\beta_A = -\beta_C$. At these ratings $p_{AB} = p_{BC} = \sigma(0.658)$ and $p_{AC} = \sigma(1.316) = 0.788$, and the expected wins are 5.000, 4.000 and 2.000, equal to the observed wins as Theorem 1 requires.
Problem 2 (an Elo update). A model rated 1,500 beats a model rated 1,600 with K = 32. Find the new ratings, and the change had the favorite won.
Solution. The underdog's expected score is $E = 1/(1 + 10^{100/400}) = 0.3599$, so it gains $32(1 - 0.3599) = 20.48$ points: the new ratings are 1,520.48 and 1,579.52. Had the favorite won, it would have gained only $32 \times 0.3599 = 11.52$, since a win it was expected to deliver carries less information. The favorite's expected score is $1 - 0.3599 = 0.6401$, the 64% of a 100-point gap. By Theorem 4 this update is a gradient step of size $Kc = 32 \times 173.72$ on the game's log-likelihood.
Problem 3 (a model that never loses). Model A won all 5 of its comparisons with B and both of its comparisons with C; B and C split 3 to 3. Show that the maximum likelihood estimate does not exist, and say what Zermelo's iteration and Elo do.
Solution. Take $S = \{A\}$: no member of $\{B, C\}$ ever beat A, so Ford's condition fails, and by the proof of Theorem 2 raising $\beta_A$ always raises the likelihood. Concretely, the terms involving A are $5\log\sigma(\beta_A - \beta_B) + 2\log\sigma(\beta_A - \beta_C)$, which increase toward 0 as $\beta_A \to \infty$ without reaching it. Zermelo's iteration gives $\pi_A^{(t+1)} = 7/\big(5/(\pi_A + \pi_B) + 2/(\pi_A + \pi_C)\big)$, evaluated at the current strengths, and this exceeds $\pi_A^{(t)}$ because the denominator is less than $5/\pi_A + 2/\pi_A = 7/\pi_A$. That growth illustrates the failure but does not prove it, since the other strengths change too and the strengths are rescaled. The proof is that the iteration cannot converge to finite positive strengths. The update followed by rescaling is continuous at positive strengths, so such a limit would be a fixed point, and a fixed point solves the score equations (shown next). A solution of the score equations is a stationary point of the concave log-likelihood and hence a maximum, and by Theorem 2 there is none. That a fixed point solves the score equations: if $\pi_i = \lambda W_i/a_i$ for every $i$, with $a_i = \sum_j n_{ij}/(\pi_i + \pi_j)$, then multiplying by $a_i$ and adding over $i$ gives $\sum_i \pi_i a_i = \sum_{i<j} n_{ij} = \lambda\sum_i W_i$, and both sums count every game once, so $\lambda = 1$, and then $\pi_i a_i = W_i$ is the score equation for model $i$. Elo, after any finite sequence of these games, gives A a finite rating that depends on K and the order. The difference between them is not a defect of Elo's arithmetic: the data contain no information about how much stronger A is, only that it is stronger, and only the likelihood says so.
Problem 4 (two models compared through discordant questions). On MMLU, Claude Instant 1.2 was right and GPT-3.5 Turbo wrong on 1,474 questions, and the reverse on 1,457. Fit the two-model Bradley–Terry model and find the standard error of the rating difference from the information.
Solution. The difference is 0.012 with standard error 0.037. With two models, the decisive games are $b + c = 2{,}931$ independent trials if questions are independent, and $p_{12} = \sigma(\beta_1 - \beta_2)$ is their success probability. Its maximum likelihood estimate is $\hat p = b/(b + c)$, so $\hat\beta_1 - \hat\beta_2 = \log(\hat p/(1 - \hat p)) = \log(b/c) = \log(1{,}474/1{,}457) = 0.012$. The information about $\beta_1 - \beta_2$ is the single Laplacian weight $n\,p(1 - p)$ with $n = b + c$. At $\hat p$ this is $(b + c)\cdot\frac{b}{b + c}\cdot\frac{c}{b + c} = \frac{bc}{b + c}$, so the variance is $\frac{b + c}{bc} = \frac1b + \frac1c = 0.001365$, and the standard error is $\sqrt{0.001365} = 0.037$. The Wald statistic is $0.012/0.037 = 0.31$, the same to two decimals as McNemar's $(b - c)/\sqrt{b + c} = 17/\sqrt{2{,}931}$.
Summary
| Quantity | Formula |
|---|---|
| Bradley–Terry model | $P(i \text{ beats } j) = \sigma(\beta_i - \beta_j) = \pi_i/(\pi_i + \pi_j)$ |
| Log-likelihood | $\ell(\beta) = \sum_{i \ne j} w_{ij}\log\sigma(\beta_i - \beta_j)$ |
| Score equations | $W_i = \sum_j n_{ij}\,\hat p_{ij}$ for every $i$ |
| Information | Laplacian with weights $n_{ij}\,p_{ij}(1 - p_{ij})$ |
| Existence | every split of the models has wins in both directions |
| Zermelo's iteration | $\pi_i \leftarrow W_i\big/\sum_j n_{ij}/(\pi_i + \pi_j)$ |
| Elo scale | $r = (400/\ln 10)\,\beta$; $E_i = 1/(1 + 10^{(r_j - r_i)/400})$ |
| Elo update | $r_i \leftarrow r_i + K(S - E_i)$, a gradient step of size $K \cdot 400/\ln 10$ |
| Davidson's ties | outcomes in proportion $\pi_i : \pi_j : \nu\sqrt{\pi_i\pi_j}$ |
| Two models | $\hat\beta_1 - \hat\beta_2 = \log(b/c)$, variance $1/b + 1/c$ |
What we have learned
A table of pairwise wins is a set of comparisons against different opponents, and win shares inherit the schedule. The Bradley–Terry model credits each win according to the opponent: its maximum likelihood ratings make every model's observed wins equal its expected wins, exist exactly when every split of the models has wins in both directions, and are found by an iteration that never lowers the likelihood. On the MT-bench votes it reverses the order of Claude-v1 and GPT-3.5 given by win shares, and says that their order is not determined. Votes clustered by question make the model's standard errors too small by a factor near 1.4 to 1.8 here, and ties can be dropped, halved or modeled. Elo's rule is a stochastic gradient step on the same likelihood, so it wanders around the batch estimate and depends on the order of the votes. The scale of any rating belongs to its judge.
Pairwise ratings answer which of several models is preferred. When the evidence is instead a score for every model on each of several tasks, ratings are replaced by ranks within each task, and Comparing Systems Across Many Tasks develops the tests that use them. The topic page Statistics of Model Evaluation lists the unit's other articles.
References
- Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. Section 2.4 and equation 2.1. record · 9 pages
- Boubdir, M., Kim, E., Ermis, B., Hooker, S., and Fadaee, M. (2023). Elo uncovered: Robustness and best practices in language model evaluation. arXiv:2311.17295. Section 2, equations 1 and 2. record
- Bradley, R. A., and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. record
- Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M. I., Gonzalez, J. E., and Stoica, I. (2024). Chatbot Arena: An open platform for evaluating LLMs by human preference. arXiv:2403.04132. record
- Davidson, R. R. (1970). On extending the Bradley–Terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329), 317–328. record
- Ford, L. R., Jr. (1957). Solution of a ranking problem from binary comparisons. The American Mathematical Monthly, 64(8, Part 2), 28–33. record
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. International Conference on Learning Representations. arXiv:2009.03300. record · 7 pages
- Hunter, D. R. (2004). MM algorithms for generalized Bradley–Terry models. The Annals of Statistics, 32(1), 384–406. record
- Lange, K., Hunter, D. R., and Yang, I. (2000). Optimization transfer using surrogate objective functions. Journal of Computational and Graphical Statistics, 9(1), 1–20. record
- Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., et al. (2022). Holistic evaluation of language models. arXiv:2211.09110. record · 6 pages
- Rao, P. V., and Kupper, L. L. (1967). Ties in paired-comparison experiments: A generalization of the Bradley–Terry model. Journal of the American Statistical Association, 62(317), 194–204. record
- Zermelo, E. (1929). Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung. Mathematische Zeitschrift, 29, 436–460. record
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track. arXiv:2306.05685. Data: Hugging Face dataset
lmsys/mt_bench_human_judgments(CC BY 4.0), revision f7d2896d2cc5d80f8b55c2bbc722613555233c25. record