Cranfield query 185 asks for experimental studies on panel flutter. BM25 answers it well: four of its first five abstracts are relevant. The fourth is not. It is abstract 390, some panel-flutter studies using piston theory, which contains every content word of the query except "experimental" and is about theory. A ranker that adds up term weights has no way to see that the one missing word is the one that matters. A model that reads the query and the abstract together, as one text, at least has the chance: the cross-encoder of this article moves abstract 390 to sixth place and puts five relevant abstracts first. Does reading the two texts together rank better in general, and what does it cost? On Cranfield and on this site, the answer measured here is: a little better, by margins the tests cannot separate from chance, at a cost of a full pass of a neural network for every pair of query and passage.
A cross-encoder is a transformer that takes a query and a passage together as one input and returns a score for the pair. The name sets it against the bi-encoder of the previous units, which encodes the two texts apart and compares two vectors. Used as a re-ranker, a cross-encoder re-orders a short list of candidates that a cheaper first stage has retrieved. This is the arrangement with which Nogueira and Cho (2019) brought BERT to passage ranking, and the model of their paper, later named monoBERT, is the reference design.
Here is the route. First we look at the input as the network sees it, and at why nothing about a passage can be computed before the query is known. Then we set out monoBERT and its training loss, and prove three facts about that loss. Next we count what one pair costs, from the model's shape, and what that implies for a whole collection. Then we run a small pretrained cross-encoder as a re-ranker on Cranfield and on this site. The article assumes the transformer as Building Self-Attention and Assembling the Transformer Block build it, and Sentence and Passage Encoders for the bi-encoder. The numbers are computed by tools/experiments/neural-ranking/cross-encoder-reranking.py.
One input for two texts
BERT was built to take either one text or a pair of texts as a single sequence (Devlin et al., 2019, §3). A pair is packed as
$$[\mathrm{CLS}]\;\; q_1 \dots q_{L_q}\;\; [\mathrm{SEP}]\;\; p_1 \dots p_{L_p}\;\; [\mathrm{SEP}],$$
where $q_1, \dots, q_{L_q}$ are the word pieces of the query and $p_1, \dots, p_{L_p}$ those of the passage, so the sequence has $L = L_q + L_p + 3$ positions. The input vector at each position is the sum of three learned vectors: one for the word piece, one for the position, and a segment vector that says whether the position belongs to the first text or the second. The first position always holds the special token [CLS], whose output vector BERT uses as "the aggregate sequence representation for classification tasks". For relevance, that vector is passed to a small output layer that produces the score. Lin, Nogueira and Yates (2021, §3.2.1) note that this way of laying out the query and the text for the transformer is what is called a cross-encoder.
Example (query 185 and abstract 856). Cranfield query 185 is experimental studies on panel flutter, and abstract 856, the first result of both BM25 and the cross-encoder below, begins "some experimental studies of panel flutter". The pinned cross-encoder's tokenizer turns the pair into 296 word pieces. The first segment is [CLS], the six pieces experimental studies on panel flutter . and [SEP], eight positions in all; the abstract and the final [SEP] fill the rest.
Inside the network, every block applies self-attention over all $L$ positions. The vector at a passage position after block 1 is a weighted sum over all positions, query positions included, of transformed input vectors, with weights computed from the passage position's query vector and every position's key vector. So from block 1 onward, each passage position's vector depends on the query. This is the cross-encoder's strength: a query word can find the passage word it matches, or its synonym, or its context, at every layer. It is also its cost.
Why the passage side cannot be prepared in advance. A bi-encoder computes each passage's vector when the collection is indexed, because the vector does not depend on the query. In a cross-encoder only the input vectors of the passage's positions are independent of the query; everything after the first block is not. The script shows this directly. Query 106 of Cranfield has as many word pieces as query 185, so with either query abstract 856 sits at the same positions with the same segment, and the input vectors of its positions are identical, which the script checks. After block 1 they differ, by a median of 0.65 percent of their length; after the sixth and last block, by a median of 51 percent. Only the embedding lookup could be stored, and it is the cheap part. Every pair therefore costs a full pass through the network at query time.
monoBERT: relevance as classification
Nogueira and Cho (2019) took BERT-large, fed "the query as sentence A and the passage text as sentence B", truncated the query to 64 tokens and the passage so that the whole input fit in 512, and used "the [CLS] vector as input to a single layer neural network to obtain the probability of the passage being relevant". They computed this probability "for each passage independently" and ranked the candidates by it. Their paper gave the model no name; Nogueira, Yang, Cho and Lin (2019) later called it monoBERT, to contrast it with a pairwise model, as Lin, Nogueira and Yates (2021, §3.2) record.
The loss
The model was fine-tuned from pretrained BERT with the cross-entropy loss (their equation 1)
$$\mathcal{L} = -\sum_{j \in J_{\mathrm{pos}}} \ln s_j \;-\; \sum_{j \in J_{\mathrm{neg}}} \ln (1 - s_j),$$
where $s_j$ is the model's probability that candidate $j$ is relevant, $J_{\mathrm{pos}}$ indexes the relevant candidates and $J_{\mathrm{neg}}$ the non-relevant ones among BM25's top 1,000. (The paper writes $\log$; for a loss the base only rescales, and we use $\ln$.) Each term involves one candidate, so in the vocabulary of learning to rank this is a pointwise loss. It is the log loss of logistic regression, applied to a network's output.
Lin, Nogueira and Yates (2021, their equation 14) write the output layer as a weight matrix $W$ with two columns, one for each class, followed by a softmax: $s = \mathrm{softmax}(\vec{h}_{\mathrm{CLS}} W + \vec{b})_1$, with $\vec{h}_{\mathrm{CLS}}$ the final vector at [CLS]. The pinned model of this article has a single output instead (its configuration lists one label). The two are the same model, and that is the first of three facts worth proving about the loss.
Fact 1. A two-way softmax is the logistic function of the difference of the two logits. Write $z_0$ and $z_1$ for the two outputs, for "not relevant" and "relevant". Then
$$s = \frac{e^{z_1}}{e^{z_0} + e^{z_1}} = \frac{1}{1 + e^{-(z_1 - z_0)}} = \sigma(z_1 - z_0),$$
dividing numerator and denominator by $e^{z_1}$. With $z_i = \vec{h}_{\mathrm{CLS}} \cdot \vec{w}_i + b_i$, the difference is $\vec{h}_{\mathrm{CLS}} \cdot (\vec{w}_1 - \vec{w}_0) + (b_1 - b_0)$: a single output layer with weights $\vec{w}_1 - \vec{w}_0$ and bias $b_1 - b_0$. Any two-output head equals a one-output head, and a one-output head with weights $\vec{w}$ and bias $b$ is the two-output head with $\vec{w}_1 = \vec{w}$, $b_1 = b$ and $\vec{w}_0 = \vec{0}$, $b_0 = 0$. In this article the score $s(q, p)$ is that single logit, and $\sigma(s(q, p))$ is its reading as a probability.
Fact 2. The gradient of the loss with respect to the logit is the prediction minus the label. For one candidate with label $y \in \{0, 1\}$, logit $z$ and $\sigma = \sigma(z)$, the loss is $\mathcal{L}(z) = -y \ln \sigma - (1 - y)\ln(1 - \sigma)$. The logistic function satisfies $\sigma'(z) = \sigma(1 - \sigma)$, since $\sigma' = e^{-z}/(1 + e^{-z})^2 = \sigma \cdot (1 - \sigma)$. So
$$\frac{\partial \mathcal{L}}{\partial z} = -\frac{y}{\sigma}\,\sigma(1 - \sigma) + \frac{1 - y}{1 - \sigma}\,\sigma(1 - \sigma) = -y(1 - \sigma) + (1 - y)\sigma = \sigma - y .$$
A relevant candidate scored with logit 2 has $\sigma(2) = 0.881$, loss $-\ln 0.881 = 0.127$ and gradient $-0.119$: descent raises its logit, by little, because it is already nearly right. A non-relevant candidate with the same logit has loss 2.127 and gradient 0.881, and its logit is pushed down hard. The gradient with respect to the network's weights follows by backpropagation through $\vec{h}_{\mathrm{CLS}}$ and every block below it; all the parameters are fine-tuned.
Fact 3. Training on balanced classes shifts every logit by the same constant, so the ranking is unchanged and the probability reading is not. Nogueira, Yang, Cho and Lin (2019) built each training batch "by sampling an equal amount of relevant and nonrelevant passages", because non-relevant examples were about a thousand times more frequent. Lin, Nogueira and Yates (2021, §3.2.1) point out that a model trained this way "is unable to accurately estimate the prevalence of relevant texts". Here is the exact effect, under the assumption that the classifier learns the probabilities of the distribution it is trained on. Let $f_1(x)$ and $f_0(x)$ be the densities of a pair's input $x$ among relevant and among non-relevant pairs, and let $\pi$ be the share of relevant pairs. By Bayes' theorem the probability of relevance is $\pi f_1 / (\pi f_1 + (1 - \pi) f_0)$, whose logit is
$$\ln \frac{\pi f_1(x)}{(1 - \pi) f_0(x)} = \ln \frac{f_1(x)}{f_0(x)} + \ln \frac{\pi}{1 - \pi} .$$
Sampling by class changes $\pi$ and leaves $f_1$ and $f_0$ alone. So the logit learned with training share $\pi_t$ exceeds the logit for the true share $\pi$ by
$$\ln \frac{\pi_t}{1 - \pi_t} - \ln \frac{\pi}{1 - \pi},$$
the same number for every pair. Adding a constant to every score changes no ranking. It does change the probabilities. With balanced batches, $\pi_t = \tfrac12$, and one relevant passage among 1,000 candidates, $\pi = 1/1{,}000$, the shift is $\ln 999 = 6.907$, and a pair whose probability of relevance among the candidates is truly 0.01 is read as $\sigma(\ln(0.01/0.99) + \ln 999) = 999/1{,}098 = 0.910$. A cross-encoder's $\sigma(s)$ is a ranking score, and reading it as a calibrated probability needs a correction of this kind and an estimate of $\pi$.
What monoBERT achieved
Nogueira and Cho fine-tuned BERT-large on MS MARCO, a collection of about 8.8 million web passages built by Microsoft from Bing questions (Bajaj et al., 2018), whose development queries have on average one relevant passage each, for 100,000 iterations of 128 pairs, which took about 30 hours on a TPU. That is 12.8 million pairs, "less than 2% of the full training set". Re-ranking BM25's top 1,000 for each development query, monoBERT reached a mean reciprocal rank at 10 (MRR@10) of 36.5, against 16.7 for BM25 and 27.8 for IRNet, the previous best entry on the task's leaderboard; on the hidden evaluation queries it scored 35.8 (their Table 1). They report that a model trained on only 100,000 pairs, less than 0.3 percent of the training data, was already 1.4 points above IRNet. On TREC CAR, a collection of Wikipedia paragraphs, it reached a MAP of 33.5 against 12.3 for untuned BM25 and 15.3 for tuned BM25. Lin, Nogueira and Yates (2021, §3.2.2) add a caution from later experiments: with few training examples, monoBERT can do worse than BM25, and "with 20K total training instances, monoBERT barely improves upon the BM25 baseline".
The model run here
Training a BERT re-ranker on MS MARCO is far outside what a laptop processor can do, so the experiments use a small cross-encoder that others have trained: cross-encoder/ms-marco-MiniLM-L6-v2 at revision 233902d25c440f23af6f7d6e94d2946bac0bee0a, with weights of SHA-256 821d1aa6…aeb0ae. Its model card says it "was trained on the MS Marco Passage Ranking task", lists cross-encoder/ms-marco-MiniLM-L12-v2 as its base model, and reports nDCG@10 of 74.30 on the TREC 2019 Deep Learning passage queries, MRR@10 of 39.01 on the MS MARCO development queries, and 1,800 passages per second on a V100 graphics processor. Its configuration describes a BERT-type network with $\ell = 6$ blocks of hidden width $m = 384$, 12 attention heads and a feed-forward width of 1,536, one output, and inputs of at most 512 word pieces. The previous unit used the same checkpoint as the teacher in Hard Negatives and Distillation, and found it a weak judge of Cranfield's labelled positives; that finding is taken up below.
What one pair costs
The cost of a cross-encoder follows from its shape, and the model run here is small enough to count exactly.
Parameters. A BERT network with vocabulary $V$, at most $P$ positions, two segment types, hidden width $m$, feed-forward width $F$ and $\ell$ blocks has:
- embeddings: $V m$ for the word pieces, $P m$ for positions, $2m$ for segments, and $2m$ for a layer normalization, $(V + P + 4)\,m$ in all;
- in each block: four $m \times m$ projections with biases for the attention's queries, keys, values and output, $4(m^2 + m)$; a feed-forward layer of $m F + F$ and one of $F m + m$; two layer normalizations of $2m$ each;
- a pooling layer over [CLS], $m^2 + m$, and the output layer, $m + 1$.
With $V = 30{,}522$, $P = 512$, $m = 384$, $F = 1{,}536$ and $\ell = 6$, that is $11{,}918{,}592$ for the embeddings, $1{,}774{,}464$ per block and $10{,}646{,}784$ for six, $147{,}840$ for the pooler and 385 for the output: $22{,}713{,}601$ parameters, which is the number the script counts in the loaded model. More than half of them are the word-piece table, which costs a lookup and no arithmetic.
Arithmetic per pair. For an input of $L$ positions, one block computes the attention's four projections, $4 L m^2$ multiply-adds; the attention scores between all pairs of positions and the weighted sums of value vectors, $L^2 m$ each; and the feed-forward layers, $2 L m F$. With $F = 4m$, as here, a block costs
$$12\,L\,m^2 + 2\,L^2 m$$
multiply-adds, and the network $\ell$ times that. The first term grows linearly with the input's length, the second quadratically. At $L = 256$ the six blocks take $3{,}019{,}898{,}880$ multiply-adds, about $3.0 \times 10^9$, of which the quadratic attention term is 10.0 percent; at $L = 512$ they take $6.6 \times 10^9$ and the attention term is 18.2 percent. The median pair of query and abstract among BM25's top 50 on Cranfield has 240 word pieces and costs about $2.8 \times 10^9$ multiply-adds. (Softmax, normalization and activation functions add terms linear in $L$ that are not counted.)
Against a bi-encoder. A bi-encoder pays that cost once per passage, when the collection is indexed, and at query time pays it once for the query. Its cost per candidate at query time is one dot product of two 384-dimensional vectors: 384 multiply-adds. A pair through the cross-encoder at $L = 256$ costs $3{,}019{,}898{,}880 / 384 = 7{,}864{,}320$ times as much.
Why it cannot be a first stage. On the laptop that ran the script, while other jobs were running, the cross-encoder took roughly 35 to 110 milliseconds per Cranfield pair in four runs, depending on the load, in batches of 32 on four processor threads (a timing the script prints and does not assert). Scoring all 1,400 abstracts for one query would then take from under a minute to about two and a half minutes. The site's passages are longer: the median pair has 334 word pieces against Cranfield's 240, which by the formula above costs 1.44 times as many multiply-adds, so all 11,264 passages would take about $11{,}264 \times 1.44 \times 35$ to $110$ milliseconds, roughly 10 to 30 minutes, for every query. Sentence and Passage Encoders counts the same contrast in passes: 675,840 cross-encoder passes to score every site passage for the 60 test queries, against 11,324 bi-encoder passes for the passages and queries together. Lin, Nogueira and Yates (2021, §3.2) make the point with "a million texts for every query". The cross-encoder's place is therefore at the end of a pipeline, on a list that something cheaper has cut short. How short, and with what loss, is the subject of Multi-Stage Ranking.
Texts longer than the input
The input holds at most 512 word pieces, and attention's cost grows with the square of the length. Cranfield's abstracts mostly fit: of the 11,250 pairs of query and abstract re-ranked below, 383 exceed 512 word pieces and are cut at the end of the abstract. This site's pages do not fit; they run to thousands of words.
Dai and Callan (2019) gave the standard answers. They split a document into overlapping passages, scored each passage with the cross-encoder, and took as the document's score the score of its first passage (FirstP), its best passage (MaxP) or the sum over passages (SumP). With no passage-level judgments for training, they labelled every passage of a relevant document relevant. The site's pages are already cut into passages by the warehouse's chunker, so the experiment below scores every passage of a page and compares MaxP with FirstP.
Re-ranking BM25 on Cranfield
Cranfield has 1,400 abstracts on aeronautics and 225 queries with judgments for every query, graded 1 (complete answer) to 4 (minimum interest); the course's article on test collections describes it. The first stage is the course's BM25 ($k_1 = 1.2$, $b = 0.75$, stemming and stop words as in Comparing Ranking Models). The cross-encoder scores the query against each of BM25's top 50 abstracts (title and text, as one passage), and the 50 are re-ordered by its logit; abstracts below rank 50 keep BM25's order. That is 11,250 pairs. For comparison the same is done to the top 50 of the unit 1 bi-encoder, which ranks all 1,400 abstracts by cosine (Dense Passage Retrieval). nDCG@10 uses gain $5 - \text{grade}$, as the course does on Cranfield; recall counts any graded abstract as relevant.
| Run | nDCG@10 | Recall@10 | Recall@50 |
|---|---|---|---|
| BM25 | 0.368 | 0.397 | 0.642 |
| BM25, top 50 re-ranked by the cross-encoder | 0.383 | 0.411 | 0.642 |
| Bi-encoder | 0.382 | 0.416 | 0.686 |
| Bi-encoder, top 50 re-ranked by the cross-encoder | 0.396 | 0.430 | 0.686 |
Recall@50 does not change under re-ranking, since the same 50 abstracts are only re-ordered; it is the ceiling that Multi-Stage Ranking proves for every re-ranker.
Is the re-ranked list better than BM25's? By nDCG@10 the difference is $+0.015$, and the paired tests of the course's article cannot tell it from chance. Over the 225 queries, $t = 1.50$ on 224 degrees of freedom, $p = 0.14$, with a 95% interval of $[-0.005, 0.035]$; the randomization test (20,000 sign assignments, seed 1) gives $p = 0.14$. The re-ranked list is ahead on 103 queries, behind on 85 and tied on 37. Re-ranking the bi-encoder's list gains about as much, $+0.014$ ($p = 0.26$). The two re-ranked lists do differ from each other: re-ranking the bi-encoder's candidates beats re-ranking BM25's by 0.012 ($t = 2.63$, $p = 0.009$), and since the re-ranker is the same, the difference comes from the candidates, which the bi-encoder supplies with higher recall.
That is a small effect beside monoBERT's on MS MARCO, where re-ranking more than doubled BM25's MRR@10. Three differences between the settings are in view, and the experiment separates none of them. The model is small, six blocks against BERT-large's 24. It was trained on web questions and is applied, with no training on the collection, to aeronautics abstracts and to questions written by the authors of the papers in the collection (Test Collections tells how). And BM25 is a strong first stage here: of the 1,612 relevant query-abstract pairs, all but 61 share at least one stemmed term with the query (Dense Passage Retrieval counts them). Published zero-shot results show both sides: in BEIR's comparison, which Limits of Single-Vector Retrieval reproduces, BM25 followed by a cross-encoder is above BM25 on 16 of 18 collections, and the margins range from large to slight.
What the score separates. Among BM25's top 50 for all queries, 938 of the 11,250 pairs are judged relevant, 8.3 percent. The cross-encoder's median logit is 1.31 for the relevant pairs and $-3.64$ for the others, and a relevant pair outscores a non-relevant one (from any two queries) with probability 0.781, the area under the ROC curve. The mean of $\sigma(s)$ over all 11,250 pairs is 0.259, about three times the share that is relevant: read as probabilities, the scores overstate relevance on this collection. Fact 3 shows one way such an offset arises, though the model card does not say how the classes were balanced in training, and a shift of domain would distort the scores too. The previous unit saw the domain shift from the other side: used as a teacher in Hard Negatives and Distillation, the same model gave its labelled positives on Cranfield, one relevant abstract chosen at random for each query, a median score of −0.76, below zero. For ranking within a query, only the scores' order matters.
Example (query 185). BM25 and the cross-encoder agree on the first two abstracts, 856 and 857, both relevant. BM25's fourth is abstract 390, the theory paper of the opening, which the judges did not mark relevant. The cross-encoder moves it to sixth and fills ranks 3 to 5 with relevant abstracts, so its first five are all relevant, where BM25's first five hold four. Lower down it does worse. It brings up abstract 948, on flutter tests of a full-scale aircraft stabilizer, which sounds experimental and was not judged relevant, and it drops abstract 859, relevant and at BM25's rank 9, out of the top ten (its logit, 2.27, is low). nDCG@10 falls from 0.807 to 0.759, while the relevant abstracts within the first five rise from four to five. The re-ranker demoted the one highly ranked abstract about theory, as a reader attending to the word "experimental" would; the experiment does not show that this is why. Its errors fell lower in the list, where nDCG@10 still counts them.
Re-ranking BM25F on this site
The site is used as unit 1 fixed it: 679 pages at one commit, cut by the warehouse's chunker into 11,264 passages of whole paragraphs within one section, each prefixed with its page's title and section heading (Dense Passage Retrieval describes the cut). The first stage is the site's live ranker, BM25F over whole pages with weighted fields (How This Search Works), run on those pages.
What the judgments allow. The 60 topical queries have graded judgments from 0 to 3, made by pooling. The pool holds one ranker's results: on 13 November 2026, the top 20 pages of the live BM25F ranker for each query, 1,200 query-page pairs, of which 1,199 were judged. A model judge (Claude Sonnet 5.5, reading each page's frontmatter, headings and the sections bearing on the need) graded them; on 150 of the pairs, which the owner also graded, its agreement with the owner was a quadratic-weighted kappa of 0.85. For those 150 pairs a second model grading, made by reading each page in full, and the owner's grades also exist, and a pair's grades are averaged over its judges. The pool was re-drawn on 17 November for pages added since, and later extended to two more rankers; every pair in it has been judged since. This article uses the judgments exactly as they stood at commit c057efe, on purpose, before those additions, when the pool's record named only BM25F and its new pairs had no grades yet; Hybrid Retrieval reports the judged comparison across rankers. Pages outside a query's pool count as not relevant, so a ranker that retrieved other pages would be judged on its rival's documents, the weakness of pooling that the course's article on pooling explains. A re-ranker of BM25F's top 20 avoids it: its top 20 is BM25F's top 20 in another order. The only gap is that the pages changed between the pooling and the snapshot used here: of the 1,200 query-page pairs re-ranked here, 1,151 have a judgment at that commit and 49 do not, and those count as not relevant: 48 entered BM25F's top 20 after the pooling, as pages changed, and one was in the pool but never judged. Every comparison in this section is therefore a comparison of orderings of the same pages, nearly all of them judged, and says nothing about pages outside BM25F's top 20.
The re-ranking. For each topical query the cross-encoder scores every passage of the 20 pages, 23,851 pairs in all, about 20 passages per page. A page is scored by its best passage (MaxP) or by its first (FirstP), and the 20 pages are re-ordered by that score. nDCG@10 uses gain $2^g - 1$ for grade $g$, as the site's evaluation does.
| Run | nDCG@10 | Judged share of the top ten |
|---|---|---|
| BM25F | 0.753 | 0.978 |
| BM25F's top 20 re-ranked, MaxP | 0.783 | 0.968 |
| BM25F's top 20 re-ranked, FirstP | 0.765 |
The MaxP re-ranking is $+0.030$ above BM25F, with $t = 1.45$ on 59 degrees of freedom, $p = 0.15$, and a 95% interval of $[-0.011, 0.070]$; it is ahead on 35 queries and behind on 25. FirstP gains $+0.011$ ($p = 0.59$), and MaxP is ahead of FirstP by 0.018 ($p = 0.26$). None of the three differences can be told from chance on 60 queries. Dai and Callan's own comparison went both ways: re-ranking with nDCG@20, MaxP was ahead of FirstP on the Robust04 news collection, while FirstP was the best of their BERT variants on description queries over the ClueWeb09-B web pages (their Table 2). This experiment does not settle which suits the site. The query set has a development half and a held-out half, which matter when something is tuned; nothing was tuned here, and on the held-out 31 queries the two runs score 0.771 and 0.781.
The known-item queries. The site's second test set needs no pool: 200 queries, each written for one target page, scored by the target's rank. Scoring every passage of BM25F's top 20 pages for 200 queries would take about 80,000 cross-encoder passes, so this run uses a cheaper design, of the kind Multi-Stage Ranking studies: BM25F's top 10 pages, of each page the 3 passages whose bi-encoder vectors are nearest the query's (all of them if a page has fewer), and the page scored by the best of those. That is 5,973 pairs. The target is in BM25F's top 10 for all 200 queries.
| Run | MRR | Target first |
|---|---|---|
| BM25F | 0.942 | 0.905 |
| Top 10 re-ranked by the cross-encoder (3 passages per page) | 0.779 | 0.615 |
Here the re-ranker is clearly worse: the mean difference in reciprocal rank is $-0.163$ ($t = -7.85$ on 199 degrees of freedom, $p < 0.001$), with the re-ranked list ahead on 9 queries, behind on 66 and tied on 125. The run does not say why. Two features of the setup may contribute, and neither was tested. The known-item queries were written from each target page's title, summary and headings, and their documentation reports that on average 89 percent of a query's terms occur in the target's summary; BM25F indexes the summary as a weighted field, while the passages the cross-encoder reads hold the page's body and its title, not its summary. And choosing three passages per page by the bi-encoder may miss the passage the cross-encoder would have scored highest, though on the topical queries Multi-Stage Ranking found that this choice cost little at ten pages, as here (0.752 against 0.755 for reading every passage, a difference the tests cannot separate from chance). A third possibility, also untested, is that a model trained to find passages that answer a question is being asked here to identify the page a query was written about. The unit 1 dense retriever, which reads the same passages, also lost to BM25F on these queries (Dense Passage Retrieval).
What is proved here, what is cited, and what the experiments show
Proved or derived on the page. That a two-output softmax head is a one-output logistic head; that the gradient of the pointwise loss with respect to the logit is $\sigma - y$; that class-balanced training shifts every logit by the same constant, which leaves rankings unchanged; the parameter count of the pinned model from its configuration, and the multiply-adds per pair as a function of input length; that the passage side of a cross-encoder depends on the query from the first block on (shown on one pair by the script, and following from the form of attention).
Cited. monoBERT's design, training and results (Nogueira and Cho, 2019); its name and balanced batches (Nogueira, Yang, Cho and Lin, 2019); BERT's input (Devlin et al., 2019); the term cross-encoder, the two-output head and the remarks on training data and priors (Lin, Nogueira and Yates, 2021); FirstP, MaxP and SumP (Dai and Callan, 2019); MS MARCO (Bajaj et al., 2018); the pinned model's training and its published scores (its model card).
Shown by the experiments. On Cranfield, with no training on the collection, re-ranking BM25's top 50 raised nDCG@10 from 0.368 to 0.383 and re-ranking the bi-encoder's top 50 from 0.382 to 0.396; neither gain is distinguishable from chance, while the candidates' recall made a detectable difference between the two re-ranked lists. The cross-encoder's score separated relevant from non-relevant abstracts with an area under the ROC curve of 0.781, and its probability reading overstated relevance. On this site, re-ranking BM25F's top 20 pages by their best passage raised nDCG@10 from 0.753 to 0.783 on the topical queries, again not distinguishable from chance, and on the known-item queries a cheaper version of the re-ranking, three passages per page, was clearly worse than BM25F. These are observations with one small cross-encoder trained on web questions. They do not show that cross-encoders are weak re-rankers in general: the published results above, with larger models trained for the task, show the opposite on the collections they were trained for.
Worked problems
These problems are posed on Cranfield query 185, on the pinned model and on the site's passages; the unit has no course assignments. The script recomputes every number.
Problem 1 (the pointwise loss on three candidates). A training batch holds three candidates for one query: a relevant one with logit 2, a non-relevant one with logit 2, and a non-relevant one with logit $-3$. Compute the loss of each, the total, and the gradient of each with respect to its logit. Which candidate will a step of gradient descent move most?
Solution. With $\sigma(2) = 0.881$ and $\sigma(-3) = 0.047$: the relevant candidate's loss is $-\ln 0.881 = 0.127$; the first non-relevant candidate's is $-\ln(1 - 0.881) = 2.127$; the second's is $-\ln(1 - 0.047) = 0.049$. The total is 2.302. By Fact 2 the gradients are $\sigma - y$: $0.881 - 1 = -0.119$, $0.881 - 0 = 0.881$, and $0.047 - 0 = 0.047$. The non-relevant candidate with a high logit has a gradient more than seven times the size of either other one's, so descent lowers its logit most. The candidate the model already scores correctly contributes almost nothing; this is why training on the candidates BM25 ranks highly, as monoBERT did, gives the model the examples it can learn from.
Problem 2 (counting a larger cross-encoder). monoBERT's base-size variant is built on BERT-base: $V = 30{,}522$, $P = 512$, $m = 768$, $F = 3{,}072$, $\ell = 12$. Count its parameters with the formula of this article, and compare with the size the BERT paper gives.
Solution. Embeddings: $(V + P + 4)\,m = 31{,}038 \times 768 = 23{,}837{,}184$. One block: $4(m^2 + m) + 2m + (mF + F) + (Fm + m) + 2m = 2{,}362{,}368 + 1{,}536 + 2{,}362{,}368 + 2{,}360{,}064 + 1{,}536 = 7{,}087{,}872$; twelve blocks, $85{,}054{,}464$. Pooler: $m^2 + m = 590{,}592$. Output: $m + 1 = 769$. Total: $109{,}483{,}009$, about 109.5 million, which matches the "Total Parameters=110M" that Devlin and colleagues give for BERT-base. With the pinned model's $22{,}713{,}601$, the ratio is 4.8. Per pair of $L$ word pieces the arithmetic grows faster than the parameter count: $\ell(12 L m^2 + 2 L^2 m)$ is twelve blocks at width 768 against six at 384, about $2 \times 4 = 8$ times as much for the linear term.
Problem 3 (scoring the whole site for one query). The 23,851 pairs of query and passage re-ranked on this site have a median length of 334 word pieces. Estimate the multiply-adds for one pair of that length, and for scoring all 11,264 passages for one query. Compare with a bi-encoder.
Solution. With $\ell = 6$, $m = 384$, $F = 1{,}536$ and $L = 334$, one block costs $12 L m^2 + 2 L^2 m = 591{,}003{,}648 + 85{,}675{,}008 = 676{,}678{,}656$ multiply-adds, and six blocks $4{,}060{,}071{,}936$, about $4.1 \times 10^9$. For 11,264 passages that is about $4.6 \times 10^{13}$ per query. A bi-encoder of the same shape with stored passage vectors needs one pass for the query, $171{,}048{,}960$ multiply-adds, under $2 \times 10^8$, for a query of 16 word pieces, and $11{,}264 \times 384 = 4{,}325{,}376$ for the dot products. The cross-encoder's cost for the whole site, $4.6 \times 10^{13}$, is about 260,000 times the bi-encoder's $1.75 \times 10^8$, and it grows in proportion to the collection, while the bi-encoder's grows only by 384 multiply-adds per passage.
Problem 4 (page scores from passages). Page $A$ has three passages with logits $-2.0$, $5.1$ and $0.3$; page $B$ has two, with $3.2$ and $3.0$. Rank the two pages by MaxP, by FirstP, by the sum of the logits, and by the sum of the probabilities $\sigma(s)$.
Solution. MaxP: $A$ scores 5.1 and $B$ 3.2, so $A$ is first. FirstP: $A$ scores $-2.0$ and $B$ 3.2, so $B$ is first. Sum of logits: $A$ has $3.4$ and $B$ $6.2$, so $B$. Sum of probabilities: $A$ has $0.119 + 0.994 + 0.574 = 1.688$ and $B$ has $0.961 + 0.953 = 1.913$, so $B$. Three of the four rules prefer $B$, the page with no passage as good as $A$'s best. FirstP depends on where a page puts its best material, and the sums depend on how many passages a page has and how good its weak ones are. MaxP asks only whether a page holds one passage that answers the query, which is the reading that suits a reader looking for an answer; in Dai and Callan's experiments it was the most effective of the three on news articles, and on web pages it led on title queries, while FirstP was the best of the three on description queries (their Table 2).
What we have learned
A cross-encoder reads the query and the passage as one sequence, so attention connects every word of each with every word of the other from the first block on, and nothing about a passage beyond its input embeddings can be computed before the query arrives. Trained as monoBERT was, as a binary classifier with a pointwise cross-entropy loss, it is logistic regression on the [CLS] vector of a fine-tuned transformer: the gradient with respect to its logit is the prediction minus the label, and balanced training shifts every logit by one constant, which leaves its rankings alone and its probabilities wrong. Its cost is a full pass per pair, about $3 \times 10^9$ multiply-adds for the small model run here on 256 word pieces, against 384 for a bi-encoder's dot product, which is why it re-ranks and does not retrieve. On MS MARCO, where it was trained, monoBERT more than doubled BM25's MRR@10. On Cranfield and this site, a small model trained elsewhere moved nDCG@10 up by 0.015 and 0.030, margins the tests could not separate from chance.
The re-ranker can only re-order what it receives, and on Cranfield the bi-encoder's candidates served it better than BM25's. How deep the list should be, what the first stage limits, and what further stages cost are the subject of Multi-Stage Ranking.
References
- Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., and Wang, T. (2018). MS MARCO: A human generated MAchine Reading COmprehension dataset. arXiv:1611.09268 (version 3). record · 2 pages
- cross-encoder (n.d.). cross-encoder/ms-marco-MiniLM-L6-v2. Model card and configuration at revision
233902d25c440f23af6f7d6e94d2946bac0bee0a. https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2 (accessed 18 November 2026). record - Dai, Z., and Callan, J. (2019). Deeper text understanding for IR with contextual neural language modeling. In Proceedings of SIGIR 2019, 985–988. arXiv:1905.09217. record · 3 pages
- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, Volume 1, 4171–4186. record · 5 pages
- Lin, J., Nogueira, R., and Yates, A. (2021). Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool. arXiv:2010.06467 (version 3). record · 5 pages
- Nogueira, R., and Cho, K. (2019). Passage re-ranking with BERT. arXiv:1901.04085. record · 7 pages
- Nogueira, R., Yang, W., Cho, K., and Lin, J. (2019). Multi-stage document ranking with BERT. arXiv:1910.14424. record · 3 pages