Neural Ranking and Hybrid Retrieval

A unit on ranking a short list precisely with a model that reads the query and the passage together, the designs that lie between that and one vector per passage, and the combination of lexical and learned retrieval into one result list. Cross-encoder re-ranking, multi-stage pipelines and their ceilings, late interaction, learned sparse retrieval, rank fusion, hybrid retrieval, and diversification of the final list. Each article says what it proves, what it cites and what its experiment measured.

The first unit of this collection built a retriever that compares meanings: every passage becomes one vector ahead of time, and a query is answered by the nearest vectors. That design is fast because the query and the passage never meet; each is encoded alone. This unit is about what happens when they do meet. A cross-encoder reads the query and a passage as one input, so that each word of the query can be compared with each word of the passage at every layer of the network. It ranks more precisely and costs a full pass of the network for every pair, which confines it to re-ordering a short list that something cheaper has retrieved. Most of the unit follows from that trade.

The unit has three parts. The first is re-ranking: the cross-encoder itself, and the pipeline of stages it sits in, with the ceilings that an early stage sets for every later one. The second is the space between one vector per passage and full interaction: keeping a vector for every token, and keeping the inverted index while learning its term weights. The third is combination: merging the lists of lexical and learned retrievers, deciding whether this site's search should add a learned stage, and choosing the final few results so that they do not repeat each other, which matters most when the reader of those results is a language model.

What the unit assumes

The previous two units: Text Embeddings and Dense Retrieval, in particular the bi-encoder of Sentence and Passage Encoders and the experiments of Dense Passage Retrieval, and Vector Search for searching stored vectors. From the course: BM25 and BM25F as first stages, the Probability Ranking Principle, learning to rank, rank fusion, and the measures of ranked retrieval evaluation and nDCG. The transformer is assumed as Building Self-Attention and Assembling the Transformer Block develop it.

What is proved, cited and measured

Derived or proved on the pages, among other results: that a two-output relevance classifier is a logistic model of one logit, the gradient of its pointwise loss, and that training on balanced classes shifts every logit by one constant, which leaves rankings unchanged; the cost of a cross-encoder per pair from its shape; the recall ceiling a first stage sets for every later stage, the oracle ceiling of nDCG for a candidate set, and that both can only rise with depth; that one pass of a sliding window with a perfect window ranker returns the true top $w - s$; for late interaction, properties of the MaxSim score (averaging in place of the maximum collapses it to a single vector, the score is monotone in the document's tokens, a search over some tokens gives a lower bound) and a bound on what compression does to it; for learned sparse retrieval, an exact identity for the cost of a sparse index and what the FLOPS regularizer minimizes; for score interpolation, that normalization only rescales the weight, that a document beaten on both scores cannot rise above the one that beats it, and that the fused ranking changes at finitely many weights; that reciprocal rank fusion ignores any increasing rescaling of its inputs (in the course's rank fusion article); for diversification, the exact threshold at which maximal marginal relevance admits a near copy, that rescaling its relevance scores is a change of its parameter, and that the coverage objective is submodular, so that greedy selection reaches at least $1 - 1/e$ of the optimum. Cited and not proved: every published measurement of effectiveness, and the other results each article lists in its own account of what it proves.

The experiments use the collection's two test beds. Cranfield, the course's standard test collection, has 1,400 abstracts and 225 queries with judgments for every query, so a re-ranker can be scored on any list. This site is used as it stood at one commit, 679 pages cut into 11,264 passages, with its 60 judged topical queries and 200 known-item queries. The site's judgments have a limit that every article states: as they stood at commit c057efe, the judged pages for each topical query were those the site's own ranker, BM25F, put in its top 20 on 13 November 2026, graded by a model judge whose agreement with the owner on 150 calibration pairs was a quadratic-weighted kappa of 0.85, so a system can be compared fairly only where its own top results lie inside that pool. Hybrid Retrieval compares rankers on the site after the pool was extended to them and judged. Re-ranking BM25F's top 20 stays inside it; retrieving from the whole site does not. The models are small pretrained networks run on a laptop processor and pinned by revision: the unit 1 bi-encoder, sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41; the cross-encoder cross-encoder/ms-marco-MiniLM-L6-v2 at revision 233902d25c440f23af6f7d6e94d2946bac0bee0a, which unit 1 used as a distillation teacher; and, in the articles on late interaction and learned sparse retrieval, a pretrained ColBERTv2 and a pretrained SPLADE model, each named with its revision on its page. Training such models is beyond that machine, so results at the scale of MS MARCO are reported from the papers with the table each number comes from.

Notation

The unit keeps the first unit's notation. A query is $q$ and a passage $p$; $d$ is used for whole documents (a Cranfield abstract, a page of this site). The score of a passage for a query is $s(q, p)$, and each article says which score it is; for a cross-encoder it is the logit read from the output at the [CLS] position, and $\sigma(s) = 1/(1 + e^{-s})$ is its reading as a probability. A transformer has $\ell$ blocks of width $m$, and a text of $L$ word pieces has token vectors $\vec{h}_1, \dots, \vec{h}_L$. In a pipeline, a stage receives a list of $n$ candidates (its depth), and a measure is computed at a cutoff $k$, as in nDCG@10. The constant of reciprocal rank fusion is the $k$ of rank fusion; where a cutoff appears on the same page it is written $k_{\mathrm{RRF}}$. The weight of a linear interpolation of two scores is $\alpha$, and the trade-off parameter of maximal marginal relevance is $\lambda$, which weights relevance (in xQuAD, $\lambda$ weights diversity; the article on diversification says so where it matters). Logarithms are natural, $\ln$, except $\log_2$ in the discounts of DCG.

Re-ranking

  • Cross-Encoder Re-ranking: the query and passage read as one input; the monoBERT model of Nogueira and Cho (2019) and its pointwise loss, with the two-output head reduced to one logit and the effect of class-balanced training proved; the cost of a pair derived from the model's shape and why a cross-encoder cannot be a first stage; BM25's top 50 re-ranked on Cranfield and BM25F's top 20 re-ranked on this site, with pages scored by their best or first passage.
  • Multi-Stage Ranking: retrieve, then re-rank, as a cascade of stages; the recall ceiling and the oracle ceiling of nDCG, proved, and measured against what the cross-encoder reaches at depths from 10 to 100; a three-stage pipeline; pairwise (duoBERT), sequence-to-sequence (monoT5) and listwise language-model re-rankers, with a proof about the sliding window the last of them use.

Between one vector and full interaction

  • Late Interaction and ColBERT: one vector per token, compared with the query only at the end by summing each query token's best match (MaxSim); ColBERT and ColBERTv2 in full, what can be proved about MaxSim, the size of the index and the error that residual compression adds; a pretrained ColBERTv2 run exhaustively on Cranfield and worked token by token on one query.
  • Learned Sparse Retrieval: keeping the inverted index and learning its weights: DeepCT's term weights, document expansion by predicted queries, SPLADE's expansion over the vocabulary with its FLOPS regularizer and what it minimizes, an exact identity for the cost of a sparse index; a pretrained SPLADE model run on Cranfield through an ordinary inverted index, its expansion measured, switched off and quantized.

Combining and choosing

  • Rank Fusion (a course article): CombSUM and CombMNZ with score normalization, reciprocal rank fusion and its constant, Condorcet fusion, what can be proved about each, and an experiment on Cranfield in which fusing three lexical rankers does not beat BM25. Dense Passage Retrieval reports the contrasting case, BM25 fused with a dense retriever.
  • Hybrid Retrieval: a lexical ranker and a dense retriever combined into one first stage; how differently the two err, measured on Cranfield; score interpolation analysed exactly and compared with rank fusion under cross-validation; and the three rankers compared on this site's judged topical and known-item queries after pooling all three, where the fusion is ahead of BM25F on the topical set and the dense ranker alone is not (whether to ship a dense stage is the owner's decision).
  • Result Diversification: why a list ranked by each result's own relevance repeats itself; maximal marginal relevance with its properties proved; the coverage objective, its submodularity and the greedy bound of $1 - 1/e$, proved, with xQuAD derived as the greedy step; alpha-nDCG; and a five-passage context on this site with and without diversification, the bridge to the next unit.

The next unit, Retrieval-Augmented Generation, takes the passages these methods select and hands them to a language model.

References

  • Carbonell, J., and Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of SIGIR 1998, 335–336. record · 4 pages
  • Lin, J., Nogueira, R., and Yates, A. (2021). Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool. arXiv:2010.06467 (version 3). record · 5 pages
  • Nogueira, R., and Cho, K. (2019). Passage re-ranking with BERT. arXiv:1901.04085. record · 7 pages
  • Nogueira, R., Yang, W., Cho, K., and Lin, J. (2019). Multi-stage document ranking with BERT. arXiv:1910.14424. record · 3 pages