Evaluating Embedding Models

How to measure an embedding model as a first-stage retriever and how to choose one for a given collection. Recall at a cutoff and nDCG@10 and what each is for, the BEIR and MTEB benchmarks with what they do and do not measure, two small pinned models compared on Cranfield and on this site with intervals and paired tests, where they cannot be told apart, how many queries such a comparison needs, and vectors shortened by truncation, with a model whose card says it supports Matryoshka truncation against truncation, principal components and random projection of an ordinary one. Every statement about a particular model is dated November 2026. Four worked problems compute the measures and a paired test by hand.

This unit has embedded Cranfield's 1,400 abstracts and this site's 11,264 passages with one small encoder, all-MiniLM-L6-v2. A second encoder of the same size produces vectors of the same 384 dimensions and could be dropped in without changing a line of the retrieval code. As of November 2026, its model card says it is built on the first and lists "Optimized for retrieval tasks" among its properties. Does that mean it would retrieve this site's pages better?

A card's claim, like a benchmark score, is evidence, and it does not settle the matter. An embedding model in this article is a bi-encoder used for first-stage retrieval, as dense passage retrieval defines it. Evaluating one means choosing measures that fit that job, reading public benchmarks for what they test, and then comparing candidates on one's own collection with enough queries to tell a difference from noise. On Cranfield and on this site's queries the two encoders turn out to be indistinguishable. What does make a difference is which measure is asked for and how the judgments were collected.

Here is the route. First we fix the two measures used for first-stage retrieval, recall at a cutoff and nDCG@10, and compute both for one query. Then we describe the two public benchmarks, BEIR and MTEB, and list what a benchmark average leaves out. Next we compare two pinned models on Cranfield and on this site with paired tests, and ask how many queries the comparison needs. Then we shorten the vectors, which is where Matryoshka training comes in. A procedure for choosing a model closes the article. It assumes the measures of retrieval evaluation, in particular nDCG, and the tests of significance testing. The numbers come from tools/experiments/dense/evaluating-embedding-models.py.

Statements about particular models and benchmark versions change quickly. Each one below carries the words "as of November 2026" and rests on a model card or a documentation page read on 17 November 2026, at the revision the script pins.

Two measures for a first stage

A first-stage retriever has one of two jobs, and each has its measure.

If its results go to a second stage, a re-ranker or a language model that reads the top passages, then what matters is whether the relevant passages are in the list at all. Order within the list will be redone. The measure is recall@$k$, the share of a query's relevant items found in the top $k$, which ranked retrieval evaluation defines. The cutoff $k$ is the number of candidates the second stage will read, commonly 100 or 1,000. Recall at that cutoff is a ceiling on everything downstream: a passage the first stage did not return cannot be ranked first by any re-ranker.

If its top results are shown directly, order matters and so do grades of relevance. The measure is nDCG@10, which its article derives: gains for each of the top ten results, discounted by the logarithm of the rank, divided by the best achievable total. The authors of the BEIR benchmark chose it as their single measure for a stated reason: measures such as precision and recall ignore rank, mean reciprocal rank and average precision cannot use graded judgments, and nDCG@$k$ "provides a good balance suitable for both tasks involving binary and graded relevance judgements" (Thakur and colleagues, 2021, section 3.3).

The two can disagree, and a model should be judged on the one that matches its use.

Query 185 by hand. Cranfield query 185, experimental studies on panel flutter, has 9 relevant abstracts: four with gain 3, two with gain 2 and three with gain 1. Cranfield grades run from 1 (a complete answer) to 4 (of minimum interest), and the course's convention gives a grade $g$ the gain $5 - g$. The unit's encoder returns these ten abstracts first, with their gains:

Rank 1 2 3 4 5 6 7 8 9 10
Abstract 658 864 856 1008 878 857 390 627 948 391
Gain 0 0 2 2 0 3 0 0 0 0

Three of the nine relevant abstracts are in the top ten, so recall@10 is $3/9 = 0.333$. All nine are in the top 100, so recall@100 is 1.000. For nDCG@10, the gains at ranks 3, 4 and 6 are discounted by $\log_2 4$, $\log_2 5$ and $\log_2 7$:

$$\mathrm{DCG@10} = \frac{2}{2} + \frac{2}{2.3219} + \frac{3}{2.8074} = 1 + 0.8614 + 1.0686 = 2.9300.$$

The ideal ordering puts the gains $3, 3, 3, 3, 2, 2, 1, 1, 1$ first, for an ideal DCG@10 of 10.1208, so nDCG@10 is $2.9300/10.1208 = 0.2895$.

BM25 on the same query has nDCG@10 0.8071 and recall@10 0.667, and its recall@100 is also 1.000. As a list to show a reader, the dense list is far worse for this query. As a candidate set of 100 for a re-ranker, the two are equally good: each contains every relevant abstract. Which system is "better on query 185" depends on what its output is for.

The public benchmarks

A benchmark answers a different question from the one a practitioner has. The benchmark asks how a model does on many collections on average. The practitioner asks how it will do on one. It helps to know exactly what the two best-known benchmarks contain.

BEIR

Thakur, Reimers, Rücklé, Srivastava and Gurevych (2021) assembled BEIR from 18 publicly available English datasets covering nine kinds of retrieval task: fact checking, citation prediction, duplicate-question retrieval, argument retrieval, news retrieval, question answering, tweet retrieval, biomedical retrieval and entity retrieval. The collections range from 3.6 thousand to 15 million documents. The benchmark is zero-shot: a system is trained on other data, in practice mostly on MS MARCO, and is tested on these 18 without further training. Each dataset is scored by nDCG@10.

What BEIR measures is therefore generalization: how well a retriever trained on web search questions transfers to other subjects and other kinds of query. Its paper's main finding, reported in full in limits of single-vector retrieval, was that the dense retrievers of 2021 did not transfer well and that BM25 was a hard baseline to beat.

MTEB

Muennighoff, Tazi, Magne and Reimers (2023) built the Massive Text Embedding Benchmark around a wider question: is one embedding model good for everything embeddings are used for? Its first version spans 8 tasks, 58 datasets and 112 languages, and the paper evaluates 33 models. The tasks and their main measures are:

Task What is scored Main measure
Retrieval ranking a corpus for each query nDCG@10
Reranking ordering a given list of relevant and irrelevant texts mean average precision
Clustering mini-batch k-means clusters of the embeddings against labels V-measure
Classification a logistic regression trained on the embeddings accuracy
Pair classification deciding whether two texts are duplicates or paraphrases average precision
Semantic textual similarity agreement of cosine with human similarity ratings Spearman correlation
Summarization scoring machine summaries against human ones Spearman correlation
Bitext mining finding each sentence's translation F1

Its retrieval task is BEIR: the paper states that "MTEB reuses datasets and evaluation from BEIR". The clustering measure is the one clustering evaluation describes.

The paper's headline finding is that "no particular text embedding method dominates across all tasks". Its Table 1, the averages over the 56 English datasets, shows how much an overall average can hide. Three rows:

Model (as named in the paper) Retrieval, 15 datasets Semantic similarity, 10 datasets Average, 56 datasets
MiniLM-L6 41.95 78.90 56.26
ST5-XXL 42.24 82.63 59.51
SGPT-5.8B-msmarco 50.25 78.10 58.81

ST5-XXL has the highest overall average of the three and, for retrieval, is level with MiniLM-L6, the paper's name for all-MiniLM-L6-v2. The authors describe a split in the field between models for retrieval, where a short query is matched to a long document ("asymmetric"), and models for similarity between texts of the same kind ("symmetric"), and observe that models built for one do badly on the other. For choosing a retriever, the overall MTEB average is the wrong column to read. The retrieval column is the right one.

MTEB has since grown. Enevoldsen and colleagues (2025) describe a community expansion to more than 500 tasks in over 250 languages, from which named benchmarks are assembled. One of these is a revised English benchmark, MTEB(eng, v2), which its authors intend "as a zero-shot benchmark, excluding tasks like MS MARCO ... and Natural Questions ..., which are frequently used in fine-tuning". As of November 2026 the benchmark's documentation lists 41 tasks under MTEB(eng, v2), 10 of them retrieval tasks, and recommends it over the original English benchmark, which remains available. As of November 2026 the documentation also lists a retrieval-only benchmark, RTEB, marked as beta, drawn from legal, financial, code and health domains, some of whose datasets are not public.

What a benchmark score does not measure

Five things separate a public score from the answer a practitioner needs. The first three are properties of the benchmarks; the last two are properties of any average.

  1. The judgments favor the systems that built the pool. Relevance is judged on documents that some system retrieved, and unjudged documents count as not relevant (relevance judgments explains pooling). The BEIR authors found that many of their datasets were pooled with lexical retrievers and measured the effect on TREC-COVID: 6.4% of BM25's top-10 results had never been judged, against 31.8% for one dense retriever, and judging the missing documents raised the dense retrievers' scores. This site's topical judgments are an extreme case, measured below: their pool is the top 20 of a single lexical ranker.
  2. The test sets are public. A model can be trained on a benchmark's training splits, or on data that overlaps its test documents, and its score then measures something closer to memory than to transfer. That is the reason the authors of MTEB(eng, v2) give for removing the datasets most used in fine-tuning, and datasets that are not public, such as some of RTEB's, are another way to guard against it. A score is comparable between two models only if both stood in the same relation to the benchmark's data, which a reader of a leaderboard usually cannot check.
  3. The domains are the benchmark's. BEIR's 18 datasets are a broad sample and are still a sample: no aeronautics abstracts, no personal sites. Averages over them say how a model does on text like theirs.
  4. An average hides the spread. In BEIR's own table the same dense retriever is 34% below BM25 on one dataset and 8% above on another. The average says which model is better on average and nothing about a particular collection.
  5. Quality is one of several costs. A leaderboard row has one effectiveness number. It does not show the time to embed a query on the machine that will serve it, the memory the vectors take, the longest input the model reads, or the license. For the unit's encoder the input limit is 256 word pieces, and 59.7% of this site's passages are longer. Whether that matters has to be measured too: dense passage retrieval reports that reading 512 word pieces instead changed nothing measurable on this site.

Example (a common misreading). "The model at the top of the leaderboard is the best model for my corpus." The leaderboard ranks models by an average over other people's collections, with judgments that may favor some kinds of system, on test sets some models may have seen. It is the right place to find candidates. The decision needs the candidates run on one's own documents and queries, which the rest of this article does.

Two models, as of November 2026

The comparison uses two models small enough to embed this site on a laptop processor. Both are pinned by revision in the script and were read at those revisions on 17 November 2026.

A: all-MiniLM-L6-v2 B: mxbai-embed-xsmall-v1
Publisher sentence-transformers Mixedbread
Revision pinned 1110a243fdf4 e6ac24e5d6ef
Parameters 22.7 million 24.1 million
Transformer layers 6 6
Dimensions 384 384
Prefix on queries none none
License on the card Apache 2.0 Apache 2.0

As of November 2026, the card of model A describes it as "a sentence and short paragraph encoder" trained on more than a billion sentence pairs, with input beyond 256 word pieces truncated. As of November 2026, the card of model B says it is "built upon" model A, lists "Optimized for retrieval tasks" and "4096 context support" among its properties, and says it supports Matryoshka truncation, of which more below. To keep the comparison even, both are run with the same input limit of 256 word pieces, although model B accepts longer input, which gives up one of its stated advantages. Model B's weights are shipped in half precision and are run here in single precision. Widening a half-precision number to single precision is exact, and computing in single precision only removes rounding, so this is not expected to change the comparison; the model was not also run in half precision to confirm it.

There is a public number for the pair. The visible text of model B's card gives no score, but as of November 2026 the card's metadata at the pinned revision lists benchmark results, among them nDCG@10 for the fifteen retrieval datasets of the first MTEB benchmark. Their mean is 42.80. The MTEB paper's figure for model A over its fifteen retrieval datasets is 41.95, and model A's own card lists no scores. If the two sets of fifteen are the same, as their names indicate, the public numbers put model B ahead by 0.85 on the benchmark's scale of 0 to 100, that is, by less than 0.01 in nDCG@10. A reader of model B's card and the MTEB paper would therefore expect a small difference in model B's favor, and the experiment below can be read as a test of that expectation on two collections neither benchmark contains.

The parameter and layer counts are read from the loaded models by the script. The two models have the same depth and width and, at 256 word pieces, should cost about the same to run; their running times were not measured. As of November 2026 a third candidate with twelve layers, bge-small-en-v1.5, was planned for this comparison and left out, since embedding the site with it on the shared machine these experiments ran on would have taken hours. That is an instance of the fifth point above: the cost of a model is part of the choice.

The comparison on Cranfield

Cranfield's 225 queries, with the gains above, give the first comparison. BM25 with the course's settings is included as the lexical reference. Each cell is a mean over queries with a 95% bootstrap interval (10,000 resamples of the queries, seed 20261113, the site's convention).

System nDCG@10 Recall@10 Recall@100
BM25 0.368 [0.334, 0.403] 0.397 [0.360, 0.436] 0.737 [0.703, 0.772]
A: all-MiniLM-L6-v2 0.382 [0.347, 0.418] 0.416 [0.377, 0.456] 0.772 [0.739, 0.805]
B: mxbai-embed-xsmall-v1 0.382 [0.349, 0.417] 0.416 [0.377, 0.456] 0.777 [0.744, 0.808]

The two encoders are level to three decimals on nDCG@10 and recall@10, and both are a little above BM25. Every interval overlaps every other in its column.

Means with overlapping intervals can still differ reliably, because the same queries are hard for every model and that shared difficulty cancels in a difference. The right analysis is paired: for each query, subtract one model's score from the other's, and ask whether the mean of those differences is distinguishable from zero. Significance testing develops the tests. The script reports the mean difference with its bootstrap interval, the two-sided randomization test (signs of the differences flipped at random 20,000 times), and the counts of queries won and lost.

Comparison Measure Mean difference [95% interval] Randomization $p$ Queries better, worse, equal
B minus A nDCG@10 +0.001 [−0.012, +0.012] 0.930 88, 81, 56
B minus A recall@100 +0.005 [−0.006, +0.015] 0.393 31, 25, 169
A minus BM25 nDCG@10 +0.014 [−0.012, +0.040] 0.316 107, 98, 20
A minus BM25 recall@100 +0.035 [+0.010, +0.061] 0.007 70, 41, 114
B minus BM25 nDCG@10 +0.014 [−0.012, +0.040] 0.298 106, 94, 25
B minus BM25 recall@100 +0.040 [+0.017, +0.063] 0.001 72, 36, 117

Three things can be read from the table. The two encoders cannot be told apart on Cranfield: the difference in nDCG@10 is 0.001 with an interval that straddles zero evenly, and model B is better on 88 queries and worse on 81. They are not the same ranker, since their nDCG@10 differs on 169 of the 225 queries; the differences cancel. Second, the encoders' lead over BM25 in nDCG@10, 0.014, is within the noise. Third, their lead in recall@100 is not: 0.035 and 0.040, with intervals clear of zero. On this collection a dense first stage delivers more of the relevant abstracts to a second stage than BM25 does, without ordering its own top ten any better. That is the distinction between the two measures, showing up in a whole collection as it did for query 185.

The comparison on this site

The site has two query sets, described in how this search works: 200 known-item queries, each with one target page, and 60 topical queries with graded judgments of pooled pages. A page's dense score is that of its best passage. The lexical reference is the site's own engine, BM25F.

System Known items: MRR Target first Target in top 10
BM25F 0.942 [0.916, 0.966] 0.905 1.000
A: all-MiniLM-L6-v2 0.795 [0.749, 0.841] 0.705 0.950
B: mxbai-embed-xsmall-v1 0.803 [0.757, 0.848] 0.720 0.955
System Topical: nDCG@10 Share of its top 10 that is judged nDCG@10 over judged pages only
BM25F 0.753 [0.705, 0.796] 0.978 0.760
A: all-MiniLM-L6-v2 0.626 [0.568, 0.682] 0.610 0.788
B: mxbai-embed-xsmall-v1 0.638 [0.578, 0.695] 0.608 0.798

On the known-item queries the site's lexical engine is far ahead of both encoders, by 0.147 and 0.139 in mean reciprocal rank, with intervals of [0.102, 0.192] and [0.096, 0.183]. Limits of single-vector retrieval examines those queries one by one. Between the two encoders the difference is 0.008, with an interval of [−0.016, +0.032] and a randomization $p$ of 0.518: model B is better on 23 queries and worse on 26.

The topical queries show the pool effect on this site. Scored in the usual way, with unjudged pages counted as not relevant, BM25F leads both encoders by more than 0.11. But only 61% of the pages in the encoders' top tens were ever judged, against 98% for BM25F. The reason is the pool: for each query, the pages that were judged are the top 20 of the site's own BM25F ranker and nothing else (tools/search/eval/pool.tsv), as dense passage retrieval also sets out. The last column removes unjudged pages from each ranking before scoring, so that each system is scored only on pages someone has judged. Every judged page is in BM25F's own top 20, so this column compares the encoders' re-orderings of BM25F's candidates with BM25F's own order of them. It says nothing about pages BM25F did not retrieve, which is where a dense retriever would differ. On that scoring the order reverses: 0.788 and 0.798 for the encoders against 0.760 for BM25F. Neither reversal is reliable (model A minus BM25F is +0.028 with an interval of [−0.018, +0.074], and model B minus BM25F is +0.038 with [−0.006, +0.080]), and neither would show that an encoder retrieves better than BM25F, only that it might re-order BM25F's twenty no worse. Neither scoring is right, either. The first assumes every unjudged page is irrelevant and the second ignores them, and the truth lies wherever the unjudged pages' real grades would put it. The honest statement is that these judgments cannot rank a dense system against the lexical one. They can compare the two encoders with each other with less bias, since the two are judged to about the same extent, though equal shares of judged pages do not make their unjudged pages equally relevant. In nDCG@10 with unjudged pages counted as not relevant, model B minus model A is +0.013 with an interval of [−0.009, +0.034]; on judged pages only it is +0.010 with [−0.007, +0.026].

So on this site, as on Cranfield, the encoder that its card describes, as of November 2026, as optimized for retrieval is not distinguishable from the one it was built on, at an input limit of 256 word pieces. That does not contradict the public numbers, which put it less than a point ahead: a difference of that size is well inside the intervals here, and the section on the number of queries below says how many queries it would take to see it. The measurement that matters for a choice is this one, and it says the choice between these two makes no detectable difference here.

How many queries does a comparison need?

A difference that is real can still fail to show with too few queries. The one reliable difference found on Cranfield is a useful case: model A's recall@100 exceeds BM25's by 0.035 on average, with a standard deviation of the per-query differences of 0.196, so the mean difference is 0.18 standard deviations.

The script draws subsets of the Cranfield queries without replacement, 2,000 subsets of each size (seed 20261117), and applies the paired $t$-test at the 0.05 level to each.

Queries in the subset Share of subsets in which the test is significant
25 0.099
50 0.210
100 0.406
150 0.665
225 (all) 1.000

With 50 queries this difference would have been detected about one time in five. With 100 it is still more likely to be missed than found. The last row is not a probability: with all 225 queries there is one subset, and it is significant. The subsets are drawn from the same 225 queries and are not independent samples of new queries, so the table describes how this particular comparison behaves at smaller sizes.

The normal approximation gives the same order of magnitude without simulation. If the per-query differences have mean $\bar{d}$ and standard deviation $s$, a two-sided test at level 0.05 detects the difference with probability 0.8 when the number of queries is about

$$n \approx \left(\frac{(1.960 + 0.842)\,s}{\bar{d}}\right)^2,$$

where 1.960 and 0.842 are the standard normal points for 0.025 and 0.2. Significance testing uses the same formula in its section on the number of topics, and power and sample size for a test of a mean derives it. For the recall difference above it gives 244 queries. For model A's lead over BM25 in nDCG@10, a mean of 0.014 against a standard deviation of 0.202, it gives 1,730, far more than Cranfield has, so a lead of that size could not have been confirmed here whether or not it is real. For the two encoders, whose nDCG@10 differences have a standard deviation of 0.092, a true difference of 0.01 would need about 660 queries to detect. The 225 queries and 200 known items used here can rule out a large difference between the two encoders and cannot rule out a small one.

Shorter vectors

Both models produce 384 numbers per passage. Storage and search time are proportional to the dimension, so a shorter vector that retrieved nearly as well would be worth having. There are four ways to get one, and they differ in what they need.

Keep the first $m'$ coordinates. Truncate each vector and rescale it to unit length. This needs nothing, and for an ordinary model it has no reason to work well: training spread the information over all coordinates in no particular order, so the first 64 are an arbitrary sixth of it.

Project onto principal directions. Find the directions in which the collection's vectors vary most, by the singular value decomposition that latent semantic indexing uses, and keep the coordinates along the first $m'$. This needs the collection's vectors once, and a matrix to apply to every query.

Project at random. There are two versions. One multiplies by a random Gaussian matrix, the map analyzed in limits of single-vector retrieval, which needs nothing from the data and comes with the guarantees given there. The other rotates the space at random and then keeps the first $m'$ coordinates, a random orthogonal projection. The second distorts no lengths before it cuts, and is the fair comparison for truncation: it is truncation in a basis chosen at random.

Train the model so that prefixes work. This is Matryoshka representation learning, named after the nested dolls (Kusupati and colleagues, 2022). The idea is to apply the training loss not only to the whole vector but also to each of a set of prefixes of it. For a set $\mathcal{M}$ of prefix lengths, such as $\{64, 128, 256, 384\}$, and a loss $\mathcal{L}$ that takes embeddings, the Matryoshka loss is

$$\mathcal{L}_{\text{MRL}} = \sum_{m' \in \mathcal{M}} c_{m'}\,\mathcal{L}\!\left(\vec{z}_{1:m'}\right),$$

where $\vec{z}_{1:m'}$ is the first $m'$ coordinates of each embedding and the weights $c_{m'}$ are all 1 in the authors' experiments. The authors state it for classification, with a separate linear classifier on each prefix (their equation 1), and note that for contrastive learning it "is applied to both the embeddings that are being contrasted with each other", with normalization handled separately for each prefix length. The gradient is the sum of the gradients of the individual losses, so nothing new needs deriving: the contrastive gradient of contrastive training for retrieval applies to each prefix, and coordinates near the front receive gradient from every term of the sum while coordinates near the end receive it from one. That is the intended mechanism by which the front of the vector comes to carry the most. The authors choose prefix lengths by repeated halving, so there are about $\log_2 m$ of them, and report that prefix lengths between the trained ones also work. Their experiments are mostly on images; the headline figure of their abstract is a vector up to 14 times smaller at the same ImageNet classification accuracy.

As of November 2026, model B's card says it supports Matryoshka truncation and gives one figure: "A 33% reduction in vector size still leaves 96.2% of model performance". Model A makes no such claim. That makes the pair a natural experiment. The tables give Cranfield nDCG@10 and the site's known-item mean reciprocal rank with the vectors cut to $m'$ dimensions. The principal directions are fitted on each collection's own passage vectors of model A, after subtracting their mean. The two random projections are each run with 30 seeds (1 to 30) and reported as a median, with quartiles for the orthogonal one.

$m'$ A, first coordinates B, first coordinates A, principal components A, random orthogonal A, random Gaussian
16 0.120 0.124 0.246 0.125 [0.114, 0.134] 0.124
32 0.200 0.203 0.307 0.208 [0.198, 0.218] 0.204
64 0.283 0.285 0.337 0.286 [0.280, 0.293] 0.273
128 0.346 0.351 0.360 0.338 [0.336, 0.341] 0.323
256 0.368 0.374 0.364 0.371 [0.368, 0.373] 0.350
384 0.382 0.382 0.364

Cranfield, nDCG@10. The Gaussian column's quartiles at 64, 128 and 256 dimensions are [0.267, 0.283], [0.319, 0.327] and [0.346, 0.355].

$m'$ A, first coordinates B, first coordinates A, principal components A, random orthogonal A, random Gaussian
16 0.163 0.180 0.342 0.202 [0.189, 0.213] 0.186
32 0.369 0.371 0.531 0.414 [0.398, 0.431] 0.393
64 0.577 0.579 0.649 0.611 [0.596, 0.624] 0.578
128 0.717 0.701 0.741 0.716 [0.711, 0.726] 0.697
256 0.782 0.776 0.782 0.775 [0.770, 0.782] 0.745
384 0.795 0.803 0.789

This site, known-item mean reciprocal rank.

Four observations, all for these two collections.

  • Model B's prefixes are no better than model A's. At 256 dimensions, the reduction of a third that model B's card describes, model B keeps 97.8% of its full nDCG@10 on Cranfield and 96.6% of its full reciprocal rank on the site, in line with the card's 96.2%. Model A, with no Matryoshka training claimed, keeps 96.4% and 98.3%. At 64 dimensions both keep about three quarters. The front of model B's vectors does not carry more than its share, either: the first 64 of its 384 coordinates hold 17.1% of the squared length of the site's passage vectors, against 16.6% for model A and 16.7% for an even spread.
  • On Cranfield, truncating either model does about what truncating in a random basis does. Model A's first coordinates score 0.120, 0.200, 0.283, 0.346 and 0.368 at 16 to 256 dimensions, and the random orthogonal projection 0.125, 0.208, 0.286, 0.338 and 0.371: within 0.008 of each other at every size, on either side. For this collection the coordinates of the model are an arbitrary basis. On the site the first coordinates do somewhat worse than a random basis at small sizes (0.163 against 0.202 at 16 dimensions, 0.577 against 0.611 at 64) and the same from 128 up. The Gaussian projection is a different matter. On Cranfield it falls below both from 64 dimensions on (0.273 at 64, where truncation has 0.283 and the orthogonal projection 0.286; 0.350 at 256, against 0.368 and 0.371). On the site it is below the orthogonal projection at every size, but against truncation it is above at 16 and 32 dimensions (0.186 against 0.163, 0.393 against 0.369), level at 64 (0.578 against 0.577) and below only from 128 on (0.697 against 0.717). The gap to the orthogonal projection presumably arises because a Gaussian matrix stretches some directions and shrinks others, where a rotation preserves every length.
  • Principal components are much better at small sizes. At 64 dimensions they score 0.337 on Cranfield where truncation scores 0.283, and at 16 dimensions twice what truncation does. The first 64 principal directions hold 66.5% of the variance of the site's passage vectors. The price is a projection fitted to one collection.
  • At full length the principal-component rotation is not neutral. At 384 dimensions it scores 0.364 on Cranfield, below the 0.382 of the vectors as shipped, because subtracting the mean of the collection's unit vectors before taking cosines changes the ranking. A rotation alone would change nothing. (Sentence and passage encoders reports 0.375 for a related repair, which subtracts the mean of the pooled vectors before they are normalized; the two operations centre different vectors and so give different scores.)

The first observation is a finding about one model at one revision on two small collections, as of November 2026, with input cut to 256 word pieces. It does not test the Matryoshka method, whose authors report clear gains from training with the nested loss, and it does not show that the card is wrong: the card's one figure is reproduced. It shows that the figure does not distinguish this model from an ordinary one here, which a reader of the card could not have known without running both.

Notice that the two solid lines, models A and B cut to their first coordinates, lie on top of each other in both panels. The random orthogonal projection of model A (dash-dotted, with its quartile band) runs with them on Cranfield and a little above them at small sizes on the site, and the Gaussian projection (dotted) falls below them from 64 dimensions on Cranfield; on the site it stays below the orthogonal projection throughout but drops below truncation only from 128 on. The dashed line, principal components fitted to the collection, is well above all of them below 128 dimensions and joins them above.
Notice that the two solid lines, models A and B cut to their first coordinates, lie on top of each other in both panels. The random orthogonal projection of model A (dash-dotted, with its quartile band) runs with them on Cranfield and a little above them at small sizes on the site, and the Gaussian projection (dotted) falls below them from 64 dimensions on Cranfield; on the site it stays below the orthogonal projection throughout but drops below truncation only from 128 on. The dashed line, principal components fitted to the collection, is well above all of them below 128 dimensions and joins them above.

The storage arithmetic is simple and worth doing once. A vector of $m$ single-precision numbers takes $4m$ bytes. This site's 11,264 passages at 384 dimensions take $11{,}264 \times 384 \times 4 = 17.3$ million bytes, and at 128 dimensions 5.8 million. At this size the saving is of no practical interest. For a collection of 10 million passages the same two figures are 15.4 and 5.1 gigabytes. How to search such a collection without comparing the query with every vector is the subject of the next unit, vector search, whose article on product quantization compresses vectors much further than truncation can.

Choosing a model for a collection

The steps the article has taken amount to a procedure.

  1. State the job. If the retriever feeds a re-ranker or a language model, the measure is recall at the number of candidates passed on. If its list is shown to a person, it is nDCG@10 or, for known-item search, the reciprocal rank.
  2. Shortlist from the retrieval column of a public benchmark, and read the cards of the candidates for the things an average omits: the longest input, the training data, any query prefix, the license, the size.
  3. Assemble test queries for the collection itself, with judgments. A few hundred known-item queries are cheap to write and free of pooling bias. Graded topical judgments are worth more and must be pooled over every candidate, or the candidates that did not contribute to the pool will be undercounted.
  4. Run every candidate and a lexical baseline on the same queries, with the candidates' recommended prefixes, and record the cost of each: time per query, bytes per vector.
  5. Compare in pairs, per query, with an interval or a test, and check that the number of queries could have shown a difference of the size that would matter.
  6. Read the failures. Twenty queries on which the winner lost say more about what will go wrong than the third decimal of a mean.

What is proved here, what is cited, and what the experiments show

Proved or derived on this page. Little needs proving: the measures are defined on the pages linked, and the Matryoshka loss is a sum of losses whose gradient is the sum of their gradients. The worked problems carry out the computations exactly.

Cited. The composition of BEIR and MTEB and the statements quoted from their papers, with the table or section named. The definition of the Matryoshka loss and its authors' reported results. The statements about models A and B, including model B's benchmark scores, each from its model card or the card's metadata at the pinned revision, as of November 2026. The statements about MTEB(eng, v2) and RTEB, from the benchmark's documentation as of November 2026.

Shown by the experiments, for these two models, these two collections and these queries. Every number in the comparison tables, the paired tests, the queries-needed simulation and the shortened-vector table. The site's queries were written by a model from the pages' summaries, the only pages judged for the topical queries are the top 20 of the site's BM25F ranker, and both encoders were limited to 256 word pieces, so the comparison ranks the models under those conditions.

Worked problems

The unit has no course assignments, so the problems are small exact cases and one calculation on the site's own numbers; the script recomputes each. Problems 1 and 4 use the gain $2^g - 1$ of the site's judgments, where the query 185 example above used Cranfield's $5 - g$.

Problem 1. A system returns five documents for a query, in order $a, b, c, d, e$. Their relevance grades on a 0 to 3 scale are 3, 0, 2, 0, 1. Two other relevant documents, with grades 3 and 1, were not returned. Compute recall@5, counting any grade above 0 as relevant, and nDCG@5 with gain $2^g - 1$.

There are five relevant documents, of which the list holds three, so recall@5 is $3/5 = 0.6$. The gains of the list are $7, 0, 3, 0, 1$ and the discounts $1/\log_2(r + 1)$ for ranks 1 to 5 are $1$, $0.6309$, $0.5$, $0.4307$, $0.3869$, so

$$\mathrm{DCG@5} = 7 + 0 + 1.5 + 0 + 0.3869 = 8.8869.$$

The ideal list orders all relevant documents by grade, 3, 3, 2, 1, 1, with gains $7, 7, 3, 1, 1$:

$$\mathrm{IDCG@5} = 7 + 4.4165 + 1.5 + 0.4307 + 0.3869 = 13.7340.$$

So nDCG@5 is $8.8869/13.7340 = 0.6471$. The two measures answer different questions here. Recall says three fifths of the relevant documents were found. nDCG is higher than that because the best document found is at the top, and lower than 1 mainly because a grade-3 document is missing.

Problem 2. Two models are compared on six queries. Model Y's nDCG@10 exceeds model X's by $0.12$, $0.05$, $-0.02$, $0.09$, $0.01$ and $0.07$. Carry out the two-sided randomization test exactly.

The mean difference is $0.32/6 = 0.0533$. Under the hypothesis that the two models are interchangeable, each difference is as likely to have had the opposite sign, so there are $2^6 = 64$ equally likely sign assignments. The test counts those whose total is at least as large as the observed 0.32 in absolute value. The largest possible total is $0.36$, with every difference positive. A total of at least 0.32 allows the negative terms to sum to at most 0.02 in absolute size, since each flipped difference of size $x$ lowers the total by $2x$: flipping nothing gives 0.36, flipping only the 0.01 gives 0.34, and flipping only the 0.02 gives 0.32. Flipping both gives 0.30, which is too small, and so does any flip of a larger difference. That is 3 assignments with a total of at least $+0.32$, and by symmetry 3 with a total of at most $-0.32$. The p-value is $6/64 = 0.094$.

Five of six queries favor Y and the test still does not reach 0.05. With six queries the smallest p-value this test can give is $2/64 = 0.031$, and only a clean sweep achieves it.

Problem 3. Compute the storage of this site's passage vectors at 384, 256, 128 and 64 dimensions, and of 10 million passages at each.

Each number takes 4 bytes. For 11,264 passages: $11{,}264 \times 384 \times 4$, which is 17,301,504 bytes (17.30 megabytes), then 11.53, 5.77 and 2.88 megabytes. For 10 million passages: $10^7 \times 384 \times 4 = 15.36$ gigabytes, then 10.24, 5.12 and 2.56. Halving the dimension halves the storage, and, for a search that compares the query with every vector, the time.

Problem 4. A system's top three results for a query are a page judged grade 2, a page nobody judged, and a page judged grade 1. One other page was judged, with grade 0. Compute nDCG@3 with gain $2^g - 1$ (a) counting the unjudged page as grade 0, (b) removing it from the ranking first, and (c) supposing its true grade is 3.

The gains of the judged pages are 3, 1 and 0, and the discounts for ranks 1 to 3 are $1$, $0.6309$ and $0.5$. (a) The ranking's gains are $3, 0, 1$, so $\mathrm{DCG@3} = 3 + 0 + 0.5 = 3.5$. The ideal ordering of the judged pages has gains $3, 1, 0$ and $\mathrm{DCG@3} = 3 + 0.6309 = 3.6309$. So nDCG@3 is $3.5/3.6309 = 0.9639$. (b) Without the unjudged page the ranking is the grade-2 page, the grade-1 page and whatever follows, and the fourth judged page has gain 0, so the gains are $3, 1, 0$, the ideal ones, and nDCG@3 is 1. (c) If the unjudged page has grade 3, its gain is 7. The ranking's gains are $3, 7, 1$ and $\mathrm{DCG@3} = 3 + 4.4165 + 0.5 = 7.9165$. The ideal ordering now starts with that page, gains $7, 3, 1$, with $\mathrm{DCG@3} = 7 + 1.8928 + 0.5 = 9.3928$. So the true nDCG@3 would be $7.9165/9.3928 = 0.8428$.

The three answers are 0.964, 1 and 0.843. Counting the unjudged page as irrelevant charged the system for a result that was in fact its best, and still scored it too high, because the missing judgment also made the ideal ranking too easy to reach. Removing the page scored it higher still. Neither convention recovers the true value, and the direction of the error depends on what the unjudged page is. That is why the comparison of the encoders with BM25F on the topical queries was left undecided above: the only cure for a hole in the judgments is a judgment.

What we have learned

A first-stage retriever is measured by recall at the depth its results are passed on, or by nDCG@10 when its list is the final answer, and the two can rank models differently. BEIR measures transfer to 18 collections unlike the training data, and MTEB sets retrieval beside seven other uses of embeddings, where its overall average mixes tasks that favor different models. A public score tells which models to try. It cannot tell which is best for a particular collection, because of pooled judgments, public test sets, domain, averaging and cost. On Cranfield and on this site, two encoders of the same size, one of them described by its card, as of November 2026, as optimized for retrieval and as supporting Matryoshka truncation, could not be told apart by any paired test at full length, and their vectors cut to the same length scored within 0.02 of each other at every size tried, on both collections (no paired test was run on the shortened vectors). Both delivered more of Cranfield's relevant abstracts in their top 100 than BM25 without ranking their top ten better, and on this site's topical queries the pooled judgments could not rank either against the lexical engine, since four in ten of their results had never been judged. Vectors of either model lose about a quarter of their effectiveness when cut to 64 coordinates, and a projection onto the collection's principal directions loses much less.

This is the last article of the unit. Its vectors, 1,400 for Cranfield and 11,264 for this site, were searched by comparing the query with every one of them, which is exact and, at this size, fast. The next unit, vector search, is about collections too large for that, and begins with nearest neighbor search in high dimensions.

References

  • Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., and others (2025). MMTEB: Massive multilingual text embedding benchmark. In Proceedings of ICLR 2025. arXiv:2502.13595. record
  • Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., and Farhadi, A. (2022). Matryoshka representation learning. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 30233–30249. arXiv:2205.13147. record
  • Mixedbread (n.d.). mixedbread-ai/mxbai-embed-xsmall-v1. Model card, including its metadata, at revision e6ac24e5d6efb8782b59de1647b3ececb4ece94e. https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1 (accessed 17 November 2026). record
  • MTEB documentation (n.d.). Available benchmarks: the entries MTEB(eng, v1), MTEB(eng, v2) and RTEB(beta). https://docs.mteb.org/overview/available_benchmarks/ (accessed 17 November 2026). record
  • Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. (2023). MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2014–2037. record · 3 pages
  • sentence-transformers (n.d.). sentence-transformers/all-MiniLM-L6-v2. Model card and train_script.py at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 (accessed 17 November 2026). record · 5 pages
  • Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track. arXiv:2104.08663; read in version 4 of the arXiv paper, whose Table 2 agrees with the published version. record · 7 pages