Cranfield query 185 is experimental studies on panel flutter. A transformer encoder reads it as eight tokens and returns eight vectors, one per token, each a list of 384 numbers. A retrieval system that compares vectors needs one vector for the query and one for each of the collection's 1,400 abstracts, not eight for the query and some two hundred for each abstract. How do the token vectors of a text become a single vector, and is the result any good for retrieval? They are combined by a fixed rule called pooling, most often a plain average. On the evidence below, whether the result is good depends mostly on how the network was trained: on Cranfield, the same six-layer network scores 0.034 in nDCG@10 before it has been trained on pairs of related texts and 0.382 after.
An encoder is a function $E$ that maps a text to a vector in $\mathbb{R}^m$, called the text's embedding. A bi-encoder is a retrieval model that encodes the query and the passage separately, $\vec{q} = E_Q(q)$ and $\vec{p} = E_P(p)$, and scores the pair by a simple function of the two vectors, a dot product or a cosine. The two encoders may be one network, in which case we write $E$ for both. This article is the unit's home for these terms; the later articles of the unit build on it.
Here is the route. First we look at what a transformer encoder emits for query 185, token by token. Then we define the pooling rules and compute the query's vector by hand. Next comes the bi-encoder and the count of encoder passes that makes it the affordable design for first-stage retrieval. Then we measure three networks as retrievers on Cranfield, see how two of them fail, and read how Sentence-BERT's siamese training repairs the failure. Finally we look at what one vector does with a long text. The article assumes the transformer as the library builds it in Building Self-Attention Step by Step and Assembling the Transformer Block, and the cosine of the course. The numbers are computed by tools/experiments/dense/sentence-and-passage-encoders.py, which is deterministic.
What an encoder emits
The library's Language Models from Scratch builds a transformer that predicts the next character. Its attention is masked so that each position sees only earlier positions, because at generation time the later ones do not exist. An encoder of the BERT family (Devlin et al., 2019) is the same stack of blocks with that mask removed: every position attends to every other position, in both directions. The blocks are the ones of the walkthrough, multi-head attention followed by a feedforward layer, with residual connections and layer normalization.
Three details of the input matter here.
- Word pieces. The text is cut into units from a fixed vocabulary of about 30,000 word pieces (Devlin et al., 2019, §3), the subword scheme described in the course's tokenization article. A common word is one piece; a rare word is several.
- Two special tokens. Every input begins with a classification token, written
[CLS], and ends with a separator,[SEP]. - A length limit. The network has a learned vector for each position up to a fixed maximum, 512 in the configuration files of the three networks used below, and cannot read past it. A model may be set to read fewer.
For a text of $L$ tokens the encoder returns $L$ token vectors $\vec{h}_1, \dots, \vec{h}_L$, the outputs of its last block, each in $\mathbb{R}^m$. They are contextual: the vector at a position depends on every token of the text, not only on the token at that position.
The encoder used throughout this unit is sentence-transformers/all-MiniLM-L6-v2, pinned at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. It has six blocks and $m = 384$. Query 185 becomes these eight tokens:
| Position | Token | Length $\lvert \vec{h}_i \rvert$ | Cosine with the mean of the eight | Cosine with the vector of abstract 856 |
|---|---|---|---|---|
| 1 | [CLS] |
7.98 | 0.575 | 0.446 |
| 2 | experimental | 4.04 | 0.750 | 0.540 |
| 3 | studies | 3.68 | 0.735 | 0.539 |
| 4 | on | 7.97 | 0.908 | 0.691 |
| 5 | panel | 11.86 | 0.660 | 0.513 |
| 6 | flutter | 12.31 | 0.768 | 0.556 |
| 7 | . | 8.92 | 0.932 | 0.690 |
| 8 | [SEP] |
10.51 | 0.927 | 0.703 |
Abstract 856, some experimental studies of panel flutter at mach 1.3, is judged relevant to the query. The last column shows that no single token vector is a good stand-in for the whole query: each has a cosine between 0.446 and 0.703 with that abstract's vector, and the query's pooled vector, computed in the next section, reaches 0.751. The table also shows that the vectors at the function word on, the period and [SEP] point closer to the mean than the vectors at panel and flutter do. In a contextual encoder the vector at a position is not a vector "of that word". It is whatever the network has written there, and in this trained network even the punctuation positions hold vectors close to that of the whole text.
Pooling: from $L$ vectors to one
Pooling is a fixed rule that turns the $L$ token vectors into one vector of the same dimension. Three rules are in use.
- Mean pooling averages the token vectors:
$$\vec{e} = \frac{1}{L} \sum_{i=1}^{L} \vec{h}_i .$$
- [CLS] pooling takes the vector at the first position, $\vec{e} = \vec{h}_1$. BERT was designed with this use in mind: its authors call the final vector at
[CLS]"the aggregate sequence representation for classification tasks" (Devlin et al., 2019, §3). - Max pooling takes, in each of the $m$ coordinates separately, the largest value over the $L$ positions.
Whatever the rule, most retrieval encoders then divide the result by its length, so that the embedding is a unit vector, $\vec{e} / \lvert \vec{e} \rvert$, and a dot product of two embeddings is their cosine. Similarity Measures for Embeddings works out what that normalization keeps and discards. The pinned model is three modules in a row: the transformer, mean pooling, and normalization.
Query 185 by hand. The script averages the eight token vectors of the table. The mean vector has length 6.61, and dividing by that length gives the query's embedding. It agrees with the vector the model's own pipeline returns to within $1.6 \times 10^{-8}$ in every coordinate, so nothing else happens between the transformer and the output. The cosine of this embedding with the embedding of abstract 856 is 0.751.
The mean is shorter than its parts: the eight token vectors have a mean length of 8.41, and their average has length 6.61. The shrinkage measures how much the token vectors disagree, and the relation is exact. Expanding the squared length of a sum,
$$\lvert \vec{e} \rvert^2 = \frac{1}{L^2} \sum_{i=1}^{L} \sum_{j=1}^{L} \vec{h}_i \cdot \vec{h}_j ,$$
so the squared length of the mean is the average of all $L^2$ dot products between token vectors, each vector with itself included. The script checks this identity on query 185 to nine decimal places. If the token vectors all pointed the same way the mean would be as long as their average length; if they were mutually orthogonal and of equal length it would shrink by a factor of $\sqrt{L}$. Here the mean cosine between two different token vectors of the query is 0.570, between the two extremes. The identity involves the vectors' lengths as well as their directions; worked problem 2 isolates the part due to direction by taking unit vectors, and turns it into a formula.
The padding mask. Encoders process texts in batches, and a batch is a rectangular array, so shorter texts are padded with a filler token up to the length of the longest. The network still emits a vector at each padded position. Mean pooling has to leave those out. With a mask $a_i$ that is 1 at real tokens and 0 at padding, the mean that implementations compute is
$$\vec{e} = \frac{\sum_i a_i \vec{h}_i}{\sum_i a_i} .$$
Forgetting the mask makes a text's embedding depend on which other texts share its batch. Worked problem 1 shows the size of the error on a small exact case.
Bi-encoders and cross-encoders
There are two ways to use a transformer to score a query against a passage.
A cross-encoder feeds the two texts to the network as one input, the query, a separator, then the passage, and reads a score from the output. Every query token attends to every passage token in every block. Lin, Nogueira and Yates (2021, §3) present it as the standard way of ranking with BERT, and it is the subject of the unit on neural ranking. Its cost is one pass through the network for each query-passage pair, at query time, because the output for a pair cannot be computed before the query is known.
A bi-encoder passes each text through the network alone. The passage vectors do not depend on the query, so they are computed once, when the collection is indexed, and stored. At query time the system encodes the query, one pass, and compares its vector with the stored ones. The term bi-encoder was introduced by Humeau et al. (2020), as Lin, Nogueira and Yates (2021, §5.4) record; Humeau and colleagues set Bi-encoders beside Cross-encoders as the two common designs (their §1), and cross-encoder is the name the same survey uses for the contrasting design.
The difference in cost is a matter of counting passes. Reimers and Gurevych (2019) give the count that made the case. To find the most similar pair among $n = 10{,}000$ sentences with a cross-encoder, every pair must be scored: $n(n-1)/2 = 49{,}995{,}000$ passes, which they report took about 65 hours on the graphics processor they used. With a bi-encoder the same task needs 10,000 passes to embed the sentences, about 5 seconds in their measurement, and then cosines between stored vectors.
The same count for this unit's two collections:
| Collection | Queries | Texts | Cross-encoder passes (every pair) | Bi-encoder passes |
|---|---|---|---|---|
| Cranfield | 225 | 1,400 abstracts | 315,000 | 1,625 |
| This site | 60 | 11,264 passages | 675,840 | 11,324 |
For the site, scoring every passage against every query with a cross-encoder would take 59.7 times as many passes as embedding everything once. The ratio grows with every further query, since each new query costs the cross-encoder another 11,264 passes and the bi-encoder one. This is why a cross-encoder is used only to re-rank a short list that something cheaper has retrieved, and why the first stage, if it is neural, is a bi-encoder.
What the bi-encoder gives up is the interaction. Its passage vector must be written without knowing what will be asked, and its score is a function of two vectors and nothing else. Limits of Single-Vector Retrieval is about what that costs.
The pinned model uses one network for both sides, $E_Q = E_P = E$. Its score is then symmetric, $s(q, p) = s(p, q)$: it cannot tell a question from its answer by role, only by content. The retriever of Dense Passage Retrieval uses two separately trained networks.
Three networks as retrievers
Pooling defines a vector for any text from any encoder. The remaining question of this article is whether those vectors are any use, and the experiment answers it for three networks.
- Trained: the pinned model. Its model card says it was made from the checkpoint below by training on about 1.17 billion pairs of related texts with a contrastive objective, the subject of the next article.
- MiniLM, untrained for sentences:
nreimers/MiniLM-L6-H384-uncasedat revision3276f0fac9d818781d7a1327b3ff818fc4e643c0, the checkpoint the pinned model's card names as its starting point. It has the same six blocks and 384 dimensions; its own card describes it as the 12-block MiniLM with every second block kept. MiniLM was made by distilling a larger pretrained transformer, training the small network to reproduce the self-attention behavior of the large one's last block (Wang et al., 2020). Nothing in that training compares one text with another. - BERT:
google-bert/bert-base-uncasedat revision86b5e0934494bd15c9632b12f734a8a67f723594, the original 12-block, 768-dimensional model, pretrained on masked-word prediction and next-sentence prediction (Devlin et al., 2019) and nothing else.
Each network embeds the 1,400 Cranfield abstracts and the 225 queries, truncated at 256 word pieces, under each pooling rule. Abstracts are ranked by cosine. A fourth rule is included as a control: the mean of the network's input vectors, the word-piece and position vectors before any attention block has run, which is a bag of static word-piece vectors. BM25 on the same collection, from the course's comparison, scores 0.368 in nDCG@10 and 0.737 in recall@100.
| Network | Pooling | nDCG@10 | Recall@100 |
|---|---|---|---|
| Trained | mean | 0.382 | 0.772 |
| Trained | [CLS] | 0.316 | 0.695 |
| Trained | max | 0.286 | 0.658 |
| Trained | input vectors, mean | 0.115 | 0.383 |
| MiniLM | mean | 0.034 | 0.168 |
| MiniLM | [CLS] | 0.017 | 0.088 |
| MiniLM | max | 0.031 | 0.126 |
| MiniLM | input vectors, mean | 0.053 | 0.267 |
| BERT | mean | 0.110 | 0.409 |
| BERT | [CLS] | 0.015 | 0.114 |
| BERT | max | 0.093 | 0.353 |
| BERT | input vectors, mean | 0.117 | 0.369 |
Three things stand out.
- Training is what matters. With mean pooling the trained network scores 0.382, a little above BM25. The same architecture before sentence training scores 0.034. Over the 225 queries the mean difference is 0.348 ($t = 19.7$ on 224 degrees of freedom), and the trained network is ahead on 193 queries and behind on 3. BERT, with twice the blocks and twice the dimensions, scores 0.110 under mean pooling, 0.258 below BM25 ($t = 15.6$).
- The [CLS] vector of an untrained network is the worst choice. It scores 0.017 for MiniLM and 0.015 for BERT, below every other row of the table.
- On this evidence, the attention blocks add nothing a cosine can use until they are trained to. For MiniLM the mean of the input vectors, which involves no attention at all, does better than the mean of the output vectors: 0.053 against 0.034 ($t = 3.44$). For BERT the two cannot be told apart: 0.117 against 0.110 ($t = 0.67$, $p = 0.50$). For the trained network the order is reversed, 0.115 against 0.382, and there the blocks are doing the work.
How the untrained vectors fail
Here is the finding in one sentence: the untrained networks give every text nearly the same vector, and what does differ between texts is barely arranged by subject, so that removing the shared part does not help.
The script measures this directly. It computes the cosine between every two of the 1,400 abstracts, 979,300 pairs, and averages. It also averages the cosine between a query and each abstract judged relevant to it (1,612 pairs) and between a query and every other abstract.
| Network, pooling | Mean cosine of two abstracts | Query and relevant abstract | Query and other abstract | Standard deviation of the last | Gap in standard deviations |
|---|---|---|---|---|---|
| Trained, mean | 0.288 | 0.485 | 0.217 | 0.127 | 2.12 |
| Trained, [CLS] | 0.768 | 0.806 | 0.699 | 0.061 | 1.77 |
| MiniLM, mean | 0.948 | 0.835 | 0.832 | 0.054 | 0.05 |
| MiniLM, [CLS] | 0.998 | 0.994 | 0.994 | 0.002 | −0.05 |
| BERT, mean | 0.881 | 0.788 | 0.757 | 0.065 | 0.46 |
| BERT, [CLS] | 0.809 | 0.651 | 0.641 | 0.092 | 0.11 |
Two abstracts drawn from MiniLM's mean-pooled vectors have a cosine of 0.948 on average. For its [CLS] vectors the average is 0.998, and the least similar pair in the whole collection has a cosine of 0.970. BERT's mean-pooled vectors average 0.881. The vectors occupy a narrow cone. The last column is the quantity that matters for ranking: the difference between a query's mean cosine with its relevant abstracts and with the others, in units of the spread of the others. For the trained model with mean pooling a relevant abstract sits 2.12 standard deviations above the rest. For untrained MiniLM it sits 0.05 above, and for BERT 0.46 above: relevance is nearly invisible.
Example (a high cosine means little by itself). It is tempting to read a cosine of 0.99 as "nearly the same meaning". Under untrained MiniLM's [CLS] pooling, query 185 and almost any abstract have a cosine near 0.99, whatever the abstract is about. A cosine is informative only against the spread of cosines the same encoder gives unrelated texts. Under the trained model's mean pooling, 0.751 for query 185 and abstract 856 is a strong match, because unrelated pairs average 0.217.
This crowding is documented for BERT itself. Ethayarajh (2019) measured the cosines between contextual token vectors of BERT, ELMo and GPT-2 and found that they are "not isotropic" in any layer of the contextualizing model: they occupy a narrow cone, so that two random words have a cosine well above zero (near 1, he reports, in the last layer of GPT-2). Li et al. (2020) argued that sentence vectors averaged from BERT's token vectors inherit the problem, and showed that mapping the vectors to a smooth, isotropic distribution improved their scores on sentence-similarity tasks without any labeled data.
The simplest repair one could try is to subtract the collection's mean vector from every vector before taking cosines, which removes the shared direction. The script does this to the pooled vectors, before they are normalized, subtracting the mean of the abstracts' pooled vectors from the abstracts and the queries alike. (Subtracting instead the mean of the normalized vectors gives a different score for the trained model, 0.364, reported in evaluating embedding models; the two operations centre different vectors.) For MiniLM with mean pooling, nDCG@10 goes from 0.034 to 0.033, a change within noise ($t = -0.56$ on the 225 paired differences); for the trained model it goes from 0.382 to 0.375. For BERT it goes from 0.110 to 0.109 ($t = -0.27$). On this collection, then, the shared direction is a symptom and not the whole fault. With it removed, what remains still does not order abstracts by relevance. Only this one repair was tried; stronger transformations of the vectors, such as the mapping of Li et al., were not. The plain reading, which the next paragraph spells out, is that nothing in the networks' training asked for such an order.
That reading starts from the pretraining objectives. Masked-word prediction rewards token vectors from which the hidden word can be predicted. BERT's next-sentence task does put a sentence-level demand on the [CLS] vector, but a weak one, and the authors say of that vector: "The vector C is not a meaningful sentence representation without fine-tuning" (Devlin et al., 2019, §3.1, footnote 6). Neither objective, and nothing in MiniLM's distillation, rewards two texts on one subject for having nearby pooled vectors.
Reimers and Gurevych (2019, Table 1) measured the same failure on sentence similarity. They scored sentence pairs by the cosine of their vectors and reported the rank correlation with human similarity judgments, times 100, averaged over seven test sets. Averaged GloVe word vectors, which involve no transformer at all, scored 61.32. BERT's averaged token vectors scored 54.81 and its [CLS] vector 29.19. Their conclusion was that the common practice of pooling an untrained BERT "yields rather bad sentence embeddings". The control rows of our table agree in kind: a mean of static input vectors does better than the mean of the contextual output vectors for MiniLM, and no worse for BERT.
Siamese training: Sentence-BERT
If the pretrained network does not put related texts near each other, it has to be trained to. Sentence-BERT (Reimers and Gurevych, 2019) is the method that established how.
Here is the idea. Take a pair of texts with a known relation. Pass each through the same network, pool each, and compute a loss from the two pooled vectors and the label. Then update the network's weights to reduce the loss. Because the loss is a function of the pooled vectors, the gradient pushes on exactly the quantity that will later be compared. An arrangement in which two inputs pass through two copies of one network with shared weights is called a siamese network.
The gradient with shared weights. Let the network have weights $\theta$, let the two texts be $a$ and $b$, and let $\vec{u} = E_\theta(a)$ and $\vec{v} = E_\theta(b)$. The loss $\mathcal{L}(\vec{u}, \vec{v})$ depends on $\theta$ through both vectors. By the chain rule, for each weight $\theta_k$,
$$\frac{\partial \mathcal{L}}{\partial \theta_k} = \frac{\partial \mathcal{L}}{\partial \vec{u}} \cdot \frac{\partial \vec{u}}{\partial \theta_k} + \frac{\partial \mathcal{L}}{\partial \vec{v}} \cdot \frac{\partial \vec{v}}{\partial \theta_k} .$$
So the update is the sum of two contributions, one from each pass. There are not two networks to keep in step. There is one, and it receives the gradient from both of its uses.
Reimers and Gurevych (2019, §3) trained with three objectives, chosen by the kind of label available. Write $\vec{u}$ and $\vec{v}$ for the pooled vectors of a pair, of dimension $n$ in their notation.
- Classification. For pairs labeled with one of $k$ classes, the two vectors and their element-wise absolute difference are concatenated and passed to a softmax classifier, $o = \mathrm{softmax}(W_t(\vec{u}, \vec{v}, \lvert \vec{u} - \vec{v} \rvert))$ with a trainable matrix $W_t \in \mathbb{R}^{3n \times k}$, trained with cross-entropy.
- Regression. For pairs labeled with a similarity score, the cosine of $\vec{u}$ and $\vec{v}$ is trained to match the score with a mean squared error loss.
- Triplet. For an anchor text $a$, a positive text $p$ and a negative text $n$, with vectors $\vec{s}_a$, $\vec{s}_p$, $\vec{s}_n$, the loss is $\max(\lVert \vec{s}_a - \vec{s}_p \rVert - \lVert \vec{s}_a - \vec{s}_n \rVert + \epsilon, 0)$, where $\epsilon$ is a margin. It is zero once the positive is closer to the anchor than the negative by the margin; they used Euclidean distance and a margin of 1. Worked problem 4 computes one.
Their main model was trained with the classification objective on natural language inference data: the SNLI collection, 570,000 sentence pairs labeled contradiction, entailment or neutral, and MultiNLI, 430,000 pairs. Training ran for one epoch with a batch size of 16 and the Adam optimizer at a learning rate of $2 \times 10^{-5}$, with mean pooling as the default (their §3.1). The classifier $W_t$ is discarded afterward. Only the encoder is kept, and its vectors are compared by cosine.
On the seven sentence-similarity sets where averaged BERT vectors scored 54.81, this model, SBERT-NLI-base, scored 74.89 (their Table 1). The network and the pooling rule are the same in both rows. Only the training differs.
Two findings from their ablation (their Table 6, on the development set of the STS benchmark) bear on the choices made in this unit. With the classification objective, the pooling rule mattered little: mean 80.78, max 79.07, [CLS] 79.80. And the element-wise difference $\lvert \vec{u} - \vec{v} \rvert$ was the most important input to the classifier: with $(\vec{u}, \vec{v})$ alone the score was 66.04, and with the difference added, 80.78. The authors' explanation is that the element-wise difference "measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart". A classifier that sees only the two vectors side by side can be satisfied without the vectors being near each other.
Sentence-BERT was trained and tested on sentence similarity, not on retrieval. The pinned model keeps its design, one shared network with mean pooling, and changes the objective to a contrastive one over many pairs at once, the kind of objective retrieval encoders are trained with: Contrastive Training for Retrieval derives that loss. Its effect on the pooling comparison is visible in our table. The trained model was trained through mean pooling, and its mean-pooled vectors beat its own [CLS] vectors by 0.065 in nDCG@10 ($t = 5.58$). For a trained network, the pooling to use is the one it was trained through; here that choice is worth 0.065.
What one vector does with a long text
Mean pooling gives every token an equal share of the vector. For a long text with several subjects, that has a predictable consequence, and the encoder's length limit adds a second one.
Truncation. The pinned model reads at most 256 word pieces and silently drops the rest. Cranfield queries are short, with a median of 22 word pieces and a maximum of 59. Abstracts have a median of 192, but 413 of the 1,400 are longer than 256 and the longest has 807. For those abstracts the embedding describes the opening and nothing else. Dense Passage Retrieval measures what raising the limit to 512 does.
Averaging over subjects. To see what the vector of a mixed text looks like, the script takes 266 short Cranfield abstracts, of 40 to 120 word pieces each, pairs them into 133 two-abstract texts by joining unrelated abstracts end to end, and embeds each joined text. Three cosines are then averaged over the 133:
- the joined text against the normalized sum of its two abstracts' own vectors: 0.952;
- the joined text against its first abstract, 0.755, and against its second, 0.732;
- the two abstracts against each other: 0.223.
So the embedding of the joined text is close to the average of its parts. It is not equal to it, since the network is not linear and each half is read in the presence of the other, but a cosine of 0.952 says the average describes it well. A vector that is the average of two directions is near neither. Among the joined texts there are 276 cases of a query with a relevant abstract in the pair. The query's cosine with the relevant abstract alone averages 0.501, and its cosine with the joined text averages 0.421; the cosine fell in 235 of the 276 cases. In these cases a passage about two subjects was a weaker match for a query about either one than the relevant half alone.
This is one reason dense retrieval indexes passages, not whole documents. This unit, likewise, embeds the site's pages not whole but as the passages the site's database already holds, cut at the pages' headings. The cost of going the other way, many short passages each stripped of its context, is taken up in Limits of Single-Vector Retrieval.
What is proved here, what is cited, and what the experiment shows
Proved or derived on the page. The identity for the squared length of a mean vector and the formula for equal cosines in worked problem 2; the gradient of a loss through two uses of shared weights; the pass counts for the two designs.
Cited. The architecture and pretraining of BERT (Devlin et al., 2019) and the distillation that made MiniLM (Wang et al., 2020); Sentence-BERT's objectives, training settings and every score quoted from its Tables 1 and 6, and its timings of 65 hours and 5 seconds, which are measurements on its authors' hardware; the anisotropy findings of Ethayarajh (2019) and Li et al. (2020); the origin of the term bi-encoder (Humeau et al., 2020, as recorded by Lin, Nogueira and Yates, 2021). The description of how the pinned model was trained is from its model card.
Shown by the experiment, on Cranfield, with these three checkpoints. That pooled vectors from the two networks not trained on text pairs retrieve far worse than BM25, and MiniLM's almost nothing, and that the network trained on pairs scores slightly above BM25 in nDCG@10, a difference that the article on dense retrieval tests and finds within chance; that the untrained vectors are crowded into a narrow cone; that subtracting the mean vector does not repair them; that the vector of two joined abstracts is close to the average of their vectors. None of these is a general law. A different pretrained network, pooling over a different layer, or a different collection could give different numbers, and the Sentence-BERT results quoted above are for sentence similarity, a different task from the one measured here.
Worked problems
These problems are posed on small exact cases and on the unit's collections, to exercise the definitions of this article; the unit has no course assignments. The script recomputes each number under "Worked problems".
Problem 1 (masked mean pooling). A batch holds two texts. Text A has three tokens with token vectors $(1, 2)$, $(3, 0)$, $(2, 4)$. Text B has two tokens with vectors $(4, 0)$, $(0, 2)$ and is padded to length three; at the padded position the network emits $(5, 5)$. Compute the cosine of the two texts' mean-pooled vectors with the padding mask and without it.
Solution. With the mask, text A's vector is $\frac{1}{3}\big((1,2) + (3,0) + (2,4)\big) = (2, 2)$ and text B's is $\frac{1}{2}\big((4,0) + (0,2)\big) = (2, 1)$. Their cosine is
$$\frac{2 \cdot 2 + 2 \cdot 1}{\sqrt{8}\,\sqrt{5}} = \frac{6}{\sqrt{40}} = \frac{3}{\sqrt{10}} = 0.949 .$$
Without the mask, text B's three positions are averaged: $\frac{1}{3}\big((4,0) + (0,2) + (5,5)\big) = (3, \tfrac{7}{3})$. Its cosine with $(2, 2)$ is
$$\frac{6 + \tfrac{14}{3}}{\sqrt{8}\,\sqrt{9 + \tfrac{49}{9}}} = \frac{\tfrac{32}{3}}{\sqrt{8}\,\tfrac{\sqrt{130}}{3}} = \frac{32}{\sqrt{1040}} = \frac{8}{\sqrt{65}} = 0.992 .$$
The unmasked value is wrong, and it is wrong in a way that depends on the batch: had text B been batched with a two-token text, there would have been no padding and its vector would have been $(2, 1)$. The same text must get the same embedding whatever it is batched with, and only the masked mean guarantees that, provided the network's attention also ignores the padded positions, as the attention mask passed with the batch makes it do.
Problem 2 (the length of a mean). Suppose the $L$ token vectors of a text are unit vectors and every two different ones have the same cosine $c$. Find the length of their mean, and evaluate it for $L = 8$ and $L = 256$ with $c = 0.6$ and with $c = 0$.
Solution. By the identity of the pooling section, $\lvert \vec{e} \rvert^2$ is the average of the $L^2$ dot products $\vec{h}_i \cdot \vec{h}_j$. Of these, $L$ are a vector with itself and equal 1, and the other $L^2 - L$ equal $c$. So
$$\lvert \vec{e} \rvert^2 = \frac{L + (L^2 - L)\,c}{L^2} = \frac{1}{L} + \Big(1 - \frac{1}{L}\Big) c, \qquad \lvert \vec{e} \rvert = \sqrt{\frac{1}{L} + \Big(1 - \frac{1}{L}\Big) c} .$$
For $c = 0.6$ this is $\sqrt{0.125 + 0.875 \times 0.6} = \sqrt{0.65} = 0.806$ at $L = 8$ and 0.776 at $L = 256$. For $c = 0$ it is $1/\sqrt{8} = 0.354$ at $L = 8$ and $1/\sqrt{256} = 0.063$ at $L = 256$. As $L$ grows the length tends to $\sqrt{c}$: for long texts the mean's length is set by how much the token vectors agree, and with no agreement at all the mean shrinks toward the zero vector. Normalizing the embedding afterward hides its length, which is one of the things normalization discards.
Problem 3 (the cost of a cross-encoder). A collection has 11,264 passages. How many encoder passes does it take to score every passage for 60 queries with a cross-encoder, and with a bi-encoder whose passage vectors are not yet computed? How many does each further query cost?
Solution. The cross-encoder needs one pass per pair, $60 \times 11{,}264 = 675{,}840$. The bi-encoder needs one pass per text, $11{,}264 + 60 = 11{,}324$, a ratio of 59.7. Each further query costs the cross-encoder 11,264 passes and the bi-encoder one pass, followed by 11,264 dot products of 384 numbers each. The passes are not equally expensive: a cross-encoder pass reads the query and passage together and is somewhat longer than a passage alone. But a dot product costs a few hundred multiplications where a pass through a six-block transformer costs many millions, so the comparison step is negligible beside either. For Reimers and Gurevych's 10,000 sentences the same reasoning gives 49,995,000 pair passes against 10,000, a ratio of 4,999.5.
Problem 4 (a triplet loss). With anchor $\vec{s}_a = (1, 0)$, positive $\vec{s}_p = (0.6, 0.8)$, negative $\vec{s}_n = (0, 1)$, Euclidean distance and margin $\epsilon = 1$, compute the triplet loss. What is the loss if the margin is 0.5?
Solution. The distances are $\lVert \vec{s}_a - \vec{s}_p \rVert = \sqrt{0.4^2 + 0.8^2} = \sqrt{0.8} = 0.894$ and $\lVert \vec{s}_a - \vec{s}_n \rVert = \sqrt{2} = 1.414$. The positive is closer, by 0.520. With $\epsilon = 1$ the loss is $\max(0.894 - 1.414 + 1, 0) = 0.480$: the positive is closer but not by the full margin, so the network is still pushed. With $\epsilon = 0.5$ the loss is $\max(0.894 - 1.414 + 0.5, 0) = 0$, since the gap of 0.520 already exceeds the margin, and this triplet contributes no gradient at all. A triplet loss stops teaching as soon as the triplet is satisfied. The contrastive loss of the next article never reaches exactly zero, but its gradient also shrinks as negatives become easy, which is the starting point of Hard Negatives and Distillation.
What we have learned
A transformer encoder emits one vector per token, and pooling turns them into one vector per text: the mean, the vector at [CLS], or a coordinate-wise maximum, usually followed by normalization. Because a bi-encoder's passage vectors do not depend on the query, they can be computed once and stored, which makes a bi-encoder the affordable design for searching a whole collection. Pooling alone does not make useful vectors. On Cranfield, the pooled vectors of two networks that were never trained on pairs of texts retrieved far worse than BM25, one of them almost nothing. Those networks give every text nearly the same vector, and removing the shared part did not help. Sentence-BERT showed the repair: train one shared network through its pooling layer on pairs of texts with a known relation.
The objective Sentence-BERT used was built for sentence similarity. A retriever must tell one relevant passage from millions of others, and the loss that trains it for that, with its gradient and the reason its batch size matters, is the subject of Contrastive Training for Retrieval.
References
- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, Volume 1, 4171–4186. record · 4 pages
- Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of EMNLP-IJCNLP 2019, 55–65. record
- Humeau, S., Shuster, K., Lachaux, M.-A., and Weston, J. (2020). Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations (ICLR 2020). arXiv:1905.01969. record
- Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. In Proceedings of EMNLP 2020, 9119–9130. record
- Lin, J., Nogueira, R., and Yates, A. (2021). Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures on Human Language Technologies, 14(4). Morgan & Claypool (reissued by Springer, 2022). Preprint: arXiv:2010.06467; section numbers here follow version 3 of the preprint. record · 2 pages
- Reimers, N. (n.d.). nreimers/MiniLM-L6-H384-uncased. Model card at revision
3276f0fac9d818781d7a1327b3ff818fc4e643c0. https://huggingface.co/nreimers/MiniLM-L6-H384-uncased (accessed 17 November 2026). record - Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019, 3982–3992. record · 2 pages
- sentence-transformers (n.d.). sentence-transformers/all-MiniLM-L6-v2. Model card and
train_script.pyat revision1110a243fdf4706b3f48f1d95db1a4f5529b4d41. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 (accessed 17 November 2026). record · 5 pages - Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 5776–5788. arXiv:2002.10957. record