Hard Negatives and Distillation

Where the negative passages of contrastive training should come from, and what replaces a relevance label when the labels are incomplete. The gradient of the contrastive loss is bounded by the loss, so negatives the encoder already separates teach nothing; negatives mined from the index, as in ANCE, keep the loss up, and bring in false negatives; a cross-encoder teacher scores every pair, and the margin mean squared error loss, derived here with its gradient and its invariances, transfers those scores to a bi-encoder. Experiments on Cranfield measure the gradient under random and mined negatives, the false-negative rate under sparse labels, and a toy training run of a linear layer on frozen vectors. Four worked problems compute the losses and gradients by hand.

Cranfield query 185 asks for experimental studies on panel flutter, and abstract 15 is one of its nine relevant abstracts. The unit's encoder gives the pair a cosine of 0.602. To train the encoder on this pair by contrastive training, we must also choose passages that are not relevant, for the encoder to push away. Seven abstracts drawn at random from the other 1,391 have cosines between 0.108 and 0.464 with the query, and against them the contrastive loss is 0.064. The encoder already tells them apart. What does a training step on these seven teach it?

Almost nothing, and this article shows why and what to do about it. A hard negative is a non-relevant passage that the current encoder scores close to, or above, a relevant one. Training on hard negatives teaches the distinctions the encoder still gets wrong. It also runs into the incompleteness of relevance labels: the passages most similar to a query are the ones most likely to be relevant and unlabelled. Distillation answers that by replacing the label with the graded judgment of a stronger model.

Here is the route. First we show that the gradient of the contrastive loss is bounded by the loss itself, so a negative that costs nothing teaches nothing, and measure the gradient on Cranfield. Then we describe mining negatives from the index, as ANCE does it, with the results its authors report. Next come false negatives, measured on Cranfield by hiding labels. Then distillation: the margin mean squared error loss, its gradient, three properties proved from the definition, and what a teacher does to false negatives. A toy training run closes the experiments. The article assumes contrastive training for retrieval, which derives the loss, and dense passage retrieval. The numbers come from tools/experiments/dense/hard-negatives-and-distillation.py, with the unit's pinned encoder sentence-transformers/all-MiniLM-L6-v2 (revision 1110a243fdf4) and, as teacher, the cross-encoder cross-encoder/ms-marco-MiniLM-L6-v2 (revision 233902d25c44).

A loss near zero means a gradient near zero

The contrastive loss for a query $q$, a positive passage $p^+$ and negatives $p^-_1, \dots, p^-_n$ is the negative log probability that a softmax over the $n + 1$ scores assigns to the positive:

$$\mathcal{L} = -\ln \pi^+, \qquad \pi^+ = \frac{e^{s(q, p^+)/\tau}}{e^{s(q, p^+)/\tau} + \sum_{j=1}^{n} e^{s(q, p^-_j)/\tau}},$$

with temperature $\tau$ and, for this unit's encoder, the cosine as the score $s$. Write $\pi_j$ for the softmax probability of the $j$-th negative, so that $\pi^+ + \sum_j \pi_j = 1$. Contrastive training for retrieval derives the gradient with respect to the scores:

$$\frac{\partial \mathcal{L}}{\partial s(q, p^+)} = -\frac{1 - \pi^+}{\tau}, \qquad \frac{\partial \mathcal{L}}{\partial s(q, p^-_j)} = \frac{\pi_j}{\tau}.$$

Each negative is pushed away in proportion to the probability the softmax gives it, and the positive is pulled in by the total of those probabilities.

Back to query 185, with $\tau = 0.05$, the temperature that the training script shipped with the pinned encoder uses by default (a scale of 20).

Negatives for query 185, positive abstract 15 (cosine 0.602) Their cosines with the query Loss $\pi^+$ Total weight on negatives, $1 - \pi^+$
Seven drawn at random (seed 185): abstracts 41, 307, 395, 504, 747, 1243, 1260 0.226, 0.135, 0.108, 0.200, 0.464, 0.257, 0.160 0.064 0.938 0.062
The seven highest-scoring non-relevant abstracts: 658, 864, 878, 390, 627, 948, 391 0.773, 0.760, 0.671, 0.664, 0.662, 0.657, 0.648 4.268 0.014 0.986

Against the random seven, the pull on the positive's score is $0.062/0.05 = 1.24$. Against the mined seven, all of which the encoder scores above the relevant abstract, it is $0.986/0.05 = 19.72$, 16 times as much. Of the random seven, abstract 747 alone carries most of the weight: its cosine of 0.464 is the only one within reach of 0.602.

The table suggests a general fact, which can be proved.

Proposition. Let the scores depend on parameters $\theta$, and suppose every score's gradient has length at most $G$: $\lvert \nabla_\theta s(q, p) \rvert \le G$. Then

$$\lvert \nabla_\theta \mathcal{L} \rvert \;\le\; \frac{2G}{\tau}\,(1 - \pi^+) \;\le\; \frac{2G}{\tau}\,\mathcal{L}.$$

Proof. By the chain rule and the two derivatives above,

$$\nabla_\theta \mathcal{L} = -\frac{1 - \pi^+}{\tau}\,\nabla_\theta s(q, p^+) + \sum_j \frac{\pi_j}{\tau}\,\nabla_\theta s(q, p^-_j) = \frac{1}{\tau}\sum_j \pi_j\left(\nabla_\theta s(q, p^-_j) - \nabla_\theta s(q, p^+)\right),$$

where the second form uses $\sum_j \pi_j = 1 - \pi^+$. Each difference in brackets has length at most $2G$ by the triangle inequality, and the weights $\pi_j$ are positive and sum to $1 - \pi^+$, which gives the first inequality. For the second, $\pi^+ = e^{-\mathcal{L}}$ by the definition of the loss, and $1 - e^{-x} \le x$ for every $x \ge 0$. $\blacksquare$

So the gradient is at most proportional to the loss. A training example whose negatives are already far below the positive has a loss near zero, and a step on it barely moves the encoder, whatever the learning rate. Xiong and colleagues (2021, their equations 11 and 12) make the same point for deep networks by citing an upper bound on per-example gradient norms from the importance-sampling literature, derived there for multilayer perceptrons; the proposition is the elementary version for this loss.

How often does a random draw contain a useful negative?

Rarely, and less often as the collection grows. Call a negative informative for a pair if the encoder scores it at least as high as the positive. For query 185 and abstract 15 there are 9 such abstracts among the 1,391 non-relevant ones. If $n$ negatives are drawn at random without replacement from $M$ candidates of which $h$ are informative, the chance that the draw contains at least one is

$$1 - \binom{M - h}{n} \Big/ \binom{M}{n} \;\approx\; \frac{nh}{M} \quad\text{when } nh \ll M.$$

With $M = 1{,}391$, $h = 9$ and $n = 7$ that is 0.0445, about one draw in 22. Cranfield is tiny. For a collection of 8.8 million passages, the size of MS MARCO, take $n = 127$ negatives (a batch of 128, each query using the other queries' passages) and suppose, for the sake of the arithmetic, that 200 passages are informative. Treating the in-batch negatives as random draws from the collection, which they are only roughly, since they are other queries' positives, the chance is 0.00288: about one training example in 347 sees any informative negative at all. Xiong and colleagues argue the same from two conditions, that the batch is far smaller than the corpus and that only a few negatives are informative, and report it as a measurement: for their models the overlap between in-batch negatives and the 100 highest-scored negatives of the dense retriever was zero.

The gradient, measured

The proposition bounds the gradient; the experiment measures it, with respect to all of the encoder's parameters and not only the scores. For each of 30 seeds the script draws 4 Cranfield queries and, for each, one relevant abstract as the positive. It then forms three sets of seven negatives: seven non-relevant abstracts drawn at random, the seven highest-ranked non-relevant abstracts of BM25, and the seven highest-ranked non-relevant abstracts of the encoder itself. Each set gives one loss and one backward pass through the encoder, 360 passes in all, a scale chosen to fit a laptop processor. The table gives, for each kind of negative, the median over seeds of the median over a seed's 4 queries, with quartiles over seeds in brackets.

Negatives Loss Length of the gradient over all parameters
Random 0.326 [0.122, 1.094] 24.98 [10.78, 49.93]
Mined with BM25 3.844 [2.804, 4.854] 85.70 [79.43, 90.31]
Mined with the encoder 4.579 [3.419, 5.818] 84.39 [79.78, 88.61]

Mined negatives produce a loss more than ten times that of random negatives. For the gradient, the random row's 24.98 is a median of medians of four and is not a typical value of anything, so the 120 draws are better read one by one:

Negatives Gradient length over the 120 draws: median Quartiles Share of draws below 10
Random 18.2 1.6 and 69.2 0.400
Mined with BM25 85.8 73.2 and 98.1 0.008
Mined with the encoder 84.9 73.0 and 95.0 0.000

The random draws are of two kinds. Two in five give a gradient under 10, a quarter give one under 1.6, and another quarter give one above 69, as long as a mined draw's. Which kind a draw is depends on whether it happened to include an abstract the encoder places near the query, as abstract 747 was for query 185. The cached draws show it. For each of the 120 random draws, take the cosine of its closest negative to the query minus the positive's cosine: the rank correlation of the gradient's length with that margin is 0.991 (with the draw's loss, 0.993). In 35 draws some random negative scores above the positive, and their median gradient is 93.7; in the other 85 it is 4.0. Mined negatives give a long gradient almost every time. Within a seed, the ratio of the encoder-mined to the random median has a median of 3.6 over seeds, with quartiles 1.8 and 7.8, but the more faithful summary is the last column: random negatives leave a large share of training examples with little to learn from, and mined negatives leave almost none. BM25's negatives and the encoder's own are equally hard by this measure. The gradient does not vanish for random negatives here, and the proposition does not say it must: it says the gradient can be no larger than a multiple of the loss, and a median loss of 0.33 is not near zero. A likely reason, not tested here, is that Cranfield's 1,400 abstracts are all on one branch of engineering, so that a random abstract is a much closer neighbor than a random passage of a web-scale collection would be.

This is one small pretrained encoder at one point in its life, already trained on over a billion pairs according to its model card, and Cranfield queries rather than training data. Xiong and colleagues report the corresponding measurement during training on MS MARCO (their Figure 4): the gradient norms with in-batch negatives stay close to zero, and those with negatives retrieved by the model are larger "by orders of magnitude".

Notice how few abstracts stand out. The curve is the encoder's cosine for all 1,400 abstracts of query 185, sorted; the nine relevant abstracts are marked, and so are the seven random and the seven mined negatives of the table. Six of the seven random draws land in the long flat part, where the softmax gives them almost no weight, and the seventh, abstract 747, is at rank 38. The mined ones sit among the relevant abstracts at the top, as the inset shows.
Notice how few abstracts stand out. The curve is the encoder's cosine for all 1,400 abstracts of query 185, sorted; the nine relevant abstracts are marked, and so are the seven random and the seven mined negatives of the table. Six of the seven random draws land in the long flat part, where the softmax gives them almost no weight, and the seventh, abstract 747, is at rank 38. The mined ones sit among the relevant abstracts at the top, as the inset shows.

Mining negatives from the index

If random negatives are too easy, harder ones must be searched for, and a retrieval system is a machine for finding passages that look relevant to a query. There are three sources, in the order they were tried.

A lexical ranker. Take the top results of BM25 for the training query and drop the labelled positives. Karpukhin and colleagues (2020) added one such passage per question to the in-batch negatives of the dense passage retriever, and its article gives their ablation: on Natural Questions the top-20 accuracy rose from 70.8 to 77.3 (their Table 3, 31 in-batch negatives without and with one BM25 negative), and a second BM25 negative did not help further. BM25 negatives are passages that share the query's words and are not marked relevant. They are fixed before training and say nothing about what the encoder itself finds confusing.

The encoder itself, once. Encode the collection with a partly trained encoder, retrieve the top passages for each training query, and use the non-relevant ones as negatives for further training. Zhan and colleagues (2021) call these static hard negatives, since they are computed once, or once every several thousand steps, while the encoder keeps changing.

The encoder itself, kept current. This is ANCE, approximate nearest neighbor negative contrastive estimation (Xiong and colleagues, 2021). Its negatives are, in the authors' words, constructed "using the being-optimized DR model to retrieve from the entire corpus". The difficulty is cost. After every parameter update the passage vectors are out of date, and re-encoding a corpus of millions of passages takes hours, so the index cannot be rebuilt at every step. ANCE runs two processes side by side:

  1. A trainer takes gradient steps, drawing each query's negatives from the top results of the most recent index.
  2. An inferencer takes the latest saved checkpoint, re-encodes the whole corpus with it, rebuilds the nearest-neighbor index, and hands the index to the trainer. Then it starts again with the next checkpoint.

The negatives are therefore always a little stale: they are the hard negatives of a checkpoint some thousands of steps old. The authors call this asynchronous index refresh. Their reported configuration splits the graphics processors evenly between the two processes, refreshes the index every 10,000 batches, and for each positive samples one negative uniformly from the top 200 retrieved passages. Training starts from an encoder already trained with BM25 negatives. Finding the top passages quickly is itself a subject, that of the vector search unit.

Their Table 1 compares ways of choosing negatives for the same model, a RoBERTa-base bi-encoder, on the MS MARCO passage collection:

Negatives MRR@10, MS MARCO development queries Recall@1,000
BM25 alone, no dense retriever 0.240 0.814
Random, in batch 0.261 0.949
Hardest in batch 0.256 0.943
BM25 negatives 0.299 0.928
BM25 and random, as in DPR 0.311 0.952
ANCE 0.330 0.959

They also report how much each kind of negative resembles what the model must separate at test time: 15% of BM25 negatives were among the dense model's 100 top-scored negatives, none of the in-batch negatives were, and ANCE's negatives started at 63% and rose toward 100%, as the method intends.

Zhan and colleagues (2021, sections 4 and 5) give an account of why the source matters, which they support by a formal analysis (their equations 5 and 6 and Theorem 4.1) and by experiment. In their reading, training on random negatives minimizes the total number of wrongly ordered pairs, so a few difficult queries with many wrong pairs dominate, while training on hard negatives minimizes the wrong pairs among the top-ranked passages, which is what measures cut off at rank 10 reward. They also find static hard negatives risky, because the passages that were hard for an earlier checkpoint drift away from what is hard now, and propose training in which the negatives are recomputed against the current query encoder.

False negatives

A mined negative is a passage the encoder thinks is relevant and the labels do not mark as relevant. Sometimes the encoder is right and the labels are incomplete. Such a passage is a false negative, and a training step on it pushes a relevant passage away from its query.

Incomplete labels are the normal state of large training collections. Qu and colleagues (2021) note that MS MARCO has on average 1.1 labelled positive passages per question in a collection of 8.8 million. They examined by hand the top-retrieved passages that were not labelled positive, for 100 questions, and found that "about 70% of them are actually positives or highly relevant". Their ablation on MS MARCO shows the consequence (their Table 3, MRR@10 in percent): 32.39 with in-batch negatives, 33.32 with negatives shared across the batches of several processors, and 26.03 when hard negatives from the dense retriever's top results were added as they came. Mined negatives made the model worse than random ones. With the false negatives filtered out, the same hard negatives gave 36.38.

Cranfield lets us measure the hazard directly, because its judgments are far more complete than one label per query. The script hides them. For each query with at least two relevant abstracts (219 of the 225; the mean number of relevant abstracts per query is 7.16), one relevant abstract is chosen at random to be the only labelled positive, as in a sparsely labelled collection. The script then takes the top $k$ unlabelled abstracts of the dense run, or of BM25, as the mined negatives, and counts how many of them are relevant according to the full judgments. Thirty seeds choose the labelled abstract; the cells are medians over seeds with quartiles.

Mined from Depth $k$ Share of mined negatives that are relevant Queries with at least one false negative
The encoder 1 0.311 [0.301, 0.320] 0.311 [0.301, 0.320]
The encoder 5 0.280 [0.279, 0.282] 0.744 [0.727, 0.749]
The encoder 10 0.214 [0.213, 0.215] 0.836 [0.831, 0.845]
The encoder 50 0.078 [0.078, 0.079] 0.963 [0.959, 0.968]
BM25 1 0.281 [0.269, 0.291] 0.281 [0.269, 0.291]
BM25 10 0.205 [0.203, 0.205] 0.826 [0.817, 0.836]
BM25 50 0.072 [0.072, 0.073] 0.945 [0.941, 0.953]
Random unlabelled abstract 0.0045

With one label per query, the single hardest negative is a relevant abstract for 31% of the queries, and more than a fifth of the top ten are. A random unlabelled abstract is relevant with probability 0.0045, so mining from the top ten raises the false-negative rate by a factor of nearly fifty. For five queries in six, at least one of the ten mined negatives is false. BM25's top results are scarcely safer. The rates are lower than the 70% that RocketQA's authors found on MS MARCO, a different collection labelled in a different way, and they point the same way: the passages most worth mining are the ones most likely to be unlabelled positives.

Example (a common misreading). "Harder negatives are better negatives." The table and RocketQA's ablation both contradict it. The hardest negative of all is a relevant passage that nobody labelled. The quality of a mined negative depends on two things that pull in opposite directions as the mining gets better: how close it is to the positive, and how likely it is to be a positive itself.

Three remedies are in use.

  • Mine deeper and sample. ANCE samples from the top 200 instead of taking the top few. In the table, the share of false negatives falls with depth, while the abstracts at depth 50 are still, by construction, among the 50 the encoder scores highest out of 1,400.
  • Filter with a stronger model. RocketQA scores the mined passages with a cross-encoder, a model that reads the query and the passage together (the subject of cross-encoder re-ranking), and keeps as negatives only those it scores low. The authors' thresholds were a score below 0.1 for a negative and above 0.9 for a positive, and they report checking by hand that the selected data were more than 90% accurate.
  • Stop treating the label as all-or-nothing. If a stronger model can tell that a mined passage is half relevant, the encoder can be trained to reproduce that judgment. That is distillation.

Distillation: learning a teacher's scores

A bi-encoder must compress each passage into a vector before it sees any query. A cross-encoder reads the two texts together and can compare them word by word. The comparisons cited below find it the more accurate ranker, and it is far too slow to score a whole collection. Knowledge distillation uses the slow, accurate model to train the fast one. Hinton, Vinyals and Dean (2015) introduced the term for training a small network on the output probabilities of a large one, the "soft targets", in place of the correct labels alone. Their argument was that the large model's probabilities for the wrong answers carry information: "The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize."

For retrieval the teacher is a cross-encoder and the student a bi-encoder. Write $s_T(q, p)$ for the teacher's score of a pair and $s_S(q, p)$ for the student's. The obvious loss asks the student to reproduce each teacher score, the squared error $(s_S - s_T)^2$ summed over pairs. Hofstätter, Althammer, Schröder, Sertkan and Hanbury (2020) observed why that is a poor fit. Different architectures settle on different score ranges. In their Figure 1 a cross-encoder's scores are positive for relevant passages and negative on average for non-relevant ones, while the scores of dot-product models are large positive numbers for both. The ranges are accidents of architecture, and a ranking depends only on differences of scores within a query.

The same mismatch is visible with this unit's two models. On 2,409 query-abstract pairs from Cranfield (described below), the teacher's scores run from −11.41 to 9.59, and the student's cosines from 0.078 to 0.940. A cosine cannot exceed 1, so on every pair the teacher scores above 1, and on every pair it scores below −1, a pointwise squared error could never reach zero, however well the student ranked.

The margin mean squared error loss

Their remedy trains the student to reproduce the difference between the teacher's scores for a positive and a negative passage. For a training triple $(q, p^+, p^-)$ define the two margins

$$\Delta_S = s_S(q, p^+) - s_S(q, p^-), \qquad \Delta_T = s_T(q, p^+) - s_T(q, p^-).$$

The margin mean squared error (Margin-MSE) loss over a batch of $B$ triples is

$$\mathcal{L} = \frac{1}{B}\sum_{i=1}^{B}\left(\Delta_{S,i} - \Delta_{T,i}\right)^2$$

(their equations 11 and 12). The teacher's margins are computed once, before the student's training, and stored. The relevance labels do not appear in the loss. They only decide which passage of a triple is written $p^+$, and the authors note that the teacher's margin "can potentially be negative, which would indicate a reverse ordering from the original training data".

The gradient

Here is the plan: differentiate with respect to the student's two scores, then with respect to the vectors, for a student whose score is a dot product.

Write $e_i = \Delta_{S,i} - \Delta_{T,i}$ for the error of triple $i$. The margin $\Delta_{S,i}$ contains $s_S(q_i, p_i^+)$ with coefficient $+1$ and $s_S(q_i, p_i^-)$ with coefficient $-1$, and no other triple's scores, so

$$\frac{\partial \mathcal{L}}{\partial s_S(q_i, p_i^+)} = \frac{2}{B}\,e_i, \qquad \frac{\partial \mathcal{L}}{\partial s_S(q_i, p_i^-)} = -\frac{2}{B}\,e_i.$$

The two derivatives are equal and opposite. When the student's margin is too small ($e_i < 0$), gradient descent raises the positive's score and lowers the negative's by the same amount. When the margin is too large ($e_i > 0$), it does the reverse and pulls the two together. The contrastive loss never does that: it has no notion of a margin that is too large.

For a student that scores by the dot product $s_S(q, p) = \vec{q} \cdot \vec{p}$, the derivative of a score with respect to the query vector is the passage vector and conversely. The chain rule then gives, for the three vectors of triple $i$,

$$\frac{\partial \mathcal{L}}{\partial \vec{q}_i} = \frac{2}{B}\,e_i\left(\vec{p}_i^{\,+} - \vec{p}_i^{\,-}\right), \qquad \frac{\partial \mathcal{L}}{\partial \vec{p}_i^{\,+}} = \frac{2}{B}\,e_i\,\vec{q}_i, \qquad \frac{\partial \mathcal{L}}{\partial \vec{p}_i^{\,-}} = -\frac{2}{B}\,e_i\,\vec{q}_i.$$

If the same passage or query occurs in several triples, its gradients add. A step of gradient descent with a negative $e_i$ moves the query vector toward $\vec{p}^{\,+} - \vec{p}^{\,-}$, the positive toward the query and the negative away from it, each by an amount proportional to how far the student's margin falls short of the teacher's. The gradient with respect to the encoder's parameters follows by one more application of the chain rule through the encoder, which is ordinary backpropagation.

Three properties

1. The loss ignores any constant added to the student's scores for a query. If $s_S(q, p)$ is replaced by $s_S(q, p) + c_q$, with $c_q$ any number depending only on the query, each margin $\Delta_S$ is unchanged, since $c_q$ cancels in the difference. So is the loss. The same holds for the teacher's scores. The pointwise squared error has no such freedom. Writing $e^+ = s_S(q, p^+) - s_T(q, p^+)$ and $e^- = s_S(q, p^-) - s_T(q, p^-)$ for the two pointwise errors of a triple,

$$\left(\Delta_S - \Delta_T\right)^2 = \left(e^+ - e^-\right)^2 = (e^+)^2 + (e^-)^2 - 2e^+e^-.$$

The margin loss is zero whenever the two pointwise errors are equal, however large they are. The student may sit at any level; only its spacing is trained. That is the freedom the authors wanted, "to find the most comfortable or natural range" for each architecture.

2. The loss does not ignore scale. Multiplying the student's scores by a constant multiplies $\Delta_S$ by it, and the loss changes. The student must reproduce the teacher's margins in the teacher's units. For a student whose score is a cosine, $\lvert \Delta_S \rvert \le 2$, since each cosine lies between $-1$ and 1. A teacher margin of 10 can then never be matched, and the loss on that triple can never fall below $(10 - 2)^2 = 64$. Hofstätter and colleagues' students score by an unnormalized dot product, whose range is unlimited. A cosine student needs a scale factor, such as dividing by the temperature: with $\tau = 0.05$ its margins can reach $\pm 40$.

3. On a single triple, Margin-MSE is the high-temperature limit of Hinton's distillation. Treat a triple as a two-way classification between $p^+$ and $p^-$. At temperature $T$ the teacher's soft target for $p^+$ is $\sigma(\Delta_T/T)$ and the student's probability is $\sigma(\Delta_S/T)$, where $\sigma(x) = 1/(1 + e^{-x})$, because a softmax over two scores is the logistic function of their difference. The distillation loss is the cross-entropy of the student's probabilities against the teacher's,

$$C = -\sigma(\Delta_T/T)\,\ln \sigma(\Delta_S/T) - \left(1 - \sigma(\Delta_T/T)\right)\ln\left(1 - \sigma(\Delta_S/T)\right).$$

Using $\frac{d}{dx}\ln\sigma(x) = 1 - \sigma(x)$ and $\frac{d}{dx}\ln(1 - \sigma(x)) = -\sigma(x)$, the derivative with respect to $x = \Delta_S/T$ is $-\sigma(\Delta_T/T)(1 - \sigma(x)) + (1 - \sigma(\Delta_T/T))\,\sigma(x) = \sigma(x) - \sigma(\Delta_T/T)$, so

$$\frac{\partial C}{\partial \Delta_S} = \frac{1}{T}\left(\sigma(\Delta_S/T) - \sigma(\Delta_T/T)\right).$$

For large $T$ both arguments are small, and $\sigma(x) = \tfrac12 + \tfrac{x}{4} + O(x^3)$, so

$$\frac{\partial C}{\partial \Delta_S} \approx \frac{\Delta_S - \Delta_T}{4T^2}.$$

That is the gradient of $(\Delta_S - \Delta_T)^2/(8T^2)$: the squared error of the margins, up to a constant. Hinton and colleagues derive the corresponding statement for a classifier with many classes, by a first-order approximation of the exponential at high temperature and with the logits zero-meaned for each case: matching logits by squared error is the high-temperature limit of distillation (their section 2.1, equations 2 to 4). The script checks the two-candidate version numerically: with $\Delta_S = 3$ and $\Delta_T = 12$, the ratio of the exact gradient to $(\Delta_S - \Delta_T)/(4T^2)$ is 0.0211 at $T = 1$, 0.9418 at $T = 16$ and 0.9998 at $T = 256$. At low temperature the logistic function saturates, and a student margin of 3 against a teacher margin of 12 produces almost no gradient, since both probabilities are close to 1. The squared error of margins keeps asking for the full 12.

What the authors report

Hofstätter and colleagues trained a cross-encoder teacher on MS MARCO, scored the training triples with it, and trained students of four architectures. For the dot-product bi-encoder (a six-layer DistilBERT), their Table 2 compares losses with a single BERT-base teacher on the MS MARCO development queries, with each model re-ranking BM25's candidates:

Training of the bi-encoder nDCG@10 MRR@10
Labels only, no teacher 0.373 0.316
Weighted RankNet with teacher margins 0.384 0.326
Pointwise squared error 0.387 0.328
Margin-MSE 0.388 0.330

All three teacher losses improve on the labels alone, and Margin-MSE is the best of the three by 0.001 in nDCG@10 on this architecture (by 0.003 and 0.004 on the two other students in that table). The setting changes for their Table 4, so its figures are not comparable with those above: there the student retrieves from the whole collection with no BM25 stage. As a first-stage retriever, that table gives the same student an MRR@10 of 0.299 without a teacher, 0.321 with one teacher and 0.323 with an ensemble of three, with recall at 1,000 of 0.930, 0.954 and 0.957. They note that this was achieved with a batch size of 32 on one graphics processor, without index refreshes. Hofstätter and colleagues (2021) later combined Margin-MSE with in-batch negatives scored by a second teacher in a model known as TAS-B, and the BEIR benchmark found TAS-B the best of the dense retrievers it tested outside the training domain (Thakur and colleagues, 2021), a result limits of single-vector retrieval reports in full. Lin, Nogueira and Yates (2021, section 5.6) survey these and related distillation methods.

What a teacher does to false negatives

Distillation was introduced above as a remedy for false negatives. Whether it is one depends on whether the teacher can tell a hidden positive from a true negative. On Cranfield this can be checked, since the full judgments say which mined negatives are false.

Take the labelling of seed 1: one labelled positive for each of the 219 queries, and the top 10 unlabelled abstracts of the dense run as mined negatives. Of these 2,190 mined negatives, 466 are relevant by the full judgments and 1,724 are not. The teacher scores every pair.

Mined negatives Teacher margin $\Delta_T$, median [quartiles] Share with $\Delta_T \le 0$
True negatives −0.24 [−3.76, 3.28] 0.524
False negatives −1.82 [−5.74, 1.19] 0.655

The teacher does lean the right way. For the false negatives its margin is lower, and for two thirds of them it is zero or negative, which says "the passage labelled negative is at least as good as the labelled positive". Used as a score for telling false negatives from true ones, the teacher's score has an area under the ROC curve of 0.687 on this labelling, where the student's own cosine has 0.615. The figure depends a little on which relevant abstract happened to be labelled: over the 30 labellings of the previous section the teacher's area has a median of 0.690 with quartiles 0.688 and 0.693 (range 0.684 to 0.701), and the student's a median of 0.619 with quartiles 0.617 and 0.622.

It is a weak teacher here, though, and the first row shows it. For more than half of the true negatives the teacher also prefers the negative to the labelled positive. Its median score for the labelled positives is −0.76, below zero, where its median for the false negatives is 2.04 and for the true negatives −1.09. The teacher was trained on web search questions and is applied here to aeronautics abstracts, and the labelled positive is a randomly chosen relevant abstract, which Cranfield may have graded as of minimum interest. A student trained to reproduce these margins would be taught a good deal of noise along with the correction. Distillation transfers what the teacher knows, and on a collection unlike the teacher's training data that can be little more than the student already knows. The gains reported above were measured on MS MARCO, where the teacher was trained.

Under the contrastive loss every mined negative, true or false, is pushed away with a weight that grows with its score. Under Margin-MSE the push is proportional to $\Delta_T - \Delta_S$. For a false negative with $\Delta_T$ near zero the student is asked to hold the two passages at about the same score, which is the right answer for two relevant passages.

Filtering in the manner of RocketQA can be simulated with the same scores. Keeping only the mined negatives whose logistic-transformed teacher score is below 0.5 keeps 1,017 of the 1,724 true negatives and 147 of the 466 false ones, and the share of false negatives among those kept falls from 0.213 to 0.126. A stricter threshold of 0.1 brings it to 0.101 and discards 58% of the true negatives. (The thresholds are applied to this teacher's score as it is. RocketQA's thresholds of 0.1 and 0.9 were chosen for its own cross-encoder, and this teacher's score is not calibrated as a probability.)

A toy training run

Training a transformer encoder on MS MARCO takes days on graphics processors and is out of reach of the laptop these articles are computed on. The mechanisms above can still be watched at toy scale. The experiment below trains only a single linear layer on top of the frozen vectors, and it should be read as an illustration of the mechanisms and not as evidence about full fine-tuning.

Setup. The 1,400 abstract vectors and 225 query vectors of the pinned encoder are fixed. The trainable part is one $384 \times 384$ matrix $W$, starting at the identity. The score is the cosine of $W\vec{q}$ and $W\vec{p}$. The 225 queries are split as elsewhere in the course into 112 development queries, used for training, and 113 test queries. Training takes 60 steps of the Adam optimizer (learning rate 0.0002) on all training pairs at once, with the contrastive loss at $\tau = 0.05$ and seven negatives per pair, redrawn at every step, uniformly and with replacement, from a pool. The mined pools hold 30 abstracts. The pool is one of four kinds:

  • random: all abstracts not labelled relevant for the query;
  • BM25: the top non-labelled abstracts of BM25;
  • static: the top non-labelled abstracts of the frozen encoder, computed once;
  • refreshed: the top non-labelled abstracts under the current $W$, recomputed every 10 steps, which is ANCE's scheme in miniature (and without its staleness, since re-encoding here is one matrix product).

Each is run with full labels, where every relevant abstract is a labelled positive, and with one label per query, where a single relevant abstract chosen by the seed is labelled and the others can turn up as negatives. Thirty seeds; the cells are medians with quartiles, on the 113 test queries. The frozen encoder scores nDCG@10 0.362 and recall@100 0.763 on them.

Labels Negatives nDCG@10 Recall@100 Share of negatives used that were relevant
Full Random 0.360 [0.358, 0.361] 0.794 [0.793, 0.795] 0.000
Full BM25 0.346 [0.346, 0.347] 0.768 [0.767, 0.769] 0.000
Full Static 0.356 [0.354, 0.357] 0.767 [0.766, 0.768] 0.000
Full Refreshed 0.359 [0.358, 0.359] 0.776 [0.773, 0.777] 0.000
One per query Random 0.357 [0.352, 0.360] 0.781 [0.776, 0.788] 0.004
One per query BM25 0.345 [0.336, 0.350] 0.751 [0.744, 0.759] 0.099
One per query Static 0.348 [0.343, 0.356] 0.757 [0.749, 0.766] 0.108
One per query Refreshed 0.351 [0.345, 0.359] 0.762 [0.758, 0.771] 0.111
One per query Static, filtered by the teacher 0.351 [0.348, 0.353] 0.770 [0.762, 0.775] 0.059
One per query Static, Margin-MSE against the teacher 0.359 [0.357, 0.362] 0.793 [0.787, 0.798] 0.108

The filtered row keeps a mined negative only if the teacher's score for it is below zero. The Margin-MSE row replaces the contrastive loss by the margin loss against the teacher's scores, with the student's score taken as the cosine divided by $\tau$.

The toy does not reproduce the headline of the published work, and it does reproduce the hazard.

  • No variant improves nDCG@10 over the frozen encoder (0.362). A single linear layer trained on 112 queries may simply have little to add to an encoder trained on a billion pairs. What it does change is recall deeper in the list: every variant with full labels raises recall@100 above the frozen 0.763.
  • Mined negatives do not beat random ones here. With full labels, random negatives give the best recall@100 (0.794), and the refreshed pool (0.776) does better than the static one (0.767) and BM25's (0.768). That current negatives do better than stale ones agrees with the published account. Their standing against random negatives does not. One possible reason is that a single linear map acts on the whole space at once, so that pushing a query away from its thirty nearest non-relevant abstracts also moves its relevant neighbors; the experiment does not test that. The published gains are for encoders whose every layer is trained, on collections thousands of times larger.
  • Hiding labels hurts mined negatives and hardly touches random ones. With one label per query, about a tenth of the mined negatives used are relevant abstracts (0.099 to 0.111), against 0.004 of the random ones. Recall@100 falls by 0.010 for the static pool (0.767 to 0.757), by 0.014 for the refreshed pool and by 0.017 for BM25's, and nDCG@10 falls with it. For random negatives the fall is from 0.794 to 0.781. False negatives are rare among them (0.004), so that fall more likely comes from having about a seventh as many positive pairs to train on; no run with the labels thinned and the negatives kept clean was made to confirm it.
  • Both remedies recover ground. Filtering by the teacher halves the share of false negatives (0.108 to 0.059) and brings recall@100 back from 0.757 to 0.770. Margin-MSE on the same unfiltered pool, false negatives included, reaches 0.793 and the best nDCG@10 of the one-label rows, 0.359. This is with the weak teacher of the previous section. The margin loss never asks for more separation than the teacher sees, which may matter more here than the teacher's accuracy. The experiment does not separate those two explanations.

These are thirty seeds of one small design, and differences of a few thousandths between rows are within what another design could reverse. The pattern that holds across the rows is the one the section set out to show: mined negatives are where missing labels do their damage.

What is proved here, what is cited, and what the experiments show

Proved on this page. The bound of the contrastive gradient by $(2G/\tau)(1 - \pi^+)$ and by $(2G/\tau)\mathcal{L}$. The chance that a random draw contains an informative negative. The gradient of Margin-MSE with respect to scores and, for a dot-product student, with respect to vectors. Its invariance to a per-query shift, its relation to the pointwise squared error, the bound $\lvert \Delta_S \rvert \le 2$ for a cosine student, and the high-temperature limit of two-candidate distillation.

Cited. The derivation of the contrastive gradient with respect to scores (the unit's own article on contrastive training). Every effectiveness figure for DPR, ANCE, RocketQA, Margin-MSE and TAS-B, each as its paper reports it, with the table named. The convergence analysis of Xiong and colleagues and the account of Zhan and colleagues, at the strength their papers state them.

Shown by the experiments, on Cranfield with one small encoder and one small teacher. The loss and gradient lengths under three kinds of negatives. The false-negative rates under one label per query. The teacher's margins for true and false negatives. The toy training results, which concern a linear layer on frozen vectors and 112 training queries.

Worked problems

The unit has no course assignments, so the problems are small exact cases chosen to exercise each result; the script recomputes every number.

Problem 1. A query has a positive with score 0.60 and two negatives. Compute the contrastive loss at $\tau = 0.05$, and the gradient with respect to each score, (a) for negatives with scores 0.20 and 0.10, and (b) for negatives with scores 0.58 and 0.55.

Divide the scores by $\tau$. (a) The scaled scores are 12, 4 and 2. Subtracting 12, the exponentials are $1$, $e^{-8} = 0.000335$ and $e^{-10} = 0.0000454$, with sum 1.000381. So $\pi^+ = 0.999619$, the loss is $-\ln 0.999619 = 0.000381$, and the gradients are $-(1 - \pi^+)/\tau = -0.00761$ for the positive, $0.000335/1.000381/0.05 = 0.00671$ for the first negative and $0.00091$ for the second. (b) The scaled scores are 12, 11.6 and 11. The exponentials relative to the positive are $1$, $e^{-0.4} = 0.6703$ and $e^{-1} = 0.3679$, with sum 2.0382. So $\pi^+ = 0.4906$, the loss is 0.7121, and the gradients are $-0.5094/0.05 = -10.19$ for the positive, $0.3289/0.05 = 6.58$ for the first negative and $0.1805/0.05 = 3.61$ for the second. The positive is 0.02 above the nearer negative in (b), yet the loss is 1,870 times that of (a) and the pull on the positive 1,338 times as large. By the proposition, the gradient length in (a) can be at most $(2G/\tau) \times 0.000381$.

Problem 2. A batch has three triples. The student's scores $(s_S^+, s_S^-)$ are $(8, 5)$, $(7, 6)$ and $(4, 6)$; the teacher's $(s_T^+, s_T^-)$ are $(9, -3)$, $(6, 5)$ and $(2, 7)$. Compute the Margin-MSE loss and the gradient with respect to each student score. Compare with the pointwise squared error, and say what happens to each loss if 100 is added to every student score.

The student margins are $3$, $1$ and $-2$; the teacher margins are $12$, $1$ and $-5$. The errors $e_i = \Delta_S - \Delta_T$ are $-9$, $0$ and $3$, with squares 81, 0 and 9. The loss is $(81 + 0 + 9)/3 = 30$. The gradients $\pm\tfrac{2}{3}e_i$ are: for the first triple $-6$ on the positive's score and $+6$ on the negative's; for the second, 0 and 0; for the third, $+2$ and $-2$.

Read the three triples in turn. In the first the student agrees with the teacher about the order and has a quarter of its margin, so the positive is raised and the negative lowered. In the second the student's scores are each 1 above the teacher's; the margins agree and nothing moves. In the third both models rank the passage labelled negative above the one labelled positive, which is what a false negative looks like, and the teacher does so by more. The gradient $+2$ on the positive's score means that descent lowers the labelled positive and raises the labelled negative. The label is overruled.

The pointwise squared error on the same scores sums $(s_S - s_T)^2$ over the six scores: $1 + 64 + 1 + 1 + 4 + 1 = 72$, or 24 per triple. It charges the second triple 2 for being one unit too high on both scores, a difference that changes no ranking. If 100 is added to every student score the margins do not change, and the Margin-MSE loss stays 30. The pointwise loss becomes $(99^2 + 108^2 + 101^2 + 101^2 + 102^2 + 99^2)/3 = 62{,}072/3$, which is 20,690.7.

Problem 3. A student scores by cosine divided by the temperature $\tau = 0.05$. For one triple the cosines are 0.70 for the positive and 0.55 for the negative, and the teacher's margin is 8. Compute the loss on this triple and the direction of the update. What is the largest teacher margin this student can match?

The student's margin is $(0.70 - 0.55)/0.05 = 3$. The error is $3 - 8 = -5$ and the loss on the triple is 25. The error is negative, so descent raises the positive's score and lowers the negative's. The gradient with respect to the positive's cosine is $2 \times (-5)/0.05 = -200$: a step of size $\eta$ raises that cosine by $200\eta$. Cosines lie between $-1$ and 1, so the student's margin lies between $-40$ and $40$, and a teacher margin beyond that cannot be matched. Without the division by $\tau$ the limit would be 2 (property 2).

Problem 4. In a collection of one million passages, 50 are informative negatives for a given training pair. What is the chance that a batch supplying 31 random negatives contains at least one? How many negatives would make the chance one half?

The chance of none is $\binom{999{,}950}{31}/\binom{1{,}000{,}000}{31}$, which is close to $(1 - 50/10^6)^{31} = 0.99845$. So the chance of at least one is 0.00155, which the approximation $nh/M = 31 \times 50/10^6$ also gives. For a chance of one half, $(1 - 0.00005)^n = 0.5$ gives $n = \ln 2/0.00005$, which is 13,863 negatives per query. That is the arithmetic behind very large batches: RocketQA's cross-batch negatives, for which Hofstätter and colleagues cite a batch size of 4,000, raise $n$ toward this range, and mining raises the informative share of the candidates directly.

What we have learned

The contrastive loss learns only from negatives that the encoder confuses with the positive: its gradient is bounded by the loss, and random negatives from a large collection almost never include a confusing one. Mining negatives with a retriever supplies them, most effectively when the retriever is the encoder being trained and its index is kept nearly current, as in ANCE. The better the mining, the more of the mined passages are relevant passages that were never labelled, and training on those damages the encoder, as RocketQA's ablation showed and the hidden-label experiment on Cranfield shows in miniature. A cross-encoder teacher can filter mined negatives or, through the margin mean squared error loss, replace the label altogether with a graded target. That loss trains only the spacing of the student's scores, its gradient pulls a pair together as readily as it pushes one apart, and it is what distillation at high temperature becomes on a pair of candidates.

All of this makes a single vector per passage as good as training can make it. It does not change what a single vector can hold, which is the subject of limits of single-vector retrieval.

References

  • Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531. record
  • Hofstätter, S., Althammer, S., Schröder, M., Sertkan, M., and Hanbury, A. (2020). Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv:2010.02666. record
  • Hofstätter, S., Lin, S.-C., Yang, J.-H., Lin, J., and Hanbury, A. (2021). Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of SIGIR 2021, 113–122. record
  • Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP 2020, 6769–6781. arXiv:2004.04906. record · 10 pages
  • Lin, J., Nogueira, R., and Yates, A. (2021). Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures on Human Language Technologies, 14(4). Morgan & Claypool (reissued by Springer, 2022). Preprint: arXiv:2010.06467; section numbers here follow version 3 of the preprint. record · 2 pages
  • Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H., and Wang, H. (2021). RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of NAACL-HLT 2021, 5835–5847. record
  • sentence-transformers (n.d.). cross-encoder/ms-marco-MiniLM-L6-v2. Model card at revision 233902d25c440f23af6f7d6e94d2946bac0bee0a. https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2 (accessed 17 November 2026). record
  • sentence-transformers (n.d.). sentence-transformers/all-MiniLM-L6-v2. Model card and train_script.py at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 (accessed 17 November 2026). record · 5 pages
  • Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track. arXiv:2104.08663; read in version 4 of the arXiv paper, whose Table 2 agrees with the published version. record · 7 pages
  • Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., and Overwijk, A. (2021). Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of ICLR 2021. arXiv:2007.00808; figures here are from version 2 of the arXiv paper, which was not compared with the published version. record
  • Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., and Ma, S. (2021). Optimizing dense retrieval model training with hard negatives. In Proceedings of SIGIR 2021, 1503–1512. record