Before it was trained on pairs of texts, the six-layer network behind this unit's encoder gave a Cranfield query nearly the same cosine with every abstract in the collection: 0.835 on average with the abstracts judged relevant to it and 0.832 with the rest. After training, query 185, experimental studies on panel flutter, has a cosine of 0.751 with abstract 856, some experimental studies of panel flutter at mach 1.3, and cosines of 0.200 and 0.143 with the relevant abstracts of two unrelated queries. What training signal pulls a query's vector toward the passages that answer it and away from all the others? A classification loss does: for each query, the network is asked to pick its own passage out of a set of candidates, and is penalized by the negative logarithm of the probability it gave the right one.
That loss is called the contrastive loss, or InfoNCE after the paper that named it (van den Oord, Li and Vinyals, 2018). The passage paired with a query is its positive, written $p^+$. The other candidates are negatives, $p^-_1, \dots, p^-_n$. This article is the unit's home for the loss: Dense Passage Retrieval uses it, and Hard Negatives and Distillation starts from its gradient.
Here is the route. First we write the loss as a softmax classification and prove what score function minimizes it. Then we derive its gradient and work it by hand, in exact fractions, on a batch of three queries and three passages. Next come the two choices that every implementation makes: taking the negatives from the same batch, and dividing cosine scores by a temperature. Then we measure, on Cranfield, what more negatives do to the loss and its gradient, and train a small encoder to see the loss work. The article assumes the bi-encoder and the softmax of the library's transformer walkthrough. The numbers are computed by tools/experiments/dense/contrastive-training-for-retrieval.py, and the training runs by tools/experiments/dense/toytrain.py.
The loss as a classification
Suppose a query $q$ comes with $n + 1$ candidate passages, of which exactly one, $p^+$, is relevant. Number the candidates $p_0, p_1, \dots, p_n$ with $p_0 = p^+$. The encoder gives each a score $s_j = s(q, p_j)$: a dot product of the two vectors, or a cosine divided by a constant, as later sections discuss. The model's probability that candidate $j$ is the relevant one is the softmax of the scores,
$$P_j = \frac{e^{s_j}}{\sum_{k=0}^{n} e^{s_k}} ,$$
and the loss for this query is the cross-entropy of that distribution against the truth, which puts all its mass on candidate 0:
$$\mathcal{L} = -\ln P_0 = -\ln \frac{e^{s(q, p^+)}}{e^{s(q, p^+)} + \sum_{j=1}^{n} e^{s(q, p^-_j)}} = -s_0 + \ln \sum_{k=0}^{n} e^{s_k} .$$
This is the loss of an $(n+1)$-way classifier, with one difference from the usual kind: the classes are not fixed. They are whatever passages happen to be the candidates, and each "class" is represented by a vector computed by the passage encoder, not by a row of a weight matrix.
Three properties follow at once from the formula.
- The loss is never negative, and it is zero only in the limit where the positive's score exceeds every negative's by an unbounded margin.
- If the encoder scores every candidate equally, $P_0 = 1/(n+1)$ and the loss is $\ln(n + 1)$. That is the loss of a guess, and the natural reference point: $\ln 2 = 0.693$ with one negative, $\ln 128 = 4.852$ with 127.
- Only differences of scores matter. Adding any quantity that depends on the query alone to all of its scores leaves every $P_j$ unchanged.
The same loss appears under several names. Sohn (2016) introduced it for learning image embeddings as the multi-class N-pair loss. Henderson et al. (2017) used it to train a dot-product model that matches an email to a reply. Van den Oord, Li and Vinyals (2018) gave it the name InfoNCE. Chen et al. (2020) call the version with cosine scores and a temperature the normalized temperature-scaled cross-entropy. The retrieval papers mostly call it the negative log-likelihood of the positive passage (Karpukhin et al., 2020, equation 2).
What score minimizes the loss?
A loss says what the encoder is rewarded for. To see what that is, we ask which score function would make the expected loss as small as possible, if the encoder could represent any function at all.
Here is the idea. Give the candidates a probability model: the positive is drawn from the distribution of passages relevant to the query, and each negative is drawn independently from some fixed "noise" distribution over passages. Given the $n + 1$ candidates in a random order, Bayes' rule says how probable it is that each one is the positive. The softmax can match that probability exactly, and it does so when the score is the logarithm of a ratio of the two densities.
The formal version. Let $\pi(p \mid q)$ be the probability of drawing passage $p$ as the positive for $q$, and $\nu(p)$ the probability of drawing it as a negative. Draw the positive from $\pi(\cdot \mid q)$ and $n$ negatives independently from $\nu$, and put the positive at a position $J$ chosen uniformly from $0, \dots, n$. The probability of seeing the candidates $p_0, \dots, p_n$ with the positive at position $j$ is
$$\frac{1}{n+1}\; \pi(p_j \mid q) \prod_{l \ne j} \nu(p_l) = \frac{1}{n+1}\; \frac{\pi(p_j \mid q)}{\nu(p_j)} \prod_{l=0}^{n} \nu(p_l) .$$
The product over all $l$ and the factor $1/(n+1)$ are the same for every $j$. So by Bayes' rule the posterior probability that position $j$ holds the positive is
$$\Pr(J = j \mid p_0, \dots, p_n, q) = \frac{r(q, p_j)}{\sum_{k=0}^{n} r(q, p_k)}, \qquad r(q, p) = \frac{\pi(p \mid q)}{\nu(p)} .$$
This is the argument of van den Oord, Li and Vinyals (2018, equation 5). Now compare it with the model's $P_j = e^{s_j} / \sum_k e^{s_k}$. The expected loss is the expected cross-entropy between the true posterior and the model's distribution, and a cross-entropy is smallest when the two distributions are equal (the difference between a cross-entropy and its minimum is a Kullback-Leibler divergence, which is zero only then). The two are equal for every set of candidates when $e^{s(q, p)}$ is proportional to $r(q, p)$, with a constant that may depend on $q$:
$$s^*(q, p) = \ln \frac{\pi(p \mid q)}{\nu(p)} + c(q) .$$
So the loss rewards a score that says how much more likely a passage is to be drawn as this query's positive than to be drawn as a negative. The optimal score does not depend on the number of negatives $n$, as the same authors note. It does depend on the noise distribution $\nu$, and that fact has two consequences that later sections meet in practice. A passage that is common among the negatives has its score pushed down, whatever its content. And the negatives decide what the encoder learns to tell apart, which is why the choice of negatives gets an article of its own.
Which quantity the optimal score is depends on where the negatives come from. If they are drawn uniformly from the collection, $\nu$ is a constant and the optimal score is $\ln \pi(p \mid q)$ up to a term in $q$ alone: the logarithm of the probability that the passage is the query's positive, with nothing subtracted for a popular passage. If they are drawn from the distribution of passages as positives, $m(p) = \sum_q \Pr(q)\, \pi(p \mid q)$, then $\ln(\pi(p \mid q) / m(p))$ is the pointwise mutual information of the query and the passage. In-batch negatives, described below, are draws of the second kind. Word Embeddings proves the corresponding result for single words, where a simpler loss with one binary decision per pair leads to the same quantity shifted by a constant. Worked problem 2 checks the optimal score on a two-passage example.
A bound on how low the loss can go. The loss of a uniform guess is $\ln(n+1)$. How far below that can any score function get? Not further than the divergence between the two distributions the candidates are drawn from:
$$\ln(n + 1) - \mathbb{E}[\mathcal{L}] \;\le\; \mathrm{KL}(\pi \,\|\, \nu) = \sum_p \pi(p \mid q) \ln \frac{\pi(p \mid q)}{\nu(p)} ,$$
for one query $q$, any score function, and any $n$, where the expectation is over the positive drawn from $\pi(\cdot \mid q)$ and $n$ negatives drawn independently from $\nu$. In plain words: the encoder can only tell the positive from the negatives to the extent that positives and negatives are differently distributed.
The proof uses one general inequality. For two distributions $P$ and $Q$ on the same set and any function $T$ with $\mathbb{E}_Q[e^T]$ finite,
$$\mathbb{E}_P[T] \le \mathrm{KL}(P \,\|\, Q) + \ln \mathbb{E}_Q[e^T] .$$
To see it, let $Q'$ be the distribution with probabilities $Q'(x) = Q(x)\, e^{T(x)} / \mathbb{E}_Q[e^T]$. A Kullback-Leibler divergence is never negative, so $0 \le \mathrm{KL}(P \,\|\, Q') = \sum_x P(x) \ln \frac{P(x)}{Q(x)} - \mathbb{E}_P[T] + \ln \mathbb{E}_Q[e^T]$, which is the inequality rearranged.
Now apply it. Let $P$ be the distribution of the candidates $(p_0, \dots, p_n)$ as the loss sees them, $p_0$ from $\pi(\cdot \mid q)$ and the rest from $\nu$, and let $Q$ be the distribution in which all $n + 1$ are drawn from $\nu$. The two differ only in the first coordinate, so $\mathrm{KL}(P \,\|\, Q) = \mathrm{KL}(\pi \,\|\, \nu)$. Take
$$T(p_0, \dots, p_n) = \ln \frac{(n+1)\, e^{s(q, p_0)}}{\sum_{k=0}^{n} e^{s(q, p_k)}} .$$
Under $P$, $\mathbb{E}_P[T] = \ln(n+1) - \mathbb{E}[\mathcal{L}]$, by the definition of the loss. Under $Q$ the $n + 1$ candidates are exchangeable, so each has the same expected share of the softmax, and the shares sum to 1: $\mathbb{E}_Q\big[e^{s_0} / \sum_k e^{s_k}\big] = 1/(n+1)$, which gives $\mathbb{E}_Q[e^T] = 1$ and $\ln \mathbb{E}_Q[e^T] = 0$. The general inequality then reads $\ln(n+1) - \mathbb{E}[\mathcal{L}] \le \mathrm{KL}(\pi \,\|\, \nu)$.
The bound on mutual information. The name InfoNCE comes from the consequence. Suppose the negatives are independent draws from the marginal distribution of positives, $\nu = m$. Averaging the inequality over queries, the right side becomes $\sum_q \Pr(q)\, \mathrm{KL}(\pi(\cdot \mid q) \,\|\, m)$, which is the mutual information $I(q; p)$ between a query and its positive. So
$$I(q; p) \ge \ln(n + 1) - \mathcal{L}_{n+1} ,$$
where $\mathcal{L}_{n+1}$ is the expected loss with $n + 1$ candidates, for any score function. Van den Oord, Li and Vinyals (2018, §2.3) state this bound, with a derivation in their Appendix A.1 that uses an approximation (their equation 8); Poole et al. (2019, §2.3, equation 10 and footnote 2) prove it without one, for candidates that are independent samples of positive pairs. Poole et al. also note, crediting van den Oord et al., that the right side can never exceed $\ln(n + 1)$, since the loss is never negative. So the bound is loose whenever the true information exceeds $\ln(n + 1)$, however good the encoder: minimizing the loss raises a lower bound on the information the two vectors share, and the number of negatives caps how much of it the loss can register. The condition matters. With negatives drawn any other way, $\ln(n+1) - \mathcal{L}$ is bounded by the divergence above and need not be a bound on mutual information.
The gradient
Training moves the encoder's weights down the gradient of the loss. The gradient reaches the weights through the scores and then through the vectors, so we take it in those two steps.
With respect to the scores. From $\mathcal{L} = -s_0 + \ln \sum_k e^{s_k}$,
$$\frac{\partial \mathcal{L}}{\partial s_j} = -[j = 0] + \frac{e^{s_j}}{\sum_k e^{s_k}} = P_j - y_j ,$$
where $y_j$ is 1 for the positive and 0 for a negative. This is the familiar gradient of softmax cross-entropy: predicted probability minus target. The positive's score is pushed up in proportion to $1 - P_0$, the probability the model failed to give it. Each negative's score is pushed down in proportion to $P_j$, the probability the model wrongly gave it. The pushes sum to zero, since $\sum_j (P_j - y_j) = 0$.
With respect to the vectors. Take the score to be a dot product divided by a temperature $\tau$, $s_j = \vec{q} \cdot \vec{p}_j / \tau$ (with $\tau = 1$ this is the plain dot product of DPR). Then $\partial s_j / \partial \vec{q} = \vec{p}_j / \tau$ and $\partial s_j / \partial \vec{p}_j = \vec{q} / \tau$, and by the chain rule
$$\frac{\partial \mathcal{L}}{\partial \vec{q}} = \frac{1}{\tau} \sum_{j=0}^{n} (P_j - y_j)\, \vec{p}_j = \frac{1}{\tau} \Big( \sum_{j=0}^{n} P_j \vec{p}_j - \vec{p}^{\,+} \Big), \qquad \frac{\partial \mathcal{L}}{\partial \vec{p}_j} = \frac{1}{\tau} (P_j - y_j)\, \vec{q} .$$
The first formula has a plain reading. $\sum_j P_j \vec{p}_j$ is the average candidate vector under the model's own beliefs. Gradient descent moves the query vector along $\vec{p}^{\,+} - \sum_j P_j \vec{p}_j$: toward the positive and away from where the model currently thinks the answer is. When the model is already right, $P_0$ is near 1, the average is near $\vec{p}^{\,+}$, and the step is near zero. The second formula says the positive passage is pulled toward the query with weight $1 - P_0$ and each negative is pushed away from it with weight $P_j$. A negative the model already ranks far below the positive has $P_j$ near zero and is hardly moved.
For a whole batch. With $B$ queries, row $i$ of a score matrix $S$ holds the scores of query $i$ against a common list of passages, and the batch loss is the mean of the $B$ row losses. Write $\mathbf{Q}$ for the matrix whose rows are the query vectors, $\mathbf{P}$ for the passage vectors, $S = \mathbf{Q}\mathbf{P}^{\mathsf{T}} / \tau$, and $G$ for the matrix of row-wise softmax probabilities minus the matrix $Y$ that has a 1 where row $i$'s positive is. Applying the two formulas row by row and summing,
$$\frac{\partial \mathcal{L}}{\partial \mathbf{Q}} = \frac{1}{B\tau}\, G\, \mathbf{P}, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{P}} = \frac{1}{B\tau}\, G^{\mathsf{T}} \mathbf{Q} .$$
A passage's gradient collects a term from every query that scored it, as a positive for one and a negative for the others.
Through length normalization. Most encoders, this unit's among them, divide each vector by its length before scoring. If $\vec{x} = \vec{u} / \lvert \vec{u} \rvert$ and $\vec{g} = \partial \mathcal{L} / \partial \vec{x}$ is the gradient with respect to the unit vector, the gradient with respect to the raw vector is
$$\frac{\partial \mathcal{L}}{\partial \vec{u}} = \frac{1}{\lvert \vec{u} \rvert} \big( \vec{g} - (\vec{g} \cdot \vec{x})\, \vec{x} \big) .$$
To derive it, differentiate $x_a = u_a / \lvert \vec{u} \rvert$ with respect to $u_b$, using $\partial \lvert \vec{u} \rvert / \partial u_b = u_b / \lvert \vec{u} \rvert$:
$$\frac{\partial x_a}{\partial u_b} = \frac{[a = b]}{\lvert \vec{u} \rvert} - \frac{u_a u_b}{\lvert \vec{u} \rvert^3} = \frac{1}{\lvert \vec{u} \rvert} \big( [a = b] - x_a x_b \big) ,$$
and then $\partial \mathcal{L} / \partial u_b = \sum_a g_a\, \partial x_a / \partial u_b = (g_b - (\vec{g} \cdot \vec{x}) x_b) / \lvert \vec{u} \rvert$. The bracket removes from $\vec{g}$ its component along $\vec{x}$. The result is orthogonal to $\vec{u}$: lengthening a vector does not change any cosine, so the loss has no slope in that direction. The script checks this formula, and the orthogonality, on the batch below.
A batch of three, by hand
Three queries and three passages, in two dimensions, with dot-product scores and $\tau = 1$. Passage $j$ is the positive of query $j$. The vectors are chosen so that every exponential is a rational number:
$$\vec{p}_1 = (1, 1), \quad \vec{p}_2 = (1, 0), \quad \vec{p}_3 = (0, 1); \qquad \vec{q}_1 = (\ln 2, \ln 3), \quad \vec{q}_2 = (\ln 4, -\ln 2), \quad \vec{q}_3 = (0, \ln 3) .$$
Numerically the query vectors are $(0.693, 1.099)$, $(1.386, -0.693)$ and $(0, 1.099)$.
Scores. For query 1, $e^{\vec{q}_1 \cdot \vec{p}_1} = e^{\ln 2 + \ln 3} = 6$, $e^{\vec{q}_1 \cdot \vec{p}_2} = 2$ and $e^{\vec{q}_1 \cdot \vec{p}_3} = 3$. Doing the same for the other two queries gives the matrix of exponentiated scores, rows for queries and columns for passages:
| $\vec{p}_1$ | $\vec{p}_2$ | $\vec{p}_3$ | Row sum | |
|---|---|---|---|---|
| $\vec{q}_1$ | 6 | 2 | 3 | 11 |
| $\vec{q}_2$ | 2 | 4 | 1/2 | 13/2 |
| $\vec{q}_3$ | 3 | 1 | 3 | 7 |
Probabilities. Dividing each row by its sum:
| $\vec{p}_1$ | $\vec{p}_2$ | $\vec{p}_3$ | |
|---|---|---|---|
| $\vec{q}_1$ | 6/11 | 2/11 | 3/11 |
| $\vec{q}_2$ | 4/13 | 8/13 | 1/13 |
| $\vec{q}_3$ | 3/7 | 1/7 | 3/7 |
The positives are on the diagonal. Query 3 has a problem: passage 1 scores exactly as high as its own positive, so the model gives each a probability of 3/7.
Losses. The three row losses are $\ln(11/6) = 0.606$, $\ln(13/8) = 0.486$ and $\ln(7/3) = 0.847$. The batch loss is their mean,
$$\mathcal{L} = \tfrac{1}{3} \ln \Big( \tfrac{11}{6} \cdot \tfrac{13}{8} \cdot \tfrac{7}{3} \Big) = \tfrac{1}{3} \ln \tfrac{1001}{144} = 0.646 ,$$
against $\ln 3 = 1.099$ for a uniform guess.
Gradient with respect to the scores. Subtracting 1 on the diagonal gives $G$:
| $\vec{p}_1$ | $\vec{p}_2$ | $\vec{p}_3$ | |
|---|---|---|---|
| $\vec{q}_1$ | −5/11 | 2/11 | 3/11 |
| $\vec{q}_2$ | 4/13 | −5/13 | 1/13 |
| $\vec{q}_3$ | 3/7 | 1/7 | −4/7 |
Each row sums to zero.
Gradient with respect to the query vectors. For each query, $\partial \mathcal{L}_i / \partial \vec{q}_i = \sum_j G_{ij} \vec{p}_j$ (its own loss, before the factor $1/B$):
- $\vec{q}_1$: $-\tfrac{5}{11}(1,1) + \tfrac{2}{11}(1,0) + \tfrac{3}{11}(0,1) = (-\tfrac{3}{11}, -\tfrac{2}{11})$.
- $\vec{q}_2$: $\tfrac{4}{13}(1,1) - \tfrac{5}{13}(1,0) + \tfrac{1}{13}(0,1) = (-\tfrac{1}{13}, \tfrac{5}{13})$.
- $\vec{q}_3$: $\tfrac{3}{7}(1,1) + \tfrac{1}{7}(1,0) - \tfrac{4}{7}(0,1) = (\tfrac{4}{7}, -\tfrac{1}{7})$.
Gradient descent moves each query against its gradient. Query 1 moves along $(\tfrac{3}{11}, \tfrac{2}{11})$, toward its positive $(1, 1)$. Query 3 is the instructive one. The model's average passage for it is $\tfrac{3}{7}(1,1) + \tfrac{1}{7}(1,0) + \tfrac{3}{7}(0,1) = (\tfrac{4}{7}, \tfrac{6}{7})$, its positive is $(0, 1)$, and it moves along the difference, $(-\tfrac{4}{7}, \tfrac{1}{7})$: up a little, toward its positive, and strongly to the left, away from the passage that tied with it. The negative that the model confuses with the positive is the one that shapes the step.
Gradient with respect to the passage vectors. For the batch loss, $\partial \mathcal{L} / \partial \vec{p}_j = \tfrac{1}{3} \sum_i G_{ij} \vec{q}_i$. For passage 1 this is $\tfrac{1}{3}\big({-\tfrac{5}{11}} \vec{q}_1 + \tfrac{4}{13} \vec{q}_2 + \tfrac{3}{7} \vec{q}_3\big) = (0.037, -0.081)$; for passages 2 and 3 it is $(-0.136, 0.208)$ and $(0.099, -0.127)$. Passage 1 is pulled toward query 1, whose positive it is, and pushed away from queries 2 and 3.
The check. The script computes all of this in exact fractions. It then compares the gradients with central finite differences of the loss, which agree to within $10^{-10}$, and with PyTorch's automatic differentiation, which agrees to within $10^{-16}$.
One step. Moving every vector one unit against its gradient (a learning rate of 1, large, for illustration) lowers the batch loss from 0.646 to 0.497. The probabilities of the three positives go from 0.545, 0.615 and 0.429 to 0.583, 0.752 and 0.514. Query 3's tie is broken.
The other direction. The matrix can also be read by columns: for each passage, which of the three queries is its own? Dividing each column of the first table by its sum gives the diagonal 6/11, 4/7 and 6/13, and the loss $\tfrac{1}{3} \ln\big(\tfrac{11}{6} \cdot \tfrac{7}{4} \cdot \tfrac{13}{6}\big) = \tfrac{1}{3} \ln \tfrac{1001}{144} = 0.646$, equal to the row loss here by coincidence of the numbers. Some training scripts average the two directions. The script that trained this unit's pinned encoder does, for pairs without an explicit hard negative.
In-batch negatives
The worked batch scored every query against every passage. That is the device that makes contrastive training affordable, and it deserves its own count.
Encoding a passage is the expensive step: a forward pass through the transformer and, in training, a backward pass. If each of $B$ queries came with its own positive and its own $n$ negatives, a batch would need $B(1 + n)$ passage encodings. With in-batch negatives, the batch is $B$ pairs $(q_i, p_i^+)$, and the positive of every other pair serves as a negative for query $i$. The encoder runs on $B$ queries and $B$ passages, and one matrix product gives all $B^2$ scores. Each query has $n = B - 1$ negatives.
| Batch size $B$ | Texts encoded | Scores | Negatives per query | Texts encoded if each query had its own $B - 1$ negatives |
|---|---|---|---|---|
| 3 | 6 | 9 | 2 | 12 |
| 128 | 256 | 16,384 | 127 | 16,512 |
| 1,024 | 2,048 | 1,048,576 | 1,023 | 1,049,600 |
At a batch of 128, sharing saves a factor of 64.5 in encodings. Henderson et al. (2017, §4.4) describe the device for their reply-suggestion model, treating the responses of the other examples in the batch as negatives, and note that the negatives a given input sees change at every pass because the batches are reshuffled. DPR trained with a batch of 128. The model card of this unit's pinned encoder gives a batch of 1,024.
In-batch negatives have two properties that follow from the derivation of the optimal score.
They are drawn from the positives' distribution. A negative for query $i$ is the positive of some other training pair. So the noise distribution $\nu$ is the distribution of passages that appear as positives, not of all passages, and a passage that is the answer to many training queries is over-represented among the negatives. The optimal score subtracts $\ln \nu(p)$, so such popular passages are scored down. Henderson et al. (2017, §4.6) observed this bias in their system, which came to favor long, specific replies over common ones, and corrected it with an added score.
Some of them are not negatives at all. Nothing guarantees that the positive of another query is irrelevant to this one. On Cranfield the judgments let us count. Draw two different queries and one relevant abstract for each: the probability that the second query's abstract is also judged relevant to the first is 0.68%. That is small for one pair and adds up over a batch. In a batch of $B$ pairs there are $B(B-1)$ in-batch negatives, so the expected number of false ones is 0.38 at $B = 8$, 6.8 at $B = 32$ and 111 at $B = 128$, a little under one per query. Over 30 seeds of 2,000 random batches each, the share of batches with at least one false negative is 0.263 at $B = 8$ and 0.996 at $B = 32$.
Example (a false negative in a batch of three). Cranfield queries 1 and 2 both concern aeroelastic problems of high-speed aircraft, and they share relevant abstracts. Put the pairs (query 185, abstract 856), (query 1, abstract 184) and (query 2, abstract 12) in a batch. Abstract 12 is judged relevant to query 1 as well, and the encoder gives it a cosine of 0.595 with query 1, against 0.653 for query 1's own positive. At the temperature the pinned model was trained with, the loss assigns query 1 a probability of 0.763 for its positive and 0.237 for abstract 12, and its gradient pushes query 1 away from an abstract that answers it. The batch loss is 0.091, where the same batch with an unrelated third pair has a loss of 0.0005. What to do about such negatives is part of Hard Negatives and Distillation.
Temperature
With dot-product scores, as in DPR, the encoder can make a score as large as it likes by lengthening its vectors. With cosine scores it cannot: a cosine lies between $-1$ and $1$. That bound makes the loss unable to become small, by an amount that is easy to compute.
The floor. Let the scores be cosines divided by a temperature, $s_j = \cos(\vec{q}, \vec{p}_j) / \tau$. The best the encoder can do for one query is a cosine of 1 with the positive and $-1$ with every negative. Since $P_0$ increases with the positive's score and decreases with each negative's, no other arrangement gives a larger $P_0$, and so
$$\mathcal{L} \ge -\ln \frac{e^{1/\tau}}{e^{1/\tau} + n\, e^{-1/\tau}} = \ln\big( 1 + n\, e^{-2/\tau} \big) .$$
At $\tau = 1$, raw cosines, the floor is high:
| Negatives $n$ | $\tau$ | Floor $\ln(1 + n e^{-2/\tau})$ | Loss of a uniform guess, $\ln(n+1)$ |
|---|---|---|---|
| 2 | 1 | 0.240 | 1.099 |
| 127 | 1 | 2.90 | 4.852 |
| 1,023 | 1 | 4.94 | 6.931 |
| 1,023 | 0.5 | 2.98 | 6.931 |
| 1,023 | 0.2 | 0.0454 | 6.931 |
| 1,023 | 0.1 | 0.00000211 | 6.931 |
With 1,023 negatives and raw cosines, a perfect encoder still has a loss of 4.94. It can never be more confident than $P_0 = e^{-4.94}$, less than 1%, and since every negative keeps a share of the probability, every negative keeps pushing, however far away it already is. The floor is reached only if all 1,023 negatives sit at the single point opposite the query, and with in-batch negatives that cannot hold for every query of a batch at once.
Dividing by a small $\tau$ stretches the range of scores from $[-1, 1]$ to $[-1/\tau, 1/\tau]$ and removes the floor. The pinned model's training script multiplies cosines by a scale of 20, which is $\tau = 0.05$; its floor with 1,023 negatives is $1023\, e^{-40}$, about $4 \times 10^{-15}$. Chen et al. (2020) report for image representations that cosine scores need "an appropriately adjusted temperature", and Gao, Yao and Chen (2021, Appendix D and Table D.1) report for sentence embeddings that cosine with $\tau = 0.05$ did better than a dot product.
Three Cranfield pairs. Take query 185 with abstract 856, query 1 with abstract 184, and query 3 (on heat conduction in composite slabs) with abstract 5, each abstract judged relevant to its query. The pinned encoder gives these cosines:
| Abstract 856 | Abstract 184 | Abstract 5 | |
|---|---|---|---|
| Query 185 | 0.751 | 0.200 | 0.143 |
| Query 1 | 0.217 | 0.653 | 0.304 |
| Query 3 | 0.155 | 0.201 | 0.605 |
The encoder plainly has these right: each diagonal entry is at least twice any other in its row. Yet at $\tau = 1$ the loss of the batch is 0.814, and the probabilities of the three positives are 0.471, 0.425 and 0.434. Differences of half a unit between scores are not much to a softmax. At $\tau = 0.05$ the same cosines give a loss of 0.0005.
Temperature decides which negatives matter. The share of the total push on the negatives that goes to negative $j$ is $P_j / \sum_{k \ge 1} P_k = e^{s_j} / \sum_{k \ge 1} e^{s_k}$, a softmax over the negatives alone. As $\tau \to \infty$ all scores shrink to zero and the shares become equal, $1/n$ each. As $\tau \to 0$ the largest score dominates: if negative $h$ has the highest cosine and the next highest is lower by $\delta > 0$, every other share is at most $e^{-\delta/\tau}$ times that of $h$, which tends to zero, so the whole push goes to the single hardest negative.
The script measures this on Cranfield. For each of the 1,612 relevant query-abstract pairs it takes every abstract not judged relevant to the query as a negative, a median of 1,391 of them, and computes the shares at seven temperatures. The effective number of negatives is the reciprocal of the sum of the squared shares: it is $n$ when the shares are equal and 1 when one negative has it all.
| $\tau$ | Effective number of negatives (median) | Share of the hardest negative (median) | Mean loss |
|---|---|---|---|
| 1 | 1,371.7 | 0.001 | 6.98 |
| 0.5 | 1,317.9 | 0.002 | 6.74 |
| 0.2 | 987.4 | 0.005 | 6.11 |
| 0.1 | 319.2 | 0.024 | 5.36 |
| 0.05 | 18.1 | 0.185 | 5.20 |
| 0.02 | 1.6 | 0.770 | 9.25 |
| 0.01 | 1.1 | 0.969 | 17.75 |
At $\tau = 1$ all 1,391 negatives push about equally. At $\tau = 0.05$ the push is spread as if over about 18 equally weighted negatives, mostly the abstracts nearest the query. At $\tau = 0.01$ a single abstract takes 97% of it. The mean loss is lowest at 0.05, the temperature the encoder was trained with; that is what one would expect of scores fitted at that scale, though nothing here proves it. A very low temperature makes training depend on one negative per query, which is a risk when that negative may be a relevant abstract nobody judged.
How many negatives?
The optimal score does not depend on $n$. In the published results the number of negatives matters a great deal, and the gradient suggests why: a negative teaches in proportion to the probability $P_j$ the model gives it, and a few random negatives are usually too easy to be given any.
The script measures this with the pinned encoder as a fixed scorer at $\tau = 0.05$. For each of the 1,612 relevant pairs it draws $n$ negatives at random from the abstracts not judged relevant, computes the loss, the probability of the positive, and the length of the gradient with respect to the query vector, $\lvert \sum_j P_j \vec{p}_j - \vec{p}^{\,+} \rvert / \tau$, and averages over the pairs. The table gives the median over 30 seeds.
| Negatives $n$ | Mean loss | Mean $P_0$ | Mean gradient length | Share of pairs with $P_0 > 0.99$ | $\ln(n+1)$ | $\ln(n+1) - \text{loss}$ (the InfoNCE estimate) |
|---|---|---|---|---|---|---|
| 1 | 0.200 | 0.897 | 2.29 | 0.564 | 0.693 | 0.493 |
| 7 | 0.795 | 0.667 | 6.27 | 0.177 | 2.079 | 1.284 |
| 31 | 1.660 | 0.431 | 9.58 | 0.041 | 3.466 | 1.806 |
| 127 | 2.799 | 0.234 | 11.99 | 0.006 | 4.852 | 2.053 |
| 1,023 | 4.871 | 0.061 | 13.83 | 0.000 | 6.931 | 2.060 |
The seeds agree closely: at $n = 1$ the interquartile range of the mean loss over the 30 seeds is 0.022.
Three things are in the table.
- With one random negative, more than half the pairs teach nothing. For 56.4% of the pairs the encoder already gives the positive a probability above 0.99, so the gradient from that pair is almost zero. With 127 negatives that share is 0.6%. More negatives make it more likely that at least one of them is hard enough to matter.
- The gradient grows with the number of negatives. Its mean length is 2.29 with one negative and 13.83 with 1,023. Each step carries more correction.
- The distance below a uniform guess levels off. The last column is $\ln(n+1)$ minus the loss, the quantity that the mutual-information bound is about. It is not a bound on mutual information here, because this experiment does not meet the bound's condition: the negatives are drawn uniformly from the abstracts not judged relevant, not from the distribution of positives. It is simply how far below a uniform guess this encoder gets. With one negative it cannot exceed $\ln 2 = 0.693$, and it is 0.493. It rises with $n$ and levels off near 2.1. The plateau is the encoder's, not the cap's: at 1,023 negatives the cap is 6.931. For scale, if Cranfield's 1,612 relevant pairs are taken as equally likely, the mutual information between query and relevant abstract is 4.35 nats.
This is consistent with the reported effect of batch size, and is one mechanism those reports would fit; it is not shown to be the cause. DPR's ablation found retrieval accuracy rising as in-batch negatives went from 7 to 127, and offered a reason of its own, that a larger batch scores more question-passage pairs. Chen et al. (2020, §5.2) found for image representations that contrastive learning gains more from large batches than supervised learning does, and gave the reason that "larger batch sizes provide more negative examples".
Even 1,023 random negatives leave the mean $P_0$ at 0.061 on this collection. A likely reason is that Cranfield's abstracts are all about aerodynamics, so that many abstracts not judged relevant to a query are close to it. Whether those are hard negatives or unjudged positives is the question the next article takes up.
Noise-contrastive estimation: where the idea came from
The "NCE" in InfoNCE is noise-contrastive estimation, a method from statistics. Gutmann and Hyvärinen (2010) faced a different problem: fitting a probability model whose normalizing constant cannot be computed. Their idea was to generate artificial noise samples and train a logistic regression to tell data from noise, using the model's log-density inside the classifier. They showed that this gives a consistent estimator of the model's parameters and that the normalizing constant can be estimated as one more parameter. A hard problem of density estimation was turned into an easy problem of classification.
That version is a binary decision per sample: data or noise. The loss of this article is the other variant, a choice of one among $n + 1$. Ma and Collins (2018) analyze the two for conditional models and call them the binary and the ranking objectives. They prove that the ranking objective gives consistent estimates under weaker assumptions (their Theorem 3.1, stated formally in their §4): the binary objective needs the model's normalizer to be the same for every input, or the score function to be able to absorb it, and the ranking objective does not. For a retriever that is the natural choice in any case. What matters for a query is the order of the passages, and the ranking loss depends on nothing else.
The binary variant, simplified further, is the negative sampling that trains word vectors, and Word Embeddings derives it.
A toy training run
Everything so far used a finished encoder. To see the loss do its work, the script toytrain.py trains one.
What full-scale training takes, and what is done here. The pinned encoder's model card reports training on about 1.17 billion pairs for 100,000 steps at a batch of 1,024, on hardware built for the purpose. DPR was trained on some 59,000 questions for up to 40 epochs on eight graphics processors. Training on MS MARCO, the large passage-ranking collection that is among the pinned encoder's training data, is likewise out of reach of a laptop processor, and the effectiveness numbers for training at that scale are quoted in this unit from the papers that report them. What a laptop can do is twenty steps.
- The network. The six-layer MiniLM checkpoint that has had no sentence training (
nreimers/MiniLM-L6-H384-uncased, revision3276f0fac9d818781d7a1327b3ff818fc4e643c0), the starting point of the pinned encoder. Sentence and Passage Encoders showed that its pooled vectors retrieve almost nothing. All of its weights are trained, through mean pooling and length normalization. - The pairs. The title of a Cranfield abstract as the "query" and the abstract without its title as the "passage": 1,392 pairs. No Cranfield query and no relevance judgment is used.
- The training. Batches of 16 pairs, 20 steps, so 320 pairs are seen, with the loss of this article over in-batch negatives, the AdamW optimizer at a learning rate of $5 \times 10^{-5}$, and dropout left on as the checkpoint configures it. Titles are cut at 32 word pieces and bodies at 64.
- The test. The trained network embeds the 1,400 abstracts and the 225 Cranfield queries, cut at 64 word pieces, and abstracts are ranked by cosine. Before any training this gives an nDCG@10 of 0.084 and a recall@100 of 0.263. The previous article reports 0.034 and 0.168 for the same checkpoint, which reads 256 word pieces. The cut is the whole difference: the same evaluation run at 256 word pieces gives 0.034 and 0.168 again.
- Three conditions, 30 seeds each. All negatives: $\tau = 0.05$ and all 15 in-batch negatives. One negative: $\tau = 0.05$, with each query's loss computed against its positive and one of the other passages chosen at random. Raw cosine: $\tau = 1$ with all 15 negatives. The seeds are 0 to 29. A seed fixes the batches and the dropout generator, which are therefore the same in all three conditions, and that is what justifies comparing the conditions seed by seed; a second generator, seeded separately, picks the one negative kept in the second condition.
| Condition | Loss at step 1 | Loss, last 5 steps | nDCG@10, median [quartiles] | Recall@100, median [quartiles] |
|---|---|---|---|---|
| No training | 0.084 | 0.263 | ||
| All 15 negatives, $\tau = 0.05$ | 2.49 | 1.45 | 0.152 [0.147, 0.157] | 0.444 [0.438, 0.457] |
| One negative, $\tau = 0.05$ | 0.56 | 0.33 | 0.131 [0.127, 0.136] | 0.391 [0.374, 0.406] |
| All 15 negatives, $\tau = 1$ | 2.76 | 2.58 | 0.071 [0.066, 0.076] | 0.249 [0.236, 0.266] |
The losses are medians over the 30 seeds; the two loss columns of different rows are not comparable with each other, since a loss against one negative starts near $\ln 2$ and a loss against 15 starts near $\ln 16 = 2.77$.
Three results.
- In this run, twenty steps changed what the vectors are for. With all 15 negatives at $\tau = 0.05$, nDCG@10 goes from 0.084 to a median of 0.152 and recall@100 from 0.263 to 0.444. Every one of the 30 runs ends above the untrained network; the lowest scores 0.134 and the highest 0.166. The network has seen 320 title-abstract pairs and no query.
- In this run, more negatives per query helped, with the same data and the same number of steps. With one negative per query the median is 0.131. The comparison is paired, since both conditions see the same batches: the run with 15 negatives is ahead in all 30 seeds, by a median of 0.018 (the smallest difference is 0.006 and the largest 0.051). The two clouds of runs overlap, with the one-negative runs reaching 0.149 at best, so it is the pairing by seed that makes the difference plain.
- On raw cosines, twenty steps did harm. At $\tau = 1$ the loss moves from 2.76 to 2.58 in twenty steps, having started near $\ln 16 = 2.77$; its floor of $\ln(1 + 15 e^{-2}) = 1.11$ is far below and is not what stopped it. The vectors it produces retrieve worse than the untrained ones: a median nDCG@10 of 0.071, below the untrained 0.084 in 29 of the 30 seeds. A likely reason, which these runs do not measure, is the one the temperature section gives: at $\tau = 1$ the gradient is spread almost evenly over the negatives, so a few steps move the vectors without sorting them.
Three cautions about what this shows. The gains are large relative to the starting point and small in absolute terms: about 0.15 is far below BM25's 0.368 and the pinned encoder's 0.382. BM25 reads whole abstracts and the encoder 256 word pieces; scored as these runs are, at 64 word pieces, the pinned encoder gives 0.357. The distance from about 0.15 to 0.357, at the same cut and from the same starting checkpoint, is what the pinned encoder's full-scale training adds to twenty steps on 320 pairs. The pairs come from the same abstracts that are then searched, so the run adapts the network to this collection and says nothing about transfer to another. And 15 negatives against one is a narrow range; the reported benefits of large batches concern hundreds or thousands.
What is proved here, what is cited, and what the experiment shows
Proved or derived on the page. The posterior over which candidate is the positive, and the optimal score as a log density ratio; the bound $\ln(n+1) - \mathbb{E}[\mathcal{L}] \le \mathrm{KL}(\pi \,\|\, \nu)$ for one query, and from it the mutual-information bound in the case where negatives are independent draws from the distribution of positives; the gradient of the loss with respect to scores, vectors and a whole batch, and through length normalization; the floor $\ln(1 + n e^{-2/\tau})$ for cosine scores; the limits of the negatives' shares as the temperature goes to zero or infinity; the counts for in-batch negatives. The three-passage batch is exact arithmetic, checked two independent ways.
Cited, not proved. The mutual-information bound in the generality of van den Oord, Li and Vinyals (2018) and Poole et al. (2019), which covers scores that see all the candidates. The consistency results for noise-contrastive estimation (Gutmann and Hyvärinen, 2010) and for its binary and ranking variants (Ma and Collins, 2018). Every statement about how published models were trained or how they performed: the bias and the batch device in Henderson et al. (2017), the temperature and batch-size findings of Chen et al. (2020) and Gao, Yao and Chen (2021), which are for images and for sentence similarity, and the training settings of DPR and of the pinned encoder, the latter from its model card and training script.
Shown by the experiments, on Cranfield. How the loss, the probability of the positive, the gradient and the distance below a uniform guess change with the number of random negatives for one fixed encoder; how temperature concentrates the push; how often an in-batch negative is judged relevant; and that twenty steps of this loss turn a network that retrieves almost nothing into one that retrieves something, more with fifteen negatives than with one and not at all on raw cosines. These are observations of one small network on one small collection.
Worked problems
These problems are posed on small exact cases, to exercise each result of the article; the unit has no course assignments. The script recomputes each number under "Worked problems" or in the section the problem refers to.
Problem 1 (a loss by hand, and the effect of temperature). A query has four candidates with scores 2, 1, 0 and −1 before any scaling, the first being the positive. Compute the loss, the probability of the positive and the share of the push that falls on the hardest negative, at $\tau = 1$ and at $\tau = 1/2$.
Solution. At $\tau = 1$ the exponentials are $e^2 = 7.389$, $e = 2.718$, $1$ and $e^{-1} = 0.368$, with sum 11.475. So $P_0 = 7.389 / 11.475 = 0.644$ and the loss is $-\ln 0.644 = 0.440$. The three negatives hold $2.718 + 1 + 0.368 = 4.086$ of the sum, and the hardest holds $2.718 / 4.086 = 0.665$ of that. At $\tau = 1/2$ the scores double to 4, 2, 0, −2, with exponentials 54.598, 7.389, 1 and 0.135 and sum 63.123. Now $P_0 = 54.598 / 63.123 = 0.865$, the loss is 0.145, and the hardest negative's share is $7.389 / 8.524 = 0.867$. Halving the temperature made the same scores look more decisive and moved the push further onto the hardest negative, as the limits in the temperature section predict.
Problem 2 (the optimal score). A collection has two passages. For a certain query the positive is passage $a$ with probability 0.8 and passage $b$ with probability 0.2. Negatives are drawn from the two passages with equal probability, and there is one negative. What difference of scores $s(q, a) - s(q, b)$ minimizes the expected loss, and what is that loss?
Solution. By the result on the optimal score, $s^*(q, p) = \ln(\pi(p \mid q) / \nu(p)) + c(q)$, so the optimal difference is $\ln(0.8/0.5) - \ln(0.2/0.5) = \ln 4 = 1.386$. To check it directly, let $\delta$ be the difference. There are four cases for (positive, negative). When they are the same passage, probability $0.8 \times 0.5 + 0.2 \times 0.5 = 0.5$, the two scores are equal and the loss is $\ln 2$ whatever $\delta$ is. When the positive is $a$ and the negative $b$, probability 0.4, the loss is $\ln(1 + e^{-\delta})$. When the positive is $b$ and the negative $a$, probability 0.1, it is $\ln(1 + e^{\delta})$. The expected loss is $0.5 \ln 2 + 0.4 \ln(1 + e^{-\delta}) + 0.1 \ln(1 + e^{\delta})$. Setting its derivative to zero, $-0.4/(1 + e^{\delta}) + 0.1\, e^{\delta}/(1 + e^{\delta}) = 0$, gives $e^{\delta} = 4$. The script confirms the minimum numerically on a grid. At $\delta = \ln 4$ the expected loss is 0.597, against $\ln 2 = 0.693$ for equal scores, so the best possible score gets 0.096 below a uniform guess. The bound proved above says this distance cannot exceed $\mathrm{KL}(\pi \,\|\, \nu) = 0.8 \ln 1.6 + 0.2 \ln 0.4 = 0.193$, and it does not. With a single query there is no mutual information between query and passage to speak of, which is why the bound is stated here with the divergence.
Problem 3 (the temperature floor). An encoder with cosine scores is trained with a batch of 1,024 and in-batch negatives. What is the smallest possible loss at $\tau = 1$, and at the pinned model's $\tau = 0.05$? At $\tau = 1$, what is the largest probability the model can ever give a positive?
Solution. Each query has $n = 1{,}023$ negatives. The floor is $\ln(1 + n e^{-2/\tau})$. At $\tau = 1$: $e^{-2} = 0.1353$, $1 + 1023 \times 0.1353 = 139.4$, and $\ln 139.4 = 4.94$. The largest possible $P_0$ is $1/139.4$, about 0.007. A uniform guess has loss $\ln 1024 = 6.93$, so the whole range available to training is from 6.93 down to 4.94. At $\tau = 0.05$: $e^{-40}$ is about $4.2 \times 10^{-18}$, and the floor is $1023 \times e^{-40}$, about $4 \times 10^{-15}$: no restriction at all.
Problem 4 (one gradient step). In the three-passage batch, apply one step of gradient descent with learning rate 1 to query 3 alone, leaving everything else fixed. What are its new scores, and does its positive now win?
Solution. The batch loss is the mean of three row losses, and only row 3 depends on $\vec{q}_3$, so $\partial \mathcal{L} / \partial \vec{q}_3 = \tfrac{1}{3}(\tfrac{4}{7}, -\tfrac{1}{7})$. The step gives $\vec{q}_3' = (0, \ln 3) - (\tfrac{4}{21}, -\tfrac{1}{21}) = (-\tfrac{4}{21}, \ln 3 + \tfrac{1}{21})$. Its new scores are $\vec{q}_3' \cdot \vec{p}_1 = \ln 3 - \tfrac{3}{21}$, $\vec{q}_3' \cdot \vec{p}_2 = -\tfrac{4}{21}$ and $\vec{q}_3' \cdot \vec{p}_3 = \ln 3 + \tfrac{1}{21}$. The positive, passage 3, now scores higher than passage 1 by $\tfrac{4}{21} = 0.190$: the tie is broken in the right direction. The step lowered the score of the confusing negative by $\tfrac{3}{21}$ and raised the positive's by $\tfrac{1}{21}$, three times as much movement away from the negative as toward the positive, because the negative shares a coordinate with the positive and only the other coordinate can separate them.
What we have learned
The contrastive loss asks the encoder to pick each query's passage out of a set of candidates and charges it the negative logarithm of the probability it gave the right one. Its best possible score is the logarithm of how much more likely a passage is as this query's positive than as a negative, so the negatives define what is learned. Its gradient moves a query toward its positive and away from the model's own average of the candidates, with each negative weighted by the probability the model wrongly gave it. Taking negatives from the batch makes many negatives cheap, at the price of a bias against popular passages and the occasional negative that is in fact relevant. Cosine scores need a temperature, and the temperature decides how many negatives share the push.
The weights $P_j$ are also the loss's weakness. On Cranfield, with one random negative, more than half the training pairs produce almost no gradient, because a random negative is too easy. The next article, Hard Negatives and Distillation, is about finding negatives that are not.
References
- Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), PMLR 119, 1597–1607. record
- Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of EMNLP 2021, 6894–6910. record
- Gutmann, M., and Hyvärinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS 2010), PMLR 9, 297–304. record
- Henderson, M., Al-Rfou, R., Strope, B., Sung, Y.-H., Lukács, L., Guo, R., Kumar, S., Miklos, B., and Kurzweil, R. (2017). Efficient natural language response suggestion for Smart Reply. arXiv:1705.00652. record · 2 pages
- Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP 2020, 6769–6781. arXiv:2004.04906. record · 10 pages
- Ma, Z., and Collins, M. (2018). Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of EMNLP 2018, 3698–3707. record
- Poole, B., Ozair, S., van den Oord, A., Alemi, A. A., and Tucker, G. (2019). On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning (ICML 2019), PMLR 97, 5171–5180. record
- sentence-transformers (n.d.). sentence-transformers/all-MiniLM-L6-v2. Model card and
train_script.pyat revision1110a243fdf4706b3f48f1d95db1a4f5529b4d41. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 (accessed 17 November 2026). record · 5 pages - Sohn, K. (2016). Improved deep metric learning with multi-class N-pair loss objective. In Advances in Neural Information Processing Systems 29 (NIPS 2016), 1857–1865. record
- van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv:1807.03748. record