Two networks are trained on the same breast cancer data, 569 tumors described by 30 standardized measurements of their cell nuclei (Wolberg, Mangasarian, Street and Street, 1993). Both have ten hidden layers of 64 units and one output unit that gives the probability of malignancy. Both start from random weights scaled by the same principle, and both are trained by the same stochastic gradient descent for 50 passes through the data. The only difference is the function each hidden unit applies to its weighted sum. With the sigmoid function, the network's training loss after 50 epochs is 0.6609, essentially the 0.6603 achieved by ignoring the measurements and predicting the overall rate of malignancy, 37.3%, for every tumor: it has learned nothing. With the rectified linear function, $\max(0, z)$, the training loss is 0.0003. Over 30 random initializations, the middle half of the sigmoid runs ended between 0.6605 and 0.6614, and the middle half of the ReLU runs between 0.0002 and 0.0004.
That function is the activation function, $g$ in the neuron $a = g(\mathbf{w}^{\mathsf{T}}\mathbf{x} + b)$ of Introduction to Artificial Neural Networks. It has two jobs. It must be nonlinear, or a network of any depth collapses into a single linear model, as the introduction shows. And because networks are trained by gradient descent, with gradients computed by the chain rule through every layer (derived in The Multilayer Perceptron), its derivative must carry a usable signal. This article goes through the functions in common use, their derivatives, and what those derivatives do to training.
The step function
The perceptron of The Single-Layer Perceptron uses the step function, 1 for $z \geq 0$ and 0 otherwise. It is nonlinear, but its derivative is zero everywhere except at $z = 0$, where it does not exist. A small change to any weight changes no output, so gradient descent gets no information from it, and a hidden layer of step units cannot be trained by backpropagation. The perceptron rule sidesteps this for a single unit by updating directly from mistakes, but it has no way to assign blame to hidden units. Hastie, Tibshirani and Friedman (2009, §11.3) note that early neural network models used the step function and that "the step function is not smooth enough for optimization," which is why it was replaced by the sigmoid.
The sigmoid
The sigmoid or logistic function,
$$\sigma(z) = \frac{1}{1 + e^{-z}},$$
rises smoothly from 0 to 1, with $\sigma(0) = 0.5$ and $\sigma(-z) = 1 - \sigma(z)$. It is the link of logistic regression, and as an output unit it turns a weighted sum into a probability. Its derivative has a convenient form. Differentiating $(1 + e^{-z})^{-1}$,
$$\sigma'(z) = \frac{e^{-z}}{(1 + e^{-z})^2} = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = \sigma(z)\,\big(1 - \sigma(z)\big),$$
so the derivative is available from the output alone. It is largest at $z = 0$, where it equals $0.5 \times 0.5 = 0.25$, and it falls off quickly on both sides:
| $z$ | −3 | −1 | 0 | 1 | 3 | 5 | 10 |
|---|---|---|---|---|---|---|---|
| $\sigma(z)$ | 0.0474 | 0.2689 | 0.5000 | 0.7311 | 0.9526 | ||
| $\sigma'(z)$ | 0.0452 | 0.1966 | 0.2500 | 0.1966 | 0.0452 | 0.0066 | 0.000045 |
A unit whose weighted sum is far from zero is saturated: its output is pinned near 0 or 1, and its derivative is so small that the gradient flowing through it almost vanishes. Two other properties matter in hidden layers. The derivative is never more than 0.25, so every sigmoid layer a gradient passes through shrinks it by at least a factor of four (the next section makes this precise). And the outputs are always positive, with a mean near 0.5 rather than 0, which shifts the weighted sums of the next layer; Glorot and Bengio (2010) found that in deep sigmoid networks this mean drove the top hidden layer into saturation at the start of training.
The hyperbolic tangent
The hyperbolic tangent,
$$\tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}},$$
has the same S shape but runs from −1 to 1 and passes through the origin. It is a rescaled sigmoid. Multiplying the numerator and denominator by $e^{-z}$ gives $\tanh(z) = (1 - e^{-2z})/(1 + e^{-2z})$, and $2\sigma(2z) - 1 = 2/(1 + e^{-2z}) - 1$ is the same fraction, so
$$\tanh(z) = 2\,\sigma(2z) - 1,$$
which holds numerically to within $4 \times 10^{-16}$ on $[-6, 6]$. Its derivative is $\tanh'(z) = 1 - \tanh^2(z)$, equal to 1 at the origin, 0.4200 at $z = \pm 1$, 0.0099 at $\pm 3$ and $1.8 \times 10^{-4}$ at $\pm 5$. Being symmetric about the origin, tanh produces outputs that average near zero, which LeCun, Bottou, Orr and Müller (1998, §4.4) give as the reason that "symmetric sigmoids such as hyperbolic tangent often converge faster than the standard logistic function", the same reason inputs should be centered. Its derivative is up to four times larger than the sigmoid's. But it saturates just the same: for large $|z|$ the derivative of tanh falls faster than the sigmoid's.
Vanishing gradients
Backpropagation, derived in The Multilayer Perceptron, computes for each layer $l$ the vector $\boldsymbol{\delta}^{(l)} = \partial L / \partial \mathbf{z}^{(l)}$, the gradient of the loss with respect to that layer's weighted sums, from the layer above:
$$\boldsymbol{\delta}^{(l)} = \big(W^{(l+1)}\,\boldsymbol{\delta}^{(l+1)}\big) \odot g'(\mathbf{z}^{(l)}),$$
where $\odot$ multiplies entry by entry and $W^{(l+1)}$ is the matrix of weights from layer $l$ to layer $l+1$. The gradient for layer $l$'s weights is its input times $\boldsymbol{\delta}^{(l)}$. Unrolled over a stack of layers, the gradient reaching the first layer is a product: one weight matrix and one diagonal matrix of derivatives $g'$ for every layer above it. If those factors are typically smaller than 1 in size, the product shrinks geometrically with depth and the early layers barely learn; if larger, it grows and training becomes unstable. These are the vanishing and exploding gradient problems, identified for recurrent networks by Bengio, Simard and Frasconi (1994) and studied for deep feed-forward networks by Glorot and Bengio (2010). For the sigmoid, every derivative factor is at most 0.25, and ten of them multiply to at most $0.25^{10} \approx 9.5 \times 10^{-7}$, before the weights are counted.
Measured layer by layer
The ten-layer network of the opening was set up with three activations and two ways of drawing the initial weights. The uniform scheme draws each weight from $U[-1/\sqrt{n}, 1/\sqrt{n}]$, where $n$ is the number of inputs to the unit, a common heuristic that Glorot and Bengio (2010, eq. 1) used as their baseline. The second scheme is matched to the activation and is explained below. For each of 30 random initializations, the gradient of the cross-entropy over all 569 tumors was computed for every layer's weights before any training. The medians of their sizes (Frobenius norms):
| Activation, initialization | Layer 1 | Layer 10 | Layer 1 / layer 10 | Activation SD, layer 1 | Activation SD, layer 10 |
|---|---|---|---|---|---|
| sigmoid, uniform | $3.8 \times 10^{-9}$ | 0.065 | $5.8 \times 10^{-8}$ | 0.126 | 0.072 |
| sigmoid, Glorot | $9.1 \times 10^{-7}$ | 0.192 | $4.7 \times 10^{-6}$ | 0.163 | 0.122 |
| tanh, uniform | 0.0037 | 0.0052 | 0.71 | 0.423 | 0.0027 |
| tanh, Glorot | 0.554 | 0.556 | 1.00 | 0.515 | 0.213 |
| ReLU, uniform | $5.2 \times 10^{-5}$ | $6.4 \times 10^{-5}$ | 0.81 | 0.348 | 0.0001 |
| ReLU, He | 0.404 | 1.25 | 0.32 | 0.857 | 0.715 |
The sigmoid rows show the vanishing gradient in its pure form. From layer 10 down to layer 1, the gradient falls by a factor of about 3.9 per layer with Glorot's initialization, close to the factor of 4 that the sigmoid's maximum derivative imposes, and the first layer's gradient is five millionths of the tenth's. No initialization fixes this, because it comes from the activation's derivative.
The tanh and ReLU rows with the uniform scheme show a second, different failure. The gradients do not shrink from layer to layer, but all of them are small, and the reason is visible in the last two columns: the signal itself fades on the way forward. The standard deviation of the activations falls from 0.42 in the first tanh layer to 0.0027 in the tenth, and for ReLU from 0.35 to 0.0001. Each layer multiplies the variance of its inputs by roughly $n\,\mathrm{Var}(w)$, and the uniform scheme gives $\mathrm{Var}(w) = 1/(3n)$, so each tanh layer shrinks it by a factor of about three, and each ReLU layer, which also discards the negative half, by about six; Glorot and Bengio (2010, eq. 15) point out exactly this property of the heuristic.
Initialization matched to the activation
Glorot and Bengio (2010) derived the variance that keeps both the forward activations and the backward gradients at a constant scale through a network whose units behave linearly near zero, as tanh does: $\mathrm{Var}(w) = 2/(n_{\text{in}} + n_{\text{out}})$, a compromise between $1/n_{\text{in}}$ for the forward pass and $1/n_{\text{out}}$ for the backward one. Their normalized initialization draws weights from
$$U\!\left[-\frac{\sqrt{6}}{\sqrt{n_{\text{in}} + n_{\text{out}}}},\; \frac{\sqrt{6}}{\sqrt{n_{\text{in}} + n_{\text{out}}}}\right]$$
(their eq. 16), now usually called Glorot or Xavier initialization. A ReLU passes only the positive half of its input, which halves the variance, so He, Zhang, Ren and Sun (2015) derived the correction for rectifiers: weights drawn from a normal distribution with mean zero and standard deviation $\sqrt{2/n_{\text{in}}}$, called He initialization. In the table, tanh with Glorot's scheme has nearly equal gradients in every layer (ratio 1.00), and ReLU with He's keeps them within a factor of three. The sigmoid row with Glorot's scheme is still hopeless, because the scheme assumes a unit with slope 1 at the origin, and the sigmoid's slope there is 0.25.
What this does to training
The same networks were then trained by stochastic gradient descent (batches of 32, learning rate 0.05) for 50 epochs from the same 30 initializations, each with its matched scheme. The training loss, median and interquartile range:
| Activation | Epoch 1 | Epoch 10 | Epoch 50 |
|---|---|---|---|
| sigmoid, Glorot | 0.6609 (0.6606–0.6619) | 0.6609 (0.6604–0.6622) | 0.6609 (0.6605–0.6614) |
| tanh, Glorot | 0.1054 (0.0990–0.1151) | 0.0394 (0.0367–0.0431) | 0.0020 (0.0013–0.0102) |
| ReLU, He | 0.1673 (0.1369–0.1952) | 0.0190 (0.0136–0.0244) | 0.0003 (0.0002–0.0004) |
The sigmoid network sits at the loss of the constant prediction for the whole run. Its output layer learns the base rate quickly, and the layers below it receive gradients a million times too small to move. The tanh and ReLU networks fit the training tumors almost perfectly. (Fitting the training data this closely is not the goal of a real analysis, where a network this large overfits; The Multilayer Perceptron shows how validation data and early stopping deal with that. Here the training loss only measures whether learning happens.) Ten layers is not deep by modern standards, and the sigmoid's failure there is why it has disappeared from the hidden layers of deep networks; James, Witten, Hastie and Tibshirani (2021, §10.1) note that the sigmoid was favored in early networks and that "the preferred choice in modern neural networks is the ReLU."
The rectified linear unit
The rectified linear unit (ReLU) is
$$\operatorname{ReLU}(z) = \max(0, z), \qquad \operatorname{ReLU}'(z) = \begin{cases} 1 & z > 0, \\ 0 & z < 0. \end{cases}$$
At $z = 0$ the derivative does not exist; implementations use 0 or 1, and Nair and Hinton (2010, §4), who brought the function into wide use as a replacement for binary units in deep networks, took it to be 0 for $z \leq 0$. The choice almost never matters, because a weighted sum is rarely exactly zero.
ReLU's advantages follow from its derivative. A unit that is active passes the gradient back unchanged, with a factor of exactly 1 rather than at most 0.25, so a path through active units does not shrink the gradient through the activation. The function is a comparison, far cheaper than an exponential. And it is exactly zero for half its inputs, so a layer's activations are sparse: for a given input, many units are off. James et al. (2021, §10.1) also note that a ReLU "can be computed and stored more efficiently than a sigmoid activation."
This does not mean ReLU networks are free of gradient problems, a point sometimes overstated. The weights still multiply the gradient at every layer, and the uniform row of the table shows a ReLU network whose gradients are all below $10^{-4}$ because its initialization shrinks the forward signal. The derivative of 1 removes the activation's contribution to vanishing gradients, not the weights'; He initialization addresses the second.
Dying ReLUs
ReLU's zero derivative for negative inputs has a cost of its own. If a unit's weighted sum is negative for every training example, its output is always 0, its derivative is always 0, and no gradient ever reaches its incoming weights: the unit is dead, and nothing in gradient descent can revive it. If every unit in a layer dies, no gradient passes that layer at all, and nothing below it can learn either. A common cause is a learning rate so large that one update throws a unit's bias or weights far into the negative region.
To see this, a network with three hidden layers of 64 ReLU units was trained on the 569 tumors for 20 epochs from 30 He initializations, once with a sensible learning rate of 0.1 and once with a deliberately large one of 1.0. Each run was classed as diverged (the weights overflowed to infinity), collapsed (some hidden layer with all 64 units inactive on every tumor), or trained:
| Learning rate, activation | Diverged | Collapsed | Trained | Units inactive on every tumor (trained runs) |
|---|---|---|---|---|
| 0.1, ReLU | 0 | 0 | 30 | median 1 of 192, max 4 |
| 1.0, ReLU | 15 | 5 | 10 | median 4, max 25 |
| 1.0, leaky ReLU | 19 | 0 | 11 | median 11, max 26 |
At the sensible learning rate, most of the few inactive units were already inactive at initialization (median 1, max 4), and training added at most two to any run. At learning rate 1.0, half the ReLU runs blew up, and 5 collapsed: a whole layer died, and their training losses froze between 0.6604 and 0.6649, at the loss of predicting the base rate. The learning rate is the real problem here, and lowering it is the fix; but the collapsed runs show the specific failure that the zero derivative allows.
Leaky ReLU
The leaky ReLU gives negative inputs a small slope $\alpha$ instead of zero:
$$\operatorname{LeakyReLU}(z) = \begin{cases} z & z > 0, \\ \alpha z & z \leq 0, \end{cases} \qquad \text{equivalently } \max(\alpha z, z) \text{ for } 0 < \alpha < 1,$$
with derivative 1 or $\alpha$. A common choice is $\alpha = 0.01$ (Maas, Hannun and Ng, 2013); others, such as 0.1 or a learned $\alpha$, are also used. (The form $\max(\alpha z, z)$ is only correct when $\alpha < 1$; for larger $\alpha$ it would choose the wrong branch.) Because the derivative is never zero, a unit whose weighted sum is negative on every input still receives a gradient and can recover. In the table, leaky ReLU at the excessive learning rate never collapsed, although it diverged just as often, and its trained runs still had units negative on every tumor, 11 in the median, all still able to learn at a reduced rate. The leak removes the trap; it does not make a bad learning rate good.
GELU
The Gaussian error linear unit (GELU) of Hendrycks and Gimpel (2016) weights its input by the probability that a standard normal variable falls below it:
$$\operatorname{GELU}(z) = z\,\Phi(z), \qquad \operatorname{GELU}'(z) = \Phi(z) + z\,\varphi(z),$$
where $\Phi$ and $\varphi$ are the standard normal distribution function and density. In the authors' words, it "weights inputs by their value, rather than gates inputs by their sign as in ReLUs." For large positive $z$, $\Phi(z) \approx 1$ and GELU behaves like the identity; for large negative $z$ it goes to zero. In between it is smooth and, unlike every function so far, not monotone: it dips to a minimum of −0.1700 at $z = -0.752$, so its derivative is negative for every $z$ below −0.752, reaching its lowest value, −0.1289, at $z = -\sqrt{2}$, and rises slightly above 1 on the positive side, peaking at 1.1289 at $z = \sqrt{2}$. At the sample points of the earlier table, GELU takes the values −0.0040, −0.1587, 0, 0.8413 and 2.9960.
Hendrycks and Gimpel give two approximations used in practice: $0.5\,z\,\big(1 + \tanh[\sqrt{2/\pi}\,(z + 0.044715\,z^3)]\big)$, which is within $4.7 \times 10^{-4}$ of the exact function on $[-6, 6]$, and the cruder $z\,\sigma(1.702\,z)$, within 0.0203. Their paper reports that GELU matched or improved on ReLU across their vision, language and speech experiments, and its 2023 revision notes that GELU came to be used in BERT and GPT, "becoming the default activation for state-of-the-art Transformers." The Library's note on Softmax Linear Units describes an Anthropic experiment that replaced GELU in transformer MLP layers to make neurons easier to interpret. The feed-forward layers of the small GPT in Assembling the Transformer Block use ReLU, as the original Transformer did.
Softmax
The functions so far act on one weighted sum at a time. The softmax acts on a whole vector $\mathbf{z} = (z_1, \ldots, z_K)$, one score per class, and returns a probability distribution:
$$p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}, \qquad k = 1, \ldots, K.$$
The exponential makes every $p_k$ positive, the division makes them sum to one, and larger scores get larger probabilities. For $\mathbf{z} = (0, 1, 2)$ the exponentials are 1, 2.7183 and 7.3891, summing to 11.1073, and the probabilities are $(0.0900, 0.2447, 0.6652)$. The largest score gets the most probability but not all of it. It is the output layer of a network for $K$ classes and the link of multinomial logistic regression (James et al., 2021, eq. 10.13; Hastie et al., 2009, eq. 11.6), and it turns attention scores into weights in Building Self-Attention.
Three properties follow from the definition.
Only differences matter. Adding the same constant $c$ to every score multiplies the numerator and the denominator by $e^c$: $(5, 6, 7)$ gives the same probabilities as $(0, 1, 2)$. This is what makes softmax computable. Evaluated as written, $(1000, 1001, 1002)$ overflows, since $e^{1000}$ exceeds the largest double-precision number, and the result is undefined (NaN); subtracting the largest score first, so the exponentials are of $(-2, -1, 0)$, gives the correct $(0.0900, 0.2447, 0.6652)$. Every practical implementation does this.
Scale sets confidence. Dividing the scores by a temperature $T$ before the softmax sharpens the distribution when $T < 1$ and flattens it when $T > 1$: for $(0, 1, 2)$, $T = 0.5$ gives $(0.0159, 0.1173, 0.8668)$ and $T = 2$ gives $(0.1863, 0.3072, 0.5065)$.
With two classes it is the sigmoid. For $K = 2$, $p_1 = e^{z_1}/(e^{z_1} + e^{z_2}) = 1/(1 + e^{-(z_1 - z_2)}) = \sigma(z_1 - z_2)$, so a two-output softmax is a one-output sigmoid applied to the difference of the scores.
The derivative, and why it pairs with cross-entropy
Differentiating $p_i = e^{z_i}/S$, with $S = \sum_j e^{z_j}$, by the quotient rule gives $\partial p_i/\partial z_i = p_i(1 - p_i)$ and $\partial p_i/\partial z_j = -p_i p_j$ for $j \neq i$; in matrix form the Jacobian is $\operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^{\mathsf{T}}$. At $\mathbf{z} = (0, 1, 2)$:
$$\frac{\partial \mathbf{p}}{\partial \mathbf{z}} = \begin{pmatrix} 0.0819 & -0.0220 & -0.0599 \\ -0.0220 & 0.1848 & -0.1628 \\ -0.0599 & -0.1628 & 0.2227 \end{pmatrix}.$$
Each row sums to zero, because the probabilities always sum to one. The softmax is almost always followed by the cross-entropy loss, $L = -\ln p_y$ for true class $y$ (see Loss Functions for Classification). Writing $L = -z_y + \ln \sum_j e^{z_j}$ and differentiating,
$$\frac{\partial L}{\partial z_k} = -[k = y] + \frac{e^{z_k}}{\sum_j e^{z_j}} = p_k - [k = y],$$
so the gradient of the loss with respect to the scores is $\mathbf{p} - \mathbf{y}$, with $\mathbf{y}$ the one-hot vector of the true class. If the true class is the third, the loss is $-\ln 0.6652 = 0.4076$ and the gradient is $(0.0900, 0.2447, -0.3348)$, matching central finite differences to four decimal places. The same simplification occurs for a sigmoid output with the binary cross-entropy, where the gradient is $p - y$ (Logistic Regression derives it). The derivative $p_k(1 - p_k)$ that saturates on its own cancels against the logarithm in the loss, so a confidently wrong output produces a large gradient, not a vanishing one. Glorot and Bengio (2010, §4.1) found that the cross-entropy with softmax outputs "worked much better (for classification problems) than the quadratic cost which was traditionally used," with fewer plateaus during training.
Choosing an activation
The output layer's activation is fixed by what the network predicts, together with the loss it is trained on:
| Task | Output activation | Loss | Gradient at the output |
|---|---|---|---|
| Numerical outcome | identity, $g(z) = z$ | squared error | $\hat{y} - y$ |
| Two classes | sigmoid | binary cross-entropy | $p - y$ |
| $K$ classes | softmax | cross-entropy | $\mathbf{p} - \mathbf{y}$ |
For hidden layers the choice is less fixed. ReLU is the usual default for feed-forward networks and is scikit-learn's default; tanh trains well in shallow networks and remains common inside recurrent networks, as the sigmoid does in the gates of some recurrent architectures; leaky ReLU and its relatives guard against dead units; GELU is standard in transformer language models. In every case the initialization should match: Glorot for tanh and sigmoid-like units, He for ReLU and its variants.
| Function | Range | Derivative | Largest derivative | Notes |
|---|---|---|---|---|
| step | {0, 1} | 0 (undefined at 0) | none | no gradient; perceptron only |
| sigmoid | (0, 1) | $\sigma(1 - \sigma)$ | 0.25 | saturates; not centered; binary outputs |
| tanh | (−1, 1) | $1 - \tanh^2$ | 1 | saturates; centered |
| ReLU | [0, ∞) | 0 or 1 | 1 | cheap, sparse; units can die |
| leaky ReLU | (−∞, ∞) | $\alpha$ or 1 | 1 | no dead units |
| GELU | [−0.17, ∞) | $\Phi + z\varphi$ | 1.13 | smooth, not monotone; transformers |
| softmax | (0, 1), sums to 1 | $\operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^{\mathsf{T}}$ | multiclass outputs |
In Python
The functions and their derivatives in NumPy, with the softmax computed stably:
import numpy as np
from scipy.special import erf
def sigmoid(z):
return 1 / (1 + np.exp(-z)) # derivative: s * (1 - s)
def relu(z):
return np.maximum(z, 0) # derivative: (z > 0)
def leaky_relu(z, alpha=0.01):
return np.where(z > 0, z, alpha * z)
def gelu(z):
return 0.5 * z * (1 + erf(z / np.sqrt(2)))
def softmax(Z):
Z = Z - Z.max(axis=-1, keepdims=True) # only differences matter
E = np.exp(Z)
return E / E.sum(axis=-1, keepdims=True)
print(softmax(np.array([1000.0, 1001.0, 1002.0]))) # [0.09 0.2447 0.6652]
This sigmoid can overflow for large negative inputs; the version in the site's network code splits positive and negative inputs to avoid it. In scikit-learn, MLPClassifier and MLPRegressor take activation="relu" (the default), "tanh", "logistic" or "identity" for the hidden layers and choose the output activation from the task. The numbers in this article are computed by tools/experiments/data-science/activation-functions.py in the site's repository, using the network code in tools/experiments/data-science/nnet.py.
References
- Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. Cited by attribution; not read for this article. record
- Glorot, X., and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9, 249–256. record
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. §§11.3–11.5. record · 23 pages
- He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1026–1034). arXiv:1502.01852. record
- Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (GELUs). arXiv:1606.08415. Read in revision 5 (2023). record
- James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. Ch. 10. record · 30 pages
- LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. (1998). Efficient BackProp. In G. B. Orr and K.-R. Müller (Eds.), Neural Networks: Tricks of the Trade (Lecture Notes in Computer Science 1524, pp. 9–50). Springer. record
- Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing. Cited by attribution; not read for this article. record
- Nair, V., and Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (pp. 807–814). record · 2 pages
- Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B record · 14 pages