Two hundred points in the plane form two interleaving crescents, the "two moons" of scikit-learn's make_moons, with enough noise that the crescents overlap a little. No straight line separates them, so logistic regression, whose boundary is a line, classifies a median of 86.1% of fresh test points correctly over 30 simulated datasets. A network with one hidden layer of eight units, trained on the same points, bends its boundary around the crescents and classifies 93.6% correctly. On the breast cancer biopsies used throughout this collection, the same kind of network does no better than logistic regression, and by some measures slightly worse. Both results are typical, and this article explains both.
A multilayer perceptron (MLP) is a feed-forward network with one or more hidden layers of units between the inputs and the output, each unit applying a nonlinear activation function to a weighted sum of the layer before (Introduction to Artificial Neural Networks describes the structure). The name is historical. The units are not perceptrons, whose step activation gives no gradient, and the network is not trained by the perceptron rule of The Single-Layer Perceptron. It is trained by gradient descent, with the gradients computed by backpropagation. This article derives backpropagation, checks it numerically, builds a network in NumPy, and compares it with logistic regression and with scikit-learn's implementation on real data.
The forward pass
Write the training data as a matrix $X$ with one row per example and one column per input, $n$ rows and $p$ columns. A network with $L$ layers of weights has, for each layer $l = 1, \ldots, L$, a weight matrix $W^{(l)}$ of shape $n_{l-1} \times n_l$, whose column $j$ holds the incoming weights of unit $j$, and a bias vector $\mathbf{b}^{(l)}$ of length $n_l$. Here $n_0 = p$ and $n_l$ is the number of units in layer $l$. Starting from $A^{(0)} = X$, each layer computes
$$Z^{(l)} = A^{(l-1)} W^{(l)} + \mathbf{1}\,\mathbf{b}^{(l)\mathsf{T}}, \qquad A^{(l)} = g\big(Z^{(l)}\big),$$
where $\mathbf{1}$ is a column of $n$ ones (adding the bias to every row) and the activation $g$ is applied to every entry. Row $i$ of $Z^{(l)}$ holds the weighted sums of layer $l$ for example $i$, and row $i$ of $A^{(l)}$ its activations. The last layer replaces $g$ with the output function: a sigmoid for two classes, giving one probability per example, a softmax for $K$ classes, giving a row of $K$ probabilities, or the identity for a numerical outcome (see Nonlinear Activation Functions). Computing every layer in turn, from the inputs to the output, is the forward pass. Processing all examples as rows of one matrix turns the work into a few matrix products, which is how every practical implementation runs.
The loss
Training chooses the weights and biases, collected into one long vector $\boldsymbol{\theta}$, to make a loss small on the training data. For two classes with labels $y_i \in \{0, 1\}$ and predicted probabilities $p_i$, the usual loss is the mean cross-entropy,
$$L(\boldsymbol{\theta}) = -\frac{1}{n}\sum_{i=1}^{n} \big[\,y_i \ln p_i + (1 - y_i) \ln (1 - p_i)\,\big],$$
the negative log-likelihood of logistic regression divided by $n$. For $K$ classes it is $-\frac{1}{n}\sum_i \ln p_{i, y_i}$, the mean negative log of the probability given to the true class, and for a numerical outcome it is usually the mean squared error. Loss Functions for Classification and Loss Functions for Regression discuss these criteria; Hastie, Tibshirani and Friedman (2009, §11.4) and James, Witten, Hastie and Tibshirani (2021, §10.2) use the same ones for networks. With a softmax output and the cross-entropy, the output layer of a network is a multinomial logistic regression whose inputs are the last hidden layer's activations.
Gradient descent
The loss of a network with hidden layers has no closed-form minimum, and unlike the log-likelihood of logistic regression it is not convex: it can have many local minima and flat regions, and a hidden layer's units can be permuted without changing the function, so there are always many equivalent solutions (James et al., 2021, §10.7). Networks are fitted by gradient descent. The gradient $\nabla L(\boldsymbol{\theta})$, the vector of partial derivatives of the loss with respect to every parameter, points in the direction in which the loss increases fastest. A small step the other way,
$$\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta\, \nabla L(\boldsymbol{\theta}),$$
decreases the loss if the learning rate $\eta$ is small enough (James et al., 2021, eq. 10.27). Repeated, the steps descend until the gradient is nearly zero, at a minimum or a flat region. Too small a learning rate makes progress slow; too large a one overshoots and can diverge, as the experiment with dying units in Nonlinear Activation Functions shows.
Computing the exact gradient needs a pass over all $n$ examples. Stochastic gradient descent instead estimates it from a small random minibatch of examples, takes a step, and moves on to the next minibatch; one pass through the whole training set in minibatches is an epoch (James et al., 2021, §10.7.2). The steps are noisier but far cheaper, so many more of them can be taken. The network in this article uses minibatches of 32, shuffled afresh each epoch, and a fixed learning rate. Refinements such as momentum and the Adam optimizer, which adapt the step to the history of the gradients, are what most libraries use; The Bigram Language Model trains with Adam in PyTorch.
Backpropagation
Gradient descent needs the derivative of the loss with respect to every weight in every layer. Backpropagation computes all of them with one pass backward through the network, using the chain rule and the quantities already stored by the forward pass. Rumelhart, Hinton and Williams (1986) presented it as "a new learning procedure" that "repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector"; Hastie et al. (2009, §11.4) note its earlier description by Werbos (1974).
One unit at a time
Consider one example and a unit $j$ in layer $l$, with weighted sum $z_j^{(l)} = \sum_i a_i^{(l-1)} w_{ij}^{(l)} + b_j^{(l)}$ and activation $a_j^{(l)} = g(z_j^{(l)})$. Define the unit's error
$$\delta_j^{(l)} = \frac{\partial L}{\partial z_j^{(l)}},$$
how much the loss would change per unit change in the unit's weighted sum. Two applications of the chain rule give everything else.
From errors to weight gradients. The weight $w_{ij}^{(l)}$ affects the loss only through $z_j^{(l)}$, and $\partial z_j^{(l)} / \partial w_{ij}^{(l)} = a_i^{(l-1)}$, so
$$\frac{\partial L}{\partial w_{ij}^{(l)}} = \delta_j^{(l)}\, a_i^{(l-1)}, \qquad \frac{\partial L}{\partial b_j^{(l)}} = \delta_j^{(l)}.$$
The gradient of a weight is the error at its destination times the activation at its source.
From errors in one layer to errors in the layer below. The weighted sum $z_i^{(l-1)}$ of unit $i$ in the layer below affects the loss through its activation $a_i^{(l-1)}$, which feeds every unit $j$ of layer $l$. Summing over those paths,
$$\delta_i^{(l-1)} = \sum_j \frac{\partial L}{\partial z_j^{(l)}}\,\frac{\partial z_j^{(l)}}{\partial a_i^{(l-1)}}\,\frac{\partial a_i^{(l-1)}}{\partial z_i^{(l-1)}} = \Big(\sum_j w_{ij}^{(l)}\,\delta_j^{(l)}\Big)\, g'\big(z_i^{(l-1)}\big).$$
Each unit's error is the weighted sum of the errors of the units it feeds, using the same weights as the forward pass but run backward, times the slope of its own activation. These are the back-propagation equations of Hastie et al. (2009, eq. 11.15).
At the output. The recursion needs a starting point. For a sigmoid output with the cross-entropy loss, $L = -[y \ln p + (1 - y)\ln(1 - p)]$ with $p = \sigma(z)$, and since $\sigma'(z) = p(1 - p)$,
$$\delta^{(L)} = \frac{\partial L}{\partial p}\,\frac{\partial p}{\partial z} = \left(-\frac{y}{p} + \frac{1 - y}{1 - p}\right) p\,(1 - p) = p - y.$$
For a softmax output with the cross-entropy, the same cancellation gives the vector $\mathbf{p} - \mathbf{y}$, with $\mathbf{y}$ the one-hot vector of the true class, and for an identity output with the loss $\tfrac12(\hat{y} - y)^2$ it gives $\hat{y} - y$ (the derivations are in Nonlinear Activation Functions and Logistic Regression). In all three, the output error is simply prediction minus target.
In matrix form
For a minibatch of $n$ examples and the mean loss, the same equations become matrix products. Let $\Delta^{(l)}$ be the $n \times n_l$ matrix of errors, row $i$ for example $i$. Then
$$\Delta^{(L)} = \tfrac{1}{n}\,(P - Y), \qquad \frac{\partial L}{\partial W^{(l)}} = A^{(l-1)\mathsf{T}}\,\Delta^{(l)}, \qquad \frac{\partial L}{\partial \mathbf{b}^{(l)}} = \Delta^{(l)\mathsf{T}}\,\mathbf{1},$$
$$\Delta^{(l-1)} = \big(\Delta^{(l)}\, W^{(l)\mathsf{T}}\big) \odot g'\big(Z^{(l-1)}\big),$$
where $P$ and $Y$ hold the predictions and targets, the factor $1/n$ comes from averaging the loss, the bias gradient sums each column of $\Delta^{(l)}$ over the examples, and $\odot$ multiplies entry by entry. The matrix $A^{(l-1)\mathsf{T}}\Delta^{(l)}$ has the shape of $W^{(l)}$, and its entry $(i, j)$ is $\sum_{\text{examples}} a_i^{(l-1)} \delta_j^{(l)}$, the one-unit formula summed over the minibatch.
The algorithm is therefore: run the forward pass, storing every $Z^{(l)}$ and $A^{(l)}$; compute $\Delta^{(L)}$ at the output; run the recursion down to layer 1, computing each layer's weight and bias gradients as its error matrix becomes available; then update all the weights with one gradient descent step. The backward pass costs about the same as the forward pass, a few matrix products per layer, which is why gradients for networks with millions of weights are affordable. The backward pass works from the output toward the inputs, so the gradients near the output are computed first; the update itself is applied to all layers together, once every gradient is known.
Worked by hand
Take a network with two inputs, two ReLU hidden units and a sigmoid output, and one example: tumor 0 of the breast cancer data (Wolberg, Mangasarian, Street and Street, 1993), malignant, so $y = 1$, with its mean radius and mean texture standardized over all 569 tumors, $\mathbf{x} = (1.0971, -2.0733)$. The weights are chosen by hand: hidden unit 1 has weights 1 and 0.5 from the two inputs and bias 0; hidden unit 2 has weights −0.5 and 1 and bias −1; the output has weights 1.5 and −2 from the hidden units and bias −0.5.
Forward. The hidden weighted sums are
$$z_1 = 1(1.0971) + 0.5(-2.0733) + 0 = 0.0604, \qquad z_2 = -0.5(1.0971) + 1(-2.0733) - 1 = -3.6219.$$
The ReLU passes the first and zeroes the second: $a_1 = 0.0604$, $a_2 = 0$. The output's weighted sum is $z = 1.5(0.0604) - 2(0) - 0.5 = -0.4094$, so $p = \sigma(-0.4094) = 0.3991$, and the loss is $-\ln 0.3991 = 0.9187$. The network gives this malignant tumor a 40% chance of malignancy.
Backward. The output error is $\delta = p - y = -0.6009$. The output weights' gradients are this error times the hidden activations, $(-0.6009 \times 0.0604,\; -0.6009 \times 0) = (-0.0363, 0)$, and the output bias's is −0.6009. The hidden errors are the output error times the output weights times the ReLU slopes, which are 1 for the active unit and 0 for the inactive one:
$$\delta_1 = 1.5 \times (-0.6009) \times 1 = -0.9014, \qquad \delta_2 = -2 \times (-0.6009) \times 0 = 0.$$
The hidden weights' gradients are these errors times the inputs: for hidden unit 1, $(1.0971 \times -0.9014,\; -2.0733 \times -0.9014) = (-0.9889, 1.8689)$, and bias gradient −0.9014; for hidden unit 2, all zero. A negative gradient means increasing the weight lowers the loss, so the step will raise the weight from radius to unit 1 and lower the weight from texture, both of which raise $z_1$ for this smooth, large tumor, and raise the output weight from unit 1. Hidden unit 2 is off for this tumor and gets no gradient from it at all: a ReLU unit learns only from the examples that activate it.
A single gradient step with learning rate 0.5 changes the prediction for this tumor from 0.3991 to 0.9871 and the loss from 0.9187 to 0.0129. A learning rate that large on a single example is only for illustration; in training, each step averages the gradient over a minibatch, and the many tumors pull the weights in different directions.
Checking the gradients
A backpropagation implementation with a mistake in it often still trains, slowly or badly, so the gradients should be checked against a definition that is hard to get wrong. The central finite difference perturbs one parameter $\theta_k$ by a small $h$ in each direction and measures the change in the loss,
$$\frac{\partial L}{\partial \theta_k} \approx \frac{L(\theta_k + h) - L(\theta_k - h)}{2h},$$
with an error proportional to $h^2$. It needs two forward passes per parameter, far too slow for training but fine for a check on a small network. For the hand-worked example, with $h = 10^{-6}$ in double precision, the nine backpropagated gradients agree with the finite differences to within $1.3 \times 10^{-10}$. For the network code used below, checked on 20 tumors and on 20 wines:
| Network | Parameters | Largest absolute difference | Relative difference |
|---|---|---|---|
| 30-8-4-1, tanh | 289 | $9.3 \times 10^{-11}$ | $1.8 \times 10^{-10}$ |
| 30-8-4-1, ReLU | 289 | $1.2 \times 10^{-10}$ | $1.6 \times 10^{-10}$ |
| 13-8-3 softmax, tanh | 139 | $2.8 \times 10^{-10}$ | $4.4 \times 10^{-10}$ |
The relative difference is the largest absolute difference divided by the largest gradient size. Agreement to about ten digits is what rounding allows with this $h$; an error in the code typically shows up as a disagreement in the first or second digit of some gradients. For ReLU networks, a check can fail spuriously if a weighted sum lies within $h$ of zero, where the function has a kink.
Starting values
The weights must start at different random values. If every hidden unit in a layer starts with the same weights, every unit computes the same output, receives the same error, and gets the same update, so the units stay identical forever and the layer acts as one unit. A 30-16-1 tanh network with every weight started at 0.05 was trained for 200 epochs on the breast cancer data: at the end, the largest difference between any two hidden units' weights was exactly zero, and a network with a single hidden unit, carrying the sum of the sixteen output weights, gave the same predictions to within $7 \times 10^{-16}$. Starting all weights at zero is worse still for ReLU units: every hidden weighted sum is zero, the ReLU's slope there is taken as zero, and the hidden layer's gradient is exactly zero, so it never moves. Hastie et al. (2009, §11.5.1) note both points: exact zero weights give "zero derivatives and perfect symmetry, and the algorithm never moves", while starting with large weights "often leads to poor solutions." They recommend random values near zero, so that sigmoid-like units start in their nearly linear range and the network starts close to a linear model and becomes nonlinear as the weights grow. How near zero matters more as networks deepen: the activation functions article measures the effect and derives the Glorot and He initializations, which the network below uses.
The inputs should be standardized for the same reason. The scale of the inputs sets the effective scale of the first layer's weights, and with inputs of mean zero and standard deviation one, a single range of starting weights suits every input (Hastie et al., 2009, §11.5.3). The breast cancer measurements range from values below 0.03 (the standard error of the fractal dimension) to areas above 4,000; unstandardized, a few large inputs would dominate every weighted sum.
A network from scratch
The network used below is about sixty lines of NumPy, in tools/experiments/data-science/nnet.py in the site's repository. Its core is the forward pass, the backward pass and the training loop:
import numpy as np
def forward(params, X, g):
A, Z = [X], []
for l, (W, b) in enumerate(zip(params["W"], params["b"])):
Z.append(A[-1] @ W + b)
if l < len(params["W"]) - 1:
A.append(g(Z[-1]))
P = 1 / (1 + np.exp(-Z[-1])) # sigmoid output (two classes)
return P, A, Z
def backward(params, A, Z, P, y, g_prime):
n = len(y)
D = (P - y.reshape(-1, 1)) / n # output errors, mean loss
grads_W, grads_b = [], []
for l in reversed(range(len(params["W"]))):
grads_W.insert(0, A[l].T @ D)
grads_b.insert(0, D.sum(axis=0))
if l > 0:
D = (D @ params["W"][l].T) * g_prime(Z[l - 1])
return grads_W, grads_b
def train(params, X, y, g, g_prime, lr=0.05, epochs=100, batch=32, seed=0):
rng = np.random.default_rng(seed)
for _ in range(epochs):
order = rng.permutation(len(y))
for start in range(0, len(y), batch):
idx = order[start:start + batch]
P, A, Z = forward(params, X[idx], g)
gW, gb = backward(params, A, Z, P, y[idx], g_prime)
for l in range(len(gW)):
params["W"][l] -= lr * gW[l]
params["b"][l] -= lr * gb[l]
return params
The version in the repository adds a softmax output for more than two classes, a numerically stable sigmoid, the initialization schemes, and early stopping: after each epoch it computes the loss on a separate validation set, keeps a copy of the weights from the epoch with the lowest validation loss, stops when that loss has not improved for a set number of epochs (the patience), and returns the copy.
Where hidden layers help: two moons
For each of 30 seeds, 200 training points, 200 validation points and 5,000 test points were drawn from make_moons with noise 0.25. Logistic regression (no penalty) was fitted to the training points, and networks with one hidden layer of 2, 8 or 64 ReLU units were trained by minibatch gradient descent (learning rate 0.1, He initialization), with early stopping on the validation points (patience 100 epochs, at most 3,000).
| Model | Test accuracy: median (IQR) |
|---|---|
| Logistic regression | 0.8614 (0.8584–0.8626) |
| Network, 2 hidden units | 0.8640 (0.8587–0.8731) |
| Network, 8 hidden units | 0.9358 (0.9328–0.9395) |
| Network, 64 hidden units | 0.9359 (0.9326–0.9375) |
Two hidden units are not enough: a ReLU network's boundary is made of straight pieces, roughly one bend for each unit, and the median result with two units was barely better than logistic regression's line. Eight units were enough to follow the crescents. Sixty-four did no better and no worse, although they have nearly eight times as many parameters (257 against 33), because early stopping halted them before they could fit the noise in 200 points; the next section shows what happens without it.
Where they do not: breast cancer and wine
The breast cancer data have 569 tumors and 30 measurements. In each of 30 splits (seeds 0 to 29), the tumors were divided at random, stratified by diagnosis, into 341 for training, 114 for validation and 114 for testing, and standardized using the training part. Three models were fitted to the training part: the NumPy network with one hidden layer of 16 ReLU units (513 parameters; learning rate 0.05, patience 50, at most 1,000 epochs, early stopping on the validation part); logistic regression with scikit-learn's default L2 penalty; and scikit-learn's MLPClassifier with one hidden layer of 16 units, up to 1,000 epochs, and its other settings at their defaults (ReLU, the Adam optimizer, a small L2 penalty, and stopping when the training loss stops improving). The network stopped at a median of epoch 73 (IQR 48 to 140).
| Model | Test accuracy: median (IQR) | Test log loss: median (IQR) |
|---|---|---|
| NumPy network, 30-16-1 | 0.9649 (0.9561–0.9803) | 0.0858 (0.0693–0.1166) |
| Logistic regression | 0.9737 (0.9649–0.9825) | 0.0764 (0.0680–0.1050) |
scikit-learn MLPClassifier, 16 units |
0.9737 (0.9649–0.9803) | 0.0763 (0.0606–0.1206) |
Compared split by split with the NumPy network, logistic regression was more accurate in 13 splits, less accurate in 8 and tied in 9, and had lower log loss in 20 of the 30, by a median of 0.0084. scikit-learn's network was more accurate in 15, less in 7, tied in 8, with lower log loss in 17. On 114 test tumors, one tumor is 0.9 percentage points of accuracy, and the differences between the three models are smaller than the differences between splits.
The wine data (Aeberhard and Forina, 1992) have 178 wines, 13 chemical measurements and three cultivars, so the network ends in a softmax over three outputs (13-16-3, 275 parameters). With 106 wines for training, 36 for validation and 36 for testing in each split, all three models had a median test accuracy of 0.9722 (IQR 0.9722 to 1.0000), one wrong wine in 36; logistic regression and the NumPy network tied on accuracy in 19 of the 30 splits. The median test log losses were 0.0747 for the NumPy network, 0.0686 for logistic regression and 0.0654 for scikit-learn's network.
These outcomes are what one would expect for these two datasets. Both are close to linearly separable once standardized, as the perceptron article found for the breast cancer training sets, so logistic regression's line is already close to the best boundary, and a network's extra flexibility buys only extra variance. With a few hundred examples, the variance of a network's fitted probabilities shows in the log loss before it shows in the accuracy. The two-moons problem is the opposite case: abundant data from a boundary that is far from linear.
Overfitting and early stopping
A network with enough units can fit its training data almost perfectly, noise included, which is the high-variance end of the bias-variance tradeoff. To see it, a much larger network, 30-256-256-1 with 73,985 parameters, more than 200 for each of the 341 training tumors, was trained on the same 30 breast cancer splits for 1,000 epochs without stopping, recording the loss on the training and validation parts after every epoch.
On the first split, the validation loss reached its lowest value, 0.0423, at epoch 38, when the training loss was 0.0165; by epoch 1,000 the training loss was 0.0002 and the validation loss had risen to 0.0515. Over the 30 splits:
| Quantity | At the epoch of lowest validation loss | At epoch 1,000 |
|---|---|---|
| Epoch | median 23.5 (IQR 12.3–42.5) | 1,000 |
| Training loss | 0.0002 (0.0002–0.0003) | |
| Validation loss | 0.0709 (0.0414–0.1091) | 0.1446 (0.0587–0.2531) |
| Test log loss | 0.0840 (0.0602–0.1153) | 0.1430 (0.0808–0.1933) |
| Test accuracy | 0.9737 (0.9649–0.9803) | 0.9649 (0.9561–0.9737) |
The test log loss was higher at the end than at the best validation epoch in 25 of the 30 splits, by a median of 0.0509. Test accuracy changed much less: it was higher at the end in 7 splits, lower in 8 and the same in 15. Overfitting here shows mostly in the probabilities. The fully trained network is as often right as the stopped one, but it is confidently right and sometimes confidently wrong, and the log loss charges heavily for confident mistakes. Judging the network by accuracy alone would have hidden most of the damage.
Early stopping trains while watching a validation loss and keeps the weights from the epoch where it was lowest. Hastie et al. (2009, §11.5.2) describe it as one of the original ways of avoiding overfitting in neural networks, and note that since training starts from nearly linear small weights, stopping early "has the effect of shrinking the final model toward a linear model." The stopping epoch is chosen on the validation data, which makes the validation loss at that epoch an optimistic estimate, the winner's curse described in Training, Validation, and Test Sets; that is why the table judges the result on test data that played no part in the choice. The other common remedies limit the weights directly. Weight decay adds a penalty $\lambda \sum w^2$ to the loss, the network analogue of ridge regression (Hastie et al., 2009, §11.5.2), and is the alpha parameter of scikit-learn's networks. Dropout sets a random fraction of the units to zero at each training step so that no unit can rely on particular others (James et al., 2021, §10.7.3); Scaling the Shakespeare GPT applies it in a transformer. Reducing the number of units or layers lowers the capacity to overfit, as for any model. A smaller learning rate slows training down, which with a fixed number of epochs acts like stopping earlier, but on its own it delays overfitting rather than preventing it.
Universal approximation
How flexible can a network with one hidden layer be? Cybenko (1989) proved that it can approximate any continuous function on the unit cube as closely as desired. In his Theorem 2, for any continuous sigmoidal activation $\sigma$ (one that tends to 1 as its input goes to $+\infty$ and to 0 as it goes to $-\infty$), sums of the form
$$G(\mathbf{x}) = \sum_{j=1}^{N} \alpha_j\, \sigma(\mathbf{y}_j^{\mathsf{T}}\mathbf{x} + \theta_j)$$
are dense in the continuous functions on $[0, 1]^p$: for any continuous $f$ and any $\varepsilon > 0$ there is such a sum with $|G(\mathbf{x}) - f(\mathbf{x})| < \varepsilon$ for every $\mathbf{x}$ in the cube. That sum is exactly a network with one hidden layer of $N$ sigmoid units and a linear output. Hornik, Stinchcombe and White (1989) obtained related results at about the same time. Hornik (1991) extended the result to any bounded, nonconstant activation, and Leshno, Lin, Pinkus and Schocken (1993) gave the cleanest condition: a network with a locally bounded, piecewise continuous activation can approximate any continuous function "if and only if the network's activation function is not a polynomial." ReLU qualifies. The same paper stresses that the bias terms are needed; without them the result fails.
For ReLU units in one dimension, an approximation can be written down directly. The function
$$f_N(x) = f(0) + \sum_{j=0}^{N-1} c_j \max(0,\; x - j/N)$$
is piecewise linear with a bend at each knot $j/N$, and choosing $c_0$ as the slope of the first piece and each later $c_j$ as the change of slope at knot $j/N$ makes it pass through $f$ at all $N + 1$ knots. It is a network with one hidden layer of $N$ ReLU units with input weight 1 and biases $-j/N$. For $f(x) = \sin(2\pi x)$ on $[0, 1]$:
| Hidden units $N$ | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|
| Largest error | 1.0000 | 0.2105 | 0.0704 | 0.0189 | 0.0048 | 0.0012 |
(With $N = 2$ every knot falls on a zero of the sine, so the network is identically zero, and the error is the sine's full height.) The error falls by about four each time $N$ doubles, as the standard bound for linear interpolation, $\max|f''|/(8N^2) = (2\pi)^2/(8N^2)$, predicts; at $N = 64$ the two agree to three decimal places.
The theorem is often quoted as if it explained why networks work, and it promises much less than that. It is a statement about existence. It does not say how many hidden units are needed; Cybenko himself wrote that "we suspect quite strongly that the overwhelming majority of approximation problems will require astronomical numbers of terms," because of the curse of dimensionality. The construction above shows why: a grid of knots fine enough to follow a function in one dimension needs $N$ units, and the analogous grid in $p$ dimensions needs on the order of $N^p$. It does not say that gradient descent will find the approximating weights from a random start. And it says nothing about learning from data: a network flexible enough to represent any function is flexible enough to represent the noise in a finite sample, which is the overfitting of the previous section. In practice, networks with several layers of moderate width are easier to train than one very wide layer; James et al. (2021, §10.2) put it that one large hidden layer can approximate most functions in theory, but "the learning task of discovering a good solution is made much easier with multiple layers each of modest size."
Beyond this article
The multilayer perceptron is the base on which more specialized networks are built. Convolutional networks share weights across the positions of an image; recurrent networks carry a state along a sequence; transformers, used in language models, combine attention layers with small MLPs applied at every position. The Language Models from Scratch collection builds one, a character-level GPT, step by step in PyTorch, where automatic differentiation computes the backward pass that is written out by hand here. Its Assembling the Transformer Block adds residual connections and layer normalization, two further techniques that keep gradients usable through many layers. The course wiki's Neuroevolution topic trains networks without gradients at all, by evolving their weights and structure.
In Python
scikit-learn's MLPClassifier and MLPRegressor implement the multilayer perceptron with backpropagation:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, stratify=data.target, random_state=0)
model = make_pipeline(
StandardScaler(),
MLPClassifier(hidden_layer_sizes=(16,), activation="relu", alpha=1e-4,
max_iter=1000, random_state=0))
model.fit(X_train, y_train)
print(model.score(X_test, y_test)) # 0.9649
mlp = model[-1]
print(mlp.n_iter_) # 391 epochs
print(mlp.loss_curve_[-1]) # training loss after the last epoch
hidden_layer_sizes lists the sizes of the hidden layers only, so (16,) is one hidden layer of 16 units and (16, 8) two layers; the input and output sizes come from the data. alpha is the weight decay penalty. The standardization step sits inside the pipeline so that it is fitted on the training data only. By default the network trains until the training loss stops improving, here after 391 epochs, which does nothing to prevent overfitting.
scikit-learn's own early stopping needs care. With early_stopping=True, it sets aside 10% of the training data and stops when the score on it has not improved for n_iter_no_change epochs (10 by default). That score is the validation accuracy, recorded in validation_scores_, not the validation loss, so it watches decisions rather than probabilities, and the overfitting seen above in the log loss can pass unnoticed. On a set of 46 tumors, accuracy also moves in coarse steps and can sit still for ten epochs while the network is still learning, especially with the small default learning rate of the Adam optimizer (0.001). On this split, turning early stopping on with the other settings unchanged stopped training after 37 epochs at a test accuracy of 0.8772, against 0.9649 without it; with n_iter_no_change=50 it ran 140 epochs and reached 0.9298. The early stopping in the NumPy network above watches the validation loss, with a patience of 50 epochs, for this reason.
The numbers in this article, including the hand-worked pass, the gradient checks, the three comparisons, the overfitting experiment and the ReLU approximations, are computed by tools/experiments/data-science/multilayer-perceptron.py in the site's repository, using tools/experiments/data-science/nnet.py.
References
- Aeberhard, S., and Forina, M. (1992). Wine [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5PC7J record · 8 pages
- Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4), 303–314. record · 2 pages
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. §§11.3–11.5. record · 23 pages
- Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2), 251–257. Cited through Leshno et al. (1993); not read for this article. record
- Hornik, K., Stinchcombe, M., and White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359–366. Cited through Cybenko (1989) and Leshno et al. (1993); not read for this article. record
- James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. Ch. 10. record · 30 pages
- Leshno, M., Lin, V. Ya., Pinkus, A., and Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6), 861–867. Read in the working-paper version, Stern School of Business IS-92-13, New York University (1992); the quoted abstract checked against the journal version. record
- Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. record · 2 pages
- scikit-learn developers (2024). MLPClassifier. scikit-learn 1.6 documentation. https://scikit-learn.org/1.6/modules/generated/sklearn.neural_network.MLPClassifier.html record
- Werbos, P. J. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University. Cited by attribution (Hastie et al., 2009). record · 2 pages
- Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B record · 14 pages