Introduction to Artificial Neural Networks

What an artificial neuron computes and how neurons are joined into networks. One neuron worked on two breast cancer measurements and shown to be logistic regression, the biological analogy and where it stops, the history from McCulloch and Pitts through Rosenblatt's perceptron to backpropagation and the deep learning revival, layers, weights and parameter counts, why hidden layers need a nonlinear activation, and a three-unit threshold network that computes XOR.

A pathologist has measured the cell nuclei in a fine-needle aspirate of a breast mass. Two of the measurements are the mean radius of the nuclei and their mean texture, the standard deviation of the gray levels inside them. For the first tumor in the Wisconsin diagnostic data (Wolberg, Mangasarian, Street and Street, 1993), these are 17.99 and 10.38. Expressed in standard deviations from the mean of all 569 tumors, they are $x_1 = 1.0971$ and $x_2 = -2.0733$: a large radius and an unusually smooth texture.

A simple rule for judging the tumor multiplies each measurement by a weight, adds the products and a constant, and passes the total through a function that turns it into a number between 0 and 1:

$$z = 3.7220 \times 1.0971 + 0.9374 \times (-2.0733) - 0.7076 = 1.4321, \qquad p = \frac{1}{1 + e^{-z}} = 0.8072.$$

The weights favor malignancy for a large radius and, more weakly, for a coarse texture. This tumor's large radius outweighs its smooth texture, and the rule assigns it a probability of malignancy of 0.81. It was malignant. The first benign tumor in the file (tumor 19, radius 13.54 and texture 14.36) gets $z = -2.4038$ and $p = 0.0829$.

That computation, a weighted sum of inputs followed by a fixed nonlinear function, is an artificial neuron, the unit from which every neural network is built. It is also exactly logistic regression: the weights above are the maximum likelihood coefficients of a logistic regression of diagnosis on the two standardized measurements. A neural network is what results when many such units are connected, the outputs of some becoming the inputs of others, and all their weights are fitted together. This article describes the neuron, where the idea came from, and how networks are put together. The rest of the topic covers the single-layer perceptron and its learning rule, the nonlinear activation functions that make layers useful, and the multilayer perceptron and how it is trained.

What a neuron computes

An artificial neuron receives $n$ inputs $x_1, \ldots, x_n$. It holds one weight $w_j$ for each input and a bias $b$, and computes

$$z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w}^{\mathsf{T}} \mathbf{x} + b, \qquad a = g(z).$$

The weighted sum $z$ is the neuron's pre-activation, $g$ is its activation function, chosen in advance and not learned, and $a$ is its output, or activation. A positive weight means a larger input pushes the output up, a negative weight that it pushes the output down, and a weight near zero that the input hardly matters. The bias sets how large the weighted sum must be before the output turns on; it plays the role of the intercept in a regression, and machine learning texts call intercepts biases (James, Witten, Hastie and Tibshirani, 2021, §10.2, who note that this "bias" has nothing to do with the bias of the bias-variance tradeoff). A common device treats the bias as the weight on an extra input that always equals 1, so that $z = \tilde{\mathbf{w}}^{\mathsf{T}} \tilde{\mathbf{x}}$ with $\tilde{\mathbf{x}} = (x_1, \ldots, x_n, 1)$.

One artificial neuron, with the weights fitted to the breast cancer data. The two standardized measurements and a constant 1 are multiplied by their weights and summed; the sum, 1.43 for tumor 0, passes through the activation function g, here the sigmoid, to give the output 0.81.
One artificial neuron, with the weights fitted to the breast cancer data. The two standardized measurements and a constant 1 are multiplied by their weights and summed; the sum, 1.43 for tumor 0, passes through the activation function g, here the sigmoid, to give the output 0.81.

Two activation functions recur throughout this topic. The step function outputs 1 when $z \geq 0$ and 0 otherwise, so the neuron makes a yes-or-no decision. The sigmoid (or logistic function) $\sigma(z) = 1/(1 + e^{-z})$ rises smoothly from 0 to 1, passing through 0.5 at $z = 0$, so the neuron reports a graded value that can be read as a probability. With the weights above, the two choices make the same decisions, since $\sigma(z) \geq 0.5$ exactly when $z \geq 0$: both call 507 of the 569 tumors correctly (89.1%). They differ in what else they say. The step neuron cannot distinguish a tumor that barely crosses the line from one far beyond it, and, more important for training, its output does not change at all when a weight changes by a small amount, so it gives no signal about which way to adjust the weights. The article on activation functions covers these and the others in use.

Whatever the activation, the boundary between the two decisions is where $z = 0$, the set of inputs with $\mathbf{w}^{\mathsf{T}}\mathbf{x} + b = 0$. With two inputs it is a straight line, with three a plane and in general a hyperplane. A single neuron is therefore a linear classifier, like logistic regression and the support vector classifier: it can only separate classes that a hyperplane can separate.

The 569 tumors by mean radius and mean texture, with the neuron's contours of constant output. The solid line is z = 0 (output 0.5), the neuron's decision boundary; the dashed lines are outputs of 0.1 and 0.9. Every contour is a straight line, so the neuron can only divide the plane in two along a line. In the original units the boundary is radius = 18.78 − 0.206 × texture.
The 569 tumors by mean radius and mean texture, with the neuron's contours of constant output. The solid line is z = 0 (output 0.5), the neuron's decision boundary; the dashed lines are outputs of 0.1 and 0.9. Every contour is a straight line, so the neuron can only divide the plane in two along a line. In the original units the boundary is radius = 18.78 − 0.206 × texture.

The biological analogy

The name comes from a loose analogy with nerve cells. A biological neuron receives signals from other neurons at junctions called synapses, mostly on its branched dendrites and its cell body; the signals combine, and if the combined excitation at a given moment exceeds a threshold the neuron fires an impulse along its axon to the synapses of the next neurons. Some synapses excite and others inhibit. In the artificial neuron, the inputs stand for incoming signals, the weights for the strength and sign of each synapse, the weighted sum for the combined excitation, and the step function for the threshold. Hastie, Tibshirani and Friedman (2009, §11.3) give the same mapping: units were neurons, connections were synapses, and in early models a neuron fired when the total signal passed to it exceeded a threshold, which corresponds to a step activation.

The analogy explains the vocabulary, not how the methods work. Real neurons signal with timed electrical pulses, and their synapses change through chemical processes over many time scales; an artificial neuron is a fixed formula with real-valued inputs and outputs. The networks used in data science are better understood as what Hastie et al. (2009, §11.3) call them, nonlinear statistical models: functions with many adjustable coefficients, fitted to data by minimizing a loss. The smooth sigmoid replaced the step function precisely because the step function "is not smooth enough for optimization" (Hastie et al., 2009, §11.3), a statistical reason, not a biological one.

A short history

McCulloch and Pitts (1943). The first artificial neurons were logical devices. Warren McCulloch, a neurophysiologist, and Walter Pitts, a logician, argued that "because of the 'all-or-none' character of nervous activity, neural events and the relations among them can be treated by means of propositional logic" (McCulloch and Pitts, 1943, abstract). Their idealized neuron fired if a fixed number of its excitatory synapses were active at once, and any active inhibitory synapse prevented it from firing at all; the structure of the net did not change with time (assumptions 1 to 5 of their Section 2). They showed that nets of such units could compute logical functions and, with loops, behave like the machines of computation theory. Their units had thresholds but no learning: the connections were designed, not fitted.

Rosenblatt (1958). Frank Rosenblatt's perceptron added learning. His paper describes a system of sensory points (a retina), association units, each receiving connections from a number of sensory points, and response units fed by the association units; an association unit fires when the algebraic sum of its excitatory and inhibitory inputs reaches its threshold, and learning changes the influence each association unit has on the responses, so that a stimulus comes to evoke the response it has been reinforced for (Rosenblatt, 1958). In the simplified form used today, a perceptron is a single neuron with a step activation, and the perceptron learning rule adjusts its weights after each mistake. When the two classes can be separated by a hyperplane the rule is guaranteed to find one in a finite number of steps; the next article proves this.

Minsky and Papert (1969). A single neuron can only draw one hyperplane, and some simple functions need more. The standard example is exclusive or (XOR) of two binary inputs, which is 1 when exactly one input is 1: no line separates its two 1s from its two 0s. Marvin Minsky and Seymour Papert's book Perceptrons (1969) analyzed what single-layer perceptrons can and cannot compute and is widely credited with turning research attention and funding away from them. A network with a hidden layer can compute XOR easily, as shown below; what was missing was a way to train the hidden layer.

Backpropagation (1970s and 1986). That method is to compute the gradient of the error with respect to every weight by the chain rule, working backward from the output, and to adjust all weights by gradient descent. It was described by Werbos (1974) and Parker (1985) and brought to wide attention by Rumelhart, Hinton and Williams (1986) (Hastie et al., 2009, bibliographic notes to Ch. 11). Rumelhart, Hinton and Williams showed that the hidden units of networks trained this way "come to represent important features of the task domain", and they contrasted this with "earlier, simpler methods such as the perceptron-convergence procedure" (1986, abstract). The multilayer perceptron article derives the method.

Decline and revival. James et al. (2021, Ch. 10) describe what followed. Neural networks were popular in the late 1980s and were then analyzed by statisticians and mathematicians; in the 1990s and 2000s support vector machines, boosting and random forests often matched or beat them with less tinkering. A core group of researchers kept going, and after 2010 neural networks returned under the name deep learning, with new architectures and successes in image classification, speech and language, which many attribute chiefly to much larger training sets. One change in the networks themselves was the switch from sigmoid to rectified linear activations, $g(z) = \max(0, z)$, which Nair and Hinton (2010) used in place of binary units and which deep networks train far more easily with; the activation functions article measures why. The Language Models from Scratch collection builds a modern deep network, a small GPT, step by step.

From neurons to networks

A feed-forward neural network arranges neurons in layers. The input layer holds the feature values; it computes nothing. Each hidden layer is a set of neurons that all receive the outputs of the layer before; each neuron has its own weights and bias, and all apply the same activation function. The output layer produces the prediction. Connections run only forward, from each layer to the next, with no loops; networks with loops, which carry a state from one item of a sequence to the next, are recurrent networks and are not covered here. When every unit of a layer receives every output of the previous layer, the layer is fully connected (or dense).

A fully connected network with four inputs, one hidden layer of five units and three outputs. Each line is a weight. The hidden units apply an activation function g to weighted sums of the inputs; the output units combine the hidden activations, and a softmax turns the three results into class probabilities.
A fully connected network with four inputs, one hidden layer of five units and three outputs. Each line is a weight. The hidden units apply an activation function g to weighted sums of the inputs; the output units combine the hidden activations, and a softmax turns the three results into class probabilities.

With one hidden layer of $K$ units and an input vector $\mathbf{x}$ of length $p$, hidden unit $k$ computes $a_k = g(\mathbf{w}_k^{\mathsf{T}} \mathbf{x} + b_k)$, and output $m$ computes $z_m = \boldsymbol{\beta}_m^{\mathsf{T}} \mathbf{a} + \beta_{m0}$. James et al. (2021, §10.1) describe the hidden layer as deriving $K$ new features, each a nonlinear transformation of a different linear combination of the inputs, and the output layer as a linear model in those derived features. Hastie et al. (2009, §11.3) make the same point: the network is a linear model, or a multinomial logistic model, applied to a basis expansion of the inputs, with the difference from fixed basis expansions that "the parameters of the basis functions are learned from the data."

The output layer depends on the task. For a numerical outcome it is a single unit with no activation, $f(\mathbf{x}) = z$. For a yes-or-no outcome it is one unit with a sigmoid, giving a probability. For $K$ classes it is $K$ units followed by a softmax, $p_m = e^{z_m} / \sum_{l} e^{z_l}$, which turns $K$ numbers into $K$ positive probabilities that sum to one (James et al., 2021, eq. 10.13). With a softmax output and the cross-entropy loss, the network is, in Hastie et al.'s words, "exactly a linear logistic regression model in the hidden units" (2009, §11.4).

Stacking layers is a matter of feeding each layer's activations to the next. For a whole dataset at once, with the $n$ examples as the rows of a matrix $X$, one hidden layer computes

$$A^{(1)} = g\big(X W^{(1)} + \mathbf{1}\mathbf{b}^{(1)\mathsf{T}}\big),$$

where $W^{(1)}$ is a $p \times K$ matrix whose column $k$ holds the weights of hidden unit $k$, and $g$ is applied to every entry. The next layer applies the same pattern to $A^{(1)}$. Every unit in a deeper layer is still a function of the original inputs, through the chain of layers before it (James et al., 2021, §10.2).

Counting parameters

A dense layer from $n_{\text{in}}$ units to $n_{\text{out}}$ units has $n_{\text{in}} \times n_{\text{out}}$ weights and $n_{\text{out}}$ biases, $(n_{\text{in}} + 1)\, n_{\text{out}}$ parameters in all. The network used later in this topic for the wine data, with 13 inputs, 16 hidden units and 3 outputs, has $14 \times 16 = 224$ parameters in its first layer and $17 \times 3 = 51$ in its second, 275 in all. The network James et al. (2021, §10.2) fit to the MNIST handwritten digits, with 784 pixel inputs, hidden layers of 256 and 128 units and 10 outputs, has $785 \times 256 + 257 \times 128 + 129 \times 10 = 235{,}146$ parameters, more than 33 times the $785 \times 9 = 7{,}065$ of a multinomial logistic regression on the same pixels (nine sets of coefficients, with one class as the reference). Parameter counts grow with the product of adjacent layer widths, which is one reason networks need a great deal of data or explicit regularization.

Descriptions of architecture count differently from source to source. Some count layers of units (input, hidden, output), others layers of weights; some describe a network by its hidden layer sizes alone, as scikit-learn's hidden_layer_sizes=(16,) does. Input "units" only hold values, so they are not neurons in the computing sense; hidden and output units both are. Stating the sizes of every layer, as in 13-16-3, avoids the ambiguity.

Why the hidden layers need a nonlinearity

If the activation function were the identity, $g(z) = z$, a hidden layer would add nothing. Two linear layers compute

$$(X W^{(1)} + \mathbf{1}\mathbf{b}^{(1)\mathsf{T}})\, W^{(2)} + \mathbf{1}\mathbf{b}^{(2)\mathsf{T}} = X\,(W^{(1)} W^{(2)}) + \mathbf{1}\,(W^{(2)\mathsf{T}}\mathbf{b}^{(1)} + \mathbf{b}^{(2)})^{\mathsf{T}},$$

a single linear layer with weight matrix $W^{(1)}W^{(2)}$. Any number of linear layers collapses the same way. For a random 3-4-2 network the two computations agree to within $9 \times 10^{-16}$, rounding error. A network without nonlinear activations is therefore a linear model however deep it is; James et al. (2021, §10.1) call the nonlinearity "essential" for this reason, and show that with it, two hidden units can produce an interaction: with $g(z) = z^2$, $\tfrac14 (x_1 + x_2)^2 - \tfrac14 (x_1 - x_2)^2 = x_1 x_2$.

XOR with three threshold units

A network with one hidden layer of two step-function units computes XOR. Let

$$h_1 = \operatorname{step}(x_1 - x_2 - 0.5), \qquad h_2 = \operatorname{step}(x_2 - x_1 - 0.5), \qquad \text{output} = \operatorname{step}(h_1 + h_2 - 0.5).$$

Unit $h_1$ is on only when $x_1 = 1$ and $x_2 = 0$ ("$x_1$ and not $x_2$"), unit $h_2$ only when $x_2 = 1$ and $x_1 = 0$, and the output unit is on when either hidden unit is (an OR):

$x_1$ $x_2$ $h_1$ $h_2$ output XOR
0 0 0 0 0 0
0 1 0 1 1 1
1 0 1 0 1 1
1 1 0 0 0 0
Left: a network of three threshold units that computes XOR, with positive weights as solid blue arrows and negative weights as dashed orange ones. Right: each hidden unit draws one line across the input square and is on in the shaded band beyond it; the output unit fires when the input lies in either band, which no single line can arrange.
Left: a network of three threshold units that computes XOR, with positive weights as solid blue arrows and negative weights as dashed orange ones. Right: each hidden unit draws one line across the input square and is on in the shaded band beyond it; the output unit fires when the input lies in either band, which no single line can arrange.

Each hidden unit draws one line, and the output combines the two half-planes. This is the general picture of what hidden layers do: each unit contributes a simple boundary or feature, and later layers combine them into shapes no single unit could make. The construction here was done by hand, in the spirit of McCulloch and Pitts. Finding such weights from data is the subject of the rest of the topic; the course wiki's Evolving Neural Network Weights finds XOR weights for a sigmoid network by a genetic algorithm instead of by gradients, and NEAT evolves the hidden structure as well.

Training in outline

A network's weights are chosen to make a loss small on the training data: the mean squared error for numerical outcomes, or the cross-entropy (log loss) for classes, the same criterion that logistic regression maximizes as a likelihood and that Loss Functions for Classification examines. There is no closed-form solution, and unlike logistic regression the loss of a network with hidden layers is not convex: it can have many local minima, and which one training reaches depends on the starting weights (Hastie et al., 2009, §11.5). Networks are fitted by gradient descent: start from small random weights, compute the gradient of the loss with respect to every weight, move every weight a small step against its gradient, and repeat. Backpropagation is the bookkeeping that computes all those gradients in one backward pass through the network. Because a network can have far more parameters than there are training examples, it is usually stopped early or penalized to keep it from fitting noise, which links neural networks to the bias-variance tradeoff and to the use of validation data. The multilayer perceptron article works all of this through and trains a network written from scratch.

Where neural networks fit

Among the methods in this collection, neural networks sit at the flexible end. A network with one hidden layer and enough units can approximate any continuous function on the unit cube (and so, after rescaling the inputs, on any bounded box) as closely as desired (Cybenko, 1989), which no fixed linear model can do. The flexibility has costs. The fitted weights do not have the direct reading of regression coefficients or the rules of a decision tree; training involves choices (architecture, learning rate, initialization, stopping) that change the result; and the networks need more data than simpler methods to estimate their many parameters well. On tabular data of modest size, such as the breast cancer measurements, a well-tuned logistic regression is often as accurate, as the multilayer perceptron article finds. Networks pay off when the relationship between inputs and outcome is strongly nonlinear and there are many examples, and above all for images, sound and text, where specialized architectures exploit the structure of the input.

In Python

The neuron of the opening example is a logistic regression, and scikit-learn fits it directly:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
cols = [list(data.feature_names).index(c) for c in ("mean radius", "mean texture")]
X = StandardScaler().fit_transform(data.data[:, cols])
y = (data.target == 0).astype(int)              # 1 = malignant

neuron = LogisticRegression(penalty=None, tol=1e-10, max_iter=10000).fit(X, y)
w, b = neuron.coef_[0], neuron.intercept_[0]    # [3.7220, 0.9374], -0.7076

z = X @ w + b                                   # pre-activations
p = 1 / (1 + np.exp(-z))                        # sigmoid activations
print(z[0], p[0])                               # 1.4321, 0.8072

In scikit-learn's data the label 0 means malignant, so the code recodes it to 1. The tight tolerance makes the solver run all the way to the maximum likelihood estimate; with its default tolerance it stops slightly short, at weights of 3.7183 and 0.9367. Networks with hidden layers are fitted with sklearn.neural_network.MLPClassifier or MLPRegressor, or written out in NumPy as in the multilayer perceptron article. The numbers in this article are computed by tools/experiments/data-science/introduction-to-neural-networks.py in the site's repository.

References

  • Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4), 303–314. record · 2 pages
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. Ch. 11. record · 23 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. Ch. 10. record · 30 pages
  • McCulloch, W. S., and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–133. Read in the reprint, Bulletin of Mathematical Biology, 52(1/2), 99–115 (1990). record
  • Minsky, M., and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press. Cited by attribution; not read for this article. record · 2 pages
  • Nair, V., and Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (pp. 807–814). record · 2 pages
  • Parker, D. B. (1985). Learning-Logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science, MIT. Cited by attribution (Hastie et al., 2009). record
  • Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408. record · 2 pages
  • Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. record · 2 pages
  • Werbos, P. J. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University. Cited by attribution (Hastie et al., 2009). record · 2 pages
  • Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B record · 14 pages