Generative AI

Models that produce new data rather than labels. Generative and discriminative models compared on handwritten digits, where a Gaussian model in 20 principal components classifies better than logistic regression in all 30 splits and also draws new digits that are not copies; why images need deep generative models; variational autoencoders, generative adversarial networks and diffusion models from their original papers; autoregressive language models as next-token predictors; scaling laws with Kaplan et al.'s power law worked through and the Chinchilla revision; turning a pretrained model into an assistant; and how generated output is evaluated.

None of the 80 digits below was written by a person.

Eighty handwritten digits drawn from a simple generative model, eight for each class 0 to 9. Each class is modeled as a normal distribution in the first 20 principal components of 1,797 real 8 × 8 digits; a sample is a random point from that distribution, mapped back to pixels. The digits are recognizable, varied and blurry, and no two are alike.
Eighty handwritten digits drawn from a simple generative model, eight for each class 0 to 9. Each class is modeled as a normal distribution in the first 20 principal components of 1,797 real 8 × 8 digits; a sample is a random point from that distribution, mapped back to pixels. The digits are recognizable, varied and blurry, and no two are alike.

Each was produced by drawing a random point from a probability distribution fitted to the 1,797 scanned digits that ship with scikit-learn (Alpaydin and Kaynak, 1998), and turning the point back into 64 pixels. They are recognizable as digits, they vary the way handwriting varies, and none of them is a copy of a training image. They are also blurry and speckled, which is the subject of most of this article: the step from a model that can produce digits like these to models that produce photographs, fluent essays and working code.

A generative model is a model of the data itself: a probability distribution from which new examples can be drawn. Generative AI is the name now given to systems built on such models that produce text, images, audio, video or code. This article starts with the statistical idea on the small example above, then describes the three families of deep generative models for images, the autoregressive models that generate text, what is known about how they improve with scale, and how they are evaluated. The prompt engineering article covers how such models are used, and the risks article what can go wrong.

Generative and discriminative models

A classifier needs $\Pr(Y = k \mid X = x)$, the probability of each class given the input. James, Witten, Hastie and Tibshirani (2021, §4.4) describe two ways to get it. A discriminative model such as logistic regression models that conditional probability directly. A generative model takes "an alternative and less direct approach": it models the distribution of the inputs $X$ separately within each class, $f_k(x)$, together with the class proportions $\pi_k$, and then uses Bayes' theorem,

$$\Pr(Y = k \mid X = x) = \frac{\pi_k f_k(x)}{\sum_{l} \pi_l f_l(x)}.$$

Linear and quadratic discriminant analysis and naive Bayes are generative classifiers in this sense, and the supervised learning overview works Gaussian naive Bayes by hand. The difference matters beyond classification. A discriminative model can only label inputs it is given. A generative model, having a distribution for $X$, can also score how probable an input is, and can draw new inputs.

The digits make the comparison concrete. The generative model projects the 64 pixels onto their first 20 principal components, which keep 89.4% of the pixels' variance (Principal Component Analysis), and models each digit class as a multivariate normal distribution in those 20 coordinates, with its own mean and covariance matrix. Trained on random halves of the data and tested on the other halves, 30 times:

Model Test accuracy, median (IQR)
Logistic regression on the 64 pixels 0.961 (0.008)
Gaussian generative model in 20 principal components 0.981 (0.006)

The generative classifier was more accurate in all 30 splits, by a median of 1.9 percentage points. That is a property of these data, whose classes happen to be well described by a normal distribution in a few principal components, not a general rule; James et al. (2021, §4.4) note that generative classifiers of this kind may be more accurate than logistic regression when the predictors are approximately normal in each class and the sample is small.

The generative model also does two things logistic regression cannot. It assigns a probability density to any image: the 1,797 real digits have an average log-density of −54.9 under the fitted mixture, while 1,797 images of random noise have −105.2, so the model can tell a plausible digit from static. And it can sample, which is how the figure above was made. A natural worry about a model that generates data is that it copies its training examples. For each of the 80 samples, the closest real digit is at a median Euclidean distance of 17.4 gray levels (interquartile range 15.6 to 18.9, minimum 10.3), while real digits lie at a median of 16.1 from their nearest other real digit. The samples are about as far from the training set as the training digits are from each other.

One correction to a common description is worth making here. Generative AI is often presented as a subset of deep learning. The model above has no neural network at all, and generative models are much older than deep learning: a bigram language model that samples the next character from a table of counts is a generative model of text. What deep learning added is the ability to model very complicated distributions, which is what photographs and long passages of text require.

Why images need deep generative models

The Gaussian model's samples are blurry because a single normal distribution in 20 dimensions cannot represent sharp strokes, and pixel space is far larger for real images: a $256 \times 256$ color image has 196,608 numbers. The distribution of natural images occupies a thin, curved region of that space, and a model must learn where it lies. The three families below all use neural networks to learn a transformation between a simple distribution that is easy to sample, usually standard normal noise, and the distribution of the data.

Variational autoencoders. Kingma and Welling (2014) consider models with continuous latent variables: each image $x$ is generated by first drawing a hidden code $z$ from a simple prior, then drawing $x$ from a distribution $p_\theta(x \mid z)$ computed by a neural network, the decoder. Fitting such a model by maximum likelihood requires the posterior $p_\theta(z \mid x)$, which is intractable. Their method fits a second network, the encoder or recognition model $q_\phi(z \mid x)$, to approximate it, and maximizes a lower bound on the log-likelihood of each data point (§2.2),

$$\log p_\theta(x) \geq \mathbb{E}_{q_\phi(z \mid x)}\big[\log p_\theta(x \mid z)\big] - D_{\mathrm{KL}}\big(q_\phi(z \mid x) \,\|\, p(z)\big).$$

The first term rewards reconstructing $x$ from its code; the second keeps the codes close to the prior, so that codes drawn from the prior decode to plausible images. The key step that makes the bound trainable by gradient descent is the reparameterization: instead of sampling $z$ from $q_\phi$ directly, sample standard normal noise $\varepsilon$ and set $z = \mu_\phi(x) + \sigma_\phi(x) \varepsilon$, so that gradients flow through $\mu$ and $\sigma$ (Kingma and Welling, 2014, §2.3 and §2.4). The randomness enters as a draw from the encoder's distribution for each input, which is the precise version of the common summary that noise is "added in the latent space". To generate, draw $z$ from the prior and decode it.

Generative adversarial networks. Goodfellow et al. (2014) took a different route that avoids writing down a likelihood at all. A generator network $G$ maps noise $z$ to a sample $G(z)$; a discriminator network $D$ outputs the probability that its input came from the training data rather than from $G$. They are trained against each other: $D$ to tell real from generated, $G$ "to maximize the probability of $D$ making a mistake" (abstract). Their analogy is a team of counterfeiters trying to pass fake currency and police trying to detect it, with competition driving both to improve "until the counterfeits are indistiguishable from the genuine articles" (§1; the misspelling is in the original). Formally the two play a minimax game whose value is

$$\min_G \max_D \; \mathbb{E}_{x \sim p_{\text{data}}}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z))\big)\big],$$

and they prove that, with unlimited capacity, the global optimum is reached exactly when the generator's distribution equals the data distribution, at which point the discriminator outputs 1/2 everywhere (§4.1, Theorem 1, and Figure 1). In practice they train the generator to maximize $\log D(G(z))$ instead, because the original term gives weak gradients early in training (§3). The authors list the method's disadvantages too: the model gives no explicit formula for the generator's distribution, and the two networks must be kept in step during training, since a generator trained too far ahead of its discriminator can collapse many different noise inputs onto the same output (§6).

Diffusion models. Ho, Jain and Abbeel (2020) developed a third approach, building on diffusion probabilistic models introduced by Sohl-Dickstein et al. (2015). A fixed forward process gradually destroys an image by adding small amounts of Gaussian noise over $T$ steps, until nothing but noise remains; in their experiments $T = 1{,}000$ and the noise variance at each step rises linearly from $10^{-4}$ to 0.02 (§4). A neural network learns the reverse process, removing a little noise at each step. With their parameterization the network is trained to predict the noise that was added to a noisy image, and they found that a simplified, unweighted version of the variational bound, the mean squared error between the true and predicted noise, gave the best samples (§3.2 and §3.4). To generate, start from pure noise and apply the learned denoising step 1,000 times. On the CIFAR-10 images they reported an Inception score of 9.46 and a state-of-the-art FID of 3.17 (abstract), measures discussed below. Adding a text input that the denoising network conditions on, so that the removal of noise is steered toward a written description, turns such a model into a text-to-image generator.

Autoregressive models of text

Text is generated differently. The chain rule of probability factors the probability of any sequence of tokens $x_1, \ldots, x_n$ exactly:

$$p(x_1, \ldots, x_n) = \prod_{t=1}^{n} p(x_t \mid x_1, \ldots, x_{t-1}).$$

An autoregressive language model learns the conditional distribution on the right, the probability of each possible next token given everything before it. Training is maximum likelihood: maximize the log-probability the model assigns to each actual next token in a large body of text, which is the same as minimizing the cross-entropy loss of Loss Functions for Classification with the vocabulary as the set of classes. Generation applies the factorization forward: sample a token from $p(x_1)$, then one from $p(x_2 \mid x_1)$, and so on, appending each to the input. The language article describes the transformer decoder that computes these conditionals.

The library's Language Models from Scratch builds such a model in five steps, from a bigram model whose conditionals depend only on the previous character to the full Shakespeare GPT with 10,788,929 parameters, and its introduction shows the same prompt producing two different poems from a chat model because each token is sampled. How the sampling is done, greedily or with a temperature, changes the output a great deal, and the prompt engineering article works through it.

Scaling

Language models improved for several years mostly by being made larger and trained on more data, and the improvement turned out to be strikingly regular. Kaplan et al. (2020) trained transformer language models from 768 to 1.5 billion non-embedding parameters on up to 23 billion tokens of web text, and found that the test loss followed power laws in model size, dataset size and training compute, "with some trends spanning more than seven orders of magnitude", while details such as the width and depth of the network mattered little within a wide range (abstract, §3). When performance is limited only by the number of non-embedding parameters $N$, the loss in nats per token is about

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \qquad \alpha_N \approx 0.076, \quad N_c \approx 8.8 \times 10^{13}$$

(Kaplan et al., 2020, eq. 1.1). The exponent is what matters. Doubling the model multiplies the loss by $2^{-0.076} = 0.949$, a 5% reduction, as the paper notes; a tenfold increase multiplies it by $10^{-0.076} = 0.840$. Taken literally, the formula gives losses of 4.02 nats at a million parameters, 2.38 at a billion and 1.99 at ten billion. Each further 16% reduction costs ten times the parameters, and the paper cautions that the constants depend on the vocabulary and tokenization and "do not have a fundamental meaning" (§1.2).

Kaplan et al. concluded that for a fixed compute budget it is best to train very large models on relatively modest amounts of data (abstract). Hoffmann et al. (2022) revisited the question with over 400 models from 70 million to over 16 billion parameters and reached a different answer: model size and training tokens should be scaled equally, "for every doubling of model size the number of training tokens should also be doubled" (abstract). They tested the prediction by training Chinchilla, a 70-billion-parameter model, on 1.4 trillion tokens, with the same compute as a larger predecessor, and found it outperformed that model while being cheaper to use (§1).

Two cautions apply. A smooth fall in average loss does not mean every capability improves smoothly: the library's note on Predictability and Surprise in Large Generative Models summarizes the argument that predictable loss and unpredictable specific capabilities go together. And the data matter as well as their quantity: the notes on Learning from Repeated Data and A small number of samples can poison LLMs of any size describe how a small repeated or poisoned fraction of the training text can have outsized effects.

Scale also changed how models are used. Brown et al. (2020) trained GPT-3, with 175 billion parameters, and found that it could perform many tasks from a description and a few examples written into its input, with no further training, and that this ability grew more rapidly with model size than performance without examples did (Figure 1.3). That finding is the starting point of the prompt engineering article.

From a pretrained model to an assistant

A model trained only to predict the next token continues text; it does not necessarily follow instructions or decline harmful requests. Ouyang et al. (2022) fine-tuned GPT-3 first on demonstrations of desired responses written by labelers, then with reinforcement learning from human feedback (RLHF): labelers rank several outputs for the same prompt, a reward model is trained to predict their rankings, and the language model is optimized against the reward model. Labelers preferred the outputs of their 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3 (Figure 1), a reminder that how a model is trained can matter as much as its size. The library's Anthropic Research collection follows one developer's versions of this step in its thread on Constitutional AI and written principles, from RLHF with crowdworker preferences to feedback generated from written principles and a long published constitution.

Current systems are also multimodal: one model accepts and produces several kinds of data, such as text and images, and can caption a photograph or answer a question about a chart. The note on Mapping the Mind of a Large Language Model reports internal features in one such model that respond to the same concept whether it appears in text or in an image.

Evaluating generated output

A classifier can be scored against labels; a generator has no single right answer. Three approaches are common, and each misses something.

  • Likelihood. A model with a computable probability can be scored by the log-likelihood it gives held-out data, as the digit model was above and as language models are by their test loss. Likelihood measures how well the model covers the data, not how good its samples look, and Holtzman et al. (2020) found that for text the most probable continuations are often repetitive and bland, so the highest-likelihood output is not the best output.
  • Sample statistics. For images, the Inception score and the Fréchet Inception distance (FID) compare features of generated and real images computed by a pretrained image classifier; Ho et al. (2020) report both. They are cheap and repeatable but depend on the classifier and say little about whether an image matches its prompt.
  • Human judgment. People rate or compare outputs, as in Ouyang et al. (2022). It is the most direct measure and the most expensive, and it rewards what raters notice, which is not always what is correct.

Two further checks apply to any generative model. Whether it reproduces its training data: Carlini et al. (2021) extracted hundreds of verbatim passages from a language model, including personal information (Risks and Ethics in AI). And whether its output is factually right, which no statistic of the output distribution measures.

In Python

The generative digit model takes a dozen lines with scikit-learn and NumPy:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

X, y = load_digits(return_X_y=True)
pca = PCA(n_components=20, random_state=0).fit(X)
Z = pca.transform(X)
rng = np.random.default_rng(0)

def sample(digit, n):
    Zc = Z[y == digit]
    z = rng.multivariate_normal(Zc.mean(0), np.cov(Zc, rowvar=False), size=n)
    return np.clip(pca.inverse_transform(z), 0, 16).reshape(n, 8, 8)

sevens = sample(7, 8)        # eight new 7s

Classification with the same model applies Bayes' theorem to the class densities, which scipy.stats.multivariate_normal(...).logpdf computes. Deep generative models are built with PyTorch or similar libraries; the Hugging Face diffusers library provides pretrained diffusion models and transformers provides pretrained language models. The comparison with logistic regression, the densities, the nearest-neighbor distances and the scaling arithmetic in this article are computed by tools/experiments/data-science/generative-ai.py in the site's repository.

References

  • Alpaydin, E., and Kaynak, C. (1998). Optical Recognition of Handwritten Digits [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C50P49 record · 2 pages
  • Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems 33. arXiv:2005.14165. record · 4 pages
  • Carlini, N., Tramèr, F., Wallace, E., et al. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (pp. 2633–2650). Read in the arXiv version, arXiv:2012.07805. record · 2 pages
  • Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems 27. arXiv:1406.2661. record
  • Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33. arXiv:2006.11239. record
  • Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556. record · 2 pages
  • Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. In International Conference on Learning Representations. arXiv:1904.09751. record · 2 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. §4.4. record · 30 pages
  • Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv:2001.08361. record · 3 pages
  • Kingma, D. P., and Welling, M. (2014). Auto-encoding variational Bayes. In International Conference on Learning Representations. arXiv:1312.6114. record
  • Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35. arXiv:2203.02155. record
  • Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning. Cited by attribution (Ho et al., 2020); not read for this article. record