This collection is my undergraduate walkthrough of building a small GPT (Generative Pre-trained Transformer) language model from nothing but PyTorch, one step at a time, and training it on about a megabyte of Shakespeare. The walkthrough follows Andrej Karpathy's lecture "Let's build GPT: from scratch, in code, spelled out" (Karpathy, 2023) and the code that accompanies it, and adds explanation at every step. The code is in my repository at github.com/mdb42/ml-demos/tree/master/gpt_text_model: train_gpt.py is the finished character-level model, train_gpt_tokenized.py is a later version that reads subword tokens instead of characters, and the steps/ folder holds one complete, runnable program for each step, so that any stage can be run and compared with the next.
These pages are a revised edition of the walkthrough I wrote as an undergraduate. In 2026 they were corrected and expanded, with the code and numbers checked against the repository and the training text, and cited claims against their sources; where a correction changes what the original said, the article notes it.
The articles show the code as the change from one step file to the next, in unified diff form (removed lines start with -, added lines with +), generated from the repository files by a script rather than retyped.
What a GPT language model does
A GPT model predicts the next token in a sequence. Given some text, it produces a probability for every token in its vocabulary to come next; generating text means sampling one token from that distribution, appending it, and repeating. Everything such a model can do, from answering questions to writing code, is built from this one operation repeated, with the patterns it uses learned from the text it was trained on.
The sampling is what makes the output vary. Given the prompt "Compose a short poem about the power of language." twice, a chat model gave two different answers. The first:
Thy power lies not in thy form alone,
But in the meaning that thou dost convey.
A single word can spark a revolution,
Or mend a broken heart, or light the way.
The second:
Language, the lifeblood of the human mind,
A tapestry of words, both bold and kind.
With every phrase, a new world comes to light,
A bridge that spans the darkness of the night.
Neither answer is retrieved from memory. Each token was drawn from a probability distribution conditioned on everything before it, so the same prompt can start down different paths. The model built in this collection works the same way, at a far smaller scale: its tokens are single characters, and its training text is one file.
The Transformer and "Attention Is All You Need"
The "T" in GPT is the Transformer, introduced by Vaswani et al. (2017) for machine translation. Earlier sequence models such as recurrent networks (RNNs and LSTMs) read a sequence one position at a time and pass a summary forward, so information from early tokens must survive many steps to reach later ones, and the steps cannot run in parallel. The Transformer replaces recurrence with attention: every position computes, directly, how relevant every other position is to it, through learned queries, keys and values, and takes a weighted average of their information. The paper's title reflects its finding that attention, together with position information and position-wise feedforward layers, was enough to outperform the recurrent and convolutional models of the time on translation (Vaswani et al., 2017, Section 6.1 and Table 2).
GPT models keep only the decoder half of the original architecture: a stack of blocks in which each position may attend only to earlier positions, so that the model can be trained to predict each next token. The articles here build exactly that, piece by piece, and Assembling the Transformer Block compares the result with the original paper's design.
Tiny Shakespeare as the training text
The training text is tiny-shakespeare, a single file of 1,115,394 characters from Shakespeare's plays (speaker names, then their lines) that Karpathy distributed with his char-rnn project, where it is described as "a subset of works of Shakespeare" (Karpathy, 2015). It is small enough to train on one GPU (nanoGPT's README reports about 3 minutes on one A100 for a character-level model with the same depth, width and context length as the final one here; Karpathy, 2022), yet rich enough that a model must learn spelling, punctuation, line structure and the format of a play to imitate it. The model is far smaller than ChatGPT, but it has every essential component of a GPT: token and position embeddings, masked multi-head self-attention, feedforward layers, residual connections, layer normalization, and a language-modeling head.
Prerequisites
The walkthrough assumes comfort with Python, a basic understanding of neural networks and backpropagation, and the calculus and linear algebra behind gradient descent and matrix multiplication. PyTorch is introduced as it is used.
The articles, in order
The build is divided into five phases, followed by a later change of tokenizer.
- Tiny Shakespeare Data Pipeline: downloading the text, building a character vocabulary, encoding the text as integers, the training and validation split, and sampling batches of fixed-length chunks.
- The Bigram Language Model: the simplest neural language model, a table that predicts the next character from the current one; cross-entropy loss, sampling, the training loop, GPU use and loss estimation.
- Building Self-Attention Step by Step: from averaging the past with a loop, to a matrix product with a lower-triangular mask, to softmax weights, to queries, keys and values with scaled dot-product attention; embeddings, a language-modeling head and positional embeddings along the way.
- Assembling the Transformer Block: multi-head attention, the feedforward layer, the Transformer block, stacking blocks, residual connections, projections and layer normalization.
- Scaling the Shakespeare GPT: dropout, the final hyperparameters, checkpointing, sample generation during training, the trained model's output, and what that output says about meaning.
- BPE Tokenization for the Shakespeare GPT: the later version that replaces characters with byte-level BPE subword tokens.
How the walkthrough's sections map to the step files
The walkthrough is divided into numbered sections (1.1 to 5.15), and the step files in steps/ are named train_gpt_<phase>_<step>.py. The numbers match exactly only in phase 4. Elsewhere a file usually carries the number of the section after the one it implements, and some files implement several sections at once. The table gives the correspondence, found by comparing each file with the next.
| Phase | Walkthrough sections | Step file |
|---|---|---|
| 1 | 1.1 (project layout) | none |
| 1 | 1.2, 1.3 (download, test) | train_gpt_1_3.py |
| 1 | 1.4 (vocabulary) | train_gpt_1_4.py |
| 1 | 1.5, 1.6 (encode, tensor) | train_gpt_1_7.py |
| 1 | 1.7, 1.8, 1.9, 1.10, 1.11 | train_gpt_1_8.py to train_gpt_1_12.py |
| 2 | 2.1, 2.2, 2.3 (model, forward, shape test) | train_gpt_2_3.py |
| 2 | 2.4, 2.5, 2.6 | train_gpt_2_4.py, _2_5, _2_6 |
| 2 | 2.7, 2.8, 2.9 (GPU, loss estimation, loop) | train_gpt_2_9.py |
| 3 | 3.1, 3.2, 3.3 | train_gpt_3_1.py, _3_2, _3_3 |
| 3 | 3.4, 3.5 (embedding size, linear layer) | train_gpt_3_6.py |
| 3 | 3.6, 3.7 | train_gpt_3_7.py, _3_8 |
| 3 | 3.8 to 3.12 (the attention head, built up) | train_gpt_3_13.py |
| 3 | 3.13, 3.14, 3.15 | train_gpt_3_14.py, _3_15, _3_16 |
| 3 | none (context cropping in generate) |
train_gpt_3_17.py |
| 3 | end of 3.15 (try other learning rates) | train_gpt_3_18.py |
| 4 | 4.1 to 4.10 | train_gpt_4_1.py to train_gpt_4_10.py |
| 5 | 5.1, 5.2 (depth parameter, hyperparameters) | train_gpt_5_3.py |
| 5 | 5.3, 5.4 | train_gpt_5_4.py, _5_5 |
| 5 | 5.5 to 5.10 (dropout everywhere, final sizes) | train_gpt_5_11.py |
| 5 | 5.11, 5.12, 5.13 | train_gpt_5_12.py, _5_13, _5_14 |
| 5 | 5.14, 5.15 (training run, samples) | none; train_gpt.py adds an interactive menu |
Each article links the step files it discusses.
Hyperparameters by phase
The model grows in stages. The table lists the settings at the end of each phase, read from the step files, with the parameter counts computed exactly from the layer sizes (the script is tools/experiments/gpt/param_counts.py in this site's repository). The last column is the lecture code gpt.py from Karpathy's ng-video-lecture repository, which the final model follows.
| Setting | Phase 2 (2_9) |
Phase 3 (3_18) |
Phase 4 (4_10) |
Phase 5 (train_gpt.py) |
Lecture gpt.py |
|---|---|---|---|---|---|
| Batch size | 32 | 32 | 32 | 64 | 64 |
Context length (block_size) |
8 | 8 | 8 | 256 | 256 |
Embedding size (C) |
65 (table) | 32 | 32 | 384 | 384 |
| Attention heads × head size | none | 1 × 32 | 4 × 8 | 6 × 64 | 6 × 64 |
| Transformer blocks | none | none | 3 | 6 | 6 |
| Dropout | none | none | none | 0.2 | 0.2 |
| Learning rate | 1e-3 | 1e-4 | 1e-4 | 3e-4 | 3e-4 |
| Training steps | 10,000 | 10,000 | 10,000 | 10,000 | 5,000 |
| Optimizer | Adam | Adam | Adam | Adam | AdamW |
| Parameters | 4,225 | 7,553 | 42,369 | 10,788,929 | 10,788,929 |
The finished model and the lecture's model have identical layer shapes, hence the same parameter count. They differ in the optimizer, the number of training steps, where dropout is applied, and weight initialization (the lecture code initializes weights from a normal distribution with standard deviation 0.02; this code keeps PyTorch's defaults). Scaling the Shakespeare GPT lists the differences.
Credits and sources
The build follows Karpathy's lecture (Karpathy, 2023) and its code, the ng-video-lecture repository, which its README describes as the code from the lecture and gives under the MIT license; Karpathy's nanoGPT repository, also MIT licensed, is the fuller version of the same model (Karpathy, 2022). The architecture is that of Vaswani et al. (2017), as adapted for GPT-2 by Radford et al. (2019). The text comes from Karpathy's char-rnn repository (Karpathy, 2015). The walkthrough's words, its step-by-step programs, and its training runs are my own; numbers reported from those runs are attributed to them in each article.
References
- Karpathy, A. (2015). char-rnn [Software repository, including
data/tinyshakespeare/input.txt]. https://github.com/karpathy/char-rnn - Karpathy, A. (2022). nanoGPT [Software repository, MIT license]. https://github.com/karpathy/nanoGPT
- Karpathy, A. (2023). Let's build GPT: from scratch, in code, spelled out [Video lecture]. YouTube. https://www.youtube.com/watch?v=kCc8FmEb1nY. Code: ng-video-lecture, https://github.com/karpathy/ng-video-lecture
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI technical report.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30. arXiv:1706.03762.