At the end of Assembling the Transformer Block the model has the full GPT architecture at toy size: 32-dimensional embeddings, 8 characters of context, 42,369 parameters. This last phase of the walkthrough makes it big enough to write recognizable Shakespearean dialogue. It makes the depth configurable, adds dropout to keep the larger model from memorizing its training text, raises every size to its final value, and adds the machinery that a long training run needs: checkpoints that save progress, and text samples that show what the model has learned so far. It ends with what the trained model wrote, and what such writing does and does not show.
The code for this phase is in steps/5_scaling the_model (the folder name contains a space), and the finished program is train_gpt.py. Parameter counts come from tools/experiments/gpt/param_counts.py and the checks on the samples from tools/experiments/gpt/sample_check.py, both in this site's repository. No model was trained for this article: training losses and samples are those reported in the walkthrough, from the run it describes.
Configurable depth and hyperparameters in one place
How deep should the model be? More blocks give more processing, at more cost. Rather than hard-coding three blocks, the next step file builds n_layer of them with a list comprehension, and moves the number of heads and the head size up to the hyperparameters at the top of the program, so that every architectural choice is set in one place. Step file: train_gpt_5_3.py, which covers the walkthrough's sections 5.1 and 5.2.
--- train_gpt_4_10.py
+++ train_gpt_5_3.py
@@ -16,2 +16,5 @@
num_train_steps = 10000
+n_layer = 3 # Number of Transformer blocks
+num_heads = 4 # Number of attention heads
+head_size = C // num_heads # Size of each head (C is the total embedding dimension)
@@ -175,3 +178,3 @@
class BigramLanguageModel(nn.Module):
- def __init__(self):
+ def __init__(self, n_layer, num_heads, head_size):
super().__init__()
@@ -180,10 +183,5 @@
- num_heads = 4 # Number of attention heads
- head_size = C // num_heads # Size of each head (C is the total embedding dimension)
-
- # Stack multiple Transformer Blocks
+ # Stack multiple Transformer Blocks dynamically based on n_layer
self.transformer_blocks = nn.Sequential(
- TransformerBlock(num_heads, head_size), # First Transformer Block
- TransformerBlock(num_heads, head_size), # Second Transformer Block
- TransformerBlock(num_heads, head_size) # Third Transformer Block (you can add more)
+ *[TransformerBlock(num_heads, head_size) for _ in range(n_layer)] # Stack n_layer blocks
)
@@ -216,2 +214,3 @@
return logits
+
@@ -229,5 +228,4 @@
-
-
-model = BigramLanguageModel()
+# Create the BigramLanguageModel with the required arguments
+model = BigramLanguageModel(n_layer=n_layer, num_heads=num_heads, head_size=head_size)
The settings are interdependent. The batch size sets how many sequences are processed in parallel; block_size how far back the model can see; C the width of every token's representation; n_layer the depth; num_heads how many attention patterns are computed in parallel; and head_size = C // num_heads how wide each is. Splitting C among the heads lets them specialize, but only if each head keeps enough dimensions to work with.
Dropout against memorization
A model with millions of parameters trained on a million characters can memorize long stretches of its text instead of learning general patterns. Dropout is a defence: during training, each element of a layer's output is set to zero with probability $p$, independently at every step, and the survivors are multiplied by $1/(1-p)$ so that the expected value of each element is unchanged; during evaluation the layer does nothing (PyTorch's nn.Dropout documentation). Because any unit may vanish, no unit can rely on a particular combination of others being present. Hinton et al. (2012) introduced the idea, reporting that randomly omitting half of the feature detectors on each training case substantially reduced overfitting. One way to see it is that every step trains a different random subnetwork, all sharing weights, and evaluation without dropout approximates averaging them.
Dropout zeroes activations, the values flowing through the network, not the weights or connections themselves. It first goes into the Transformer block, applied to the output of the attention and of the feedforward layer before each is added to the residual stream, with a default rate of 0.1. Step file: train_gpt_5_4.py.
--- train_gpt_5_3.py
+++ train_gpt_5_4.py
@@ -156,6 +156,6 @@
####################################################################################################
-# Transformer Block with Projection Layers
+# Transformer Block with Dropout
class TransformerBlock(nn.Module):
- def __init__(self, num_heads, head_size):
+ def __init__(self, num_heads, head_size, dropout=0.1):
super().__init__()
@@ -165,8 +165,10 @@
self.ln2 = nn.LayerNorm(C) # LayerNorm before feedforward network
+ self.dropout = nn.Dropout(dropout) # Dropout layer
def forward(self, x):
- # Communication through multi-head attention with LayerNorm
- x = x + self.multi_head_attention(self.ln1(x)) # Apply LayerNorm1 and residual connection
- # Computation through feedforward network with LayerNorm
- x = x + self.feed_forward(self.ln2(x)) # Apply LayerNorm2 and residual connection
+ # Communication through multi-head attention with LayerNorm and Dropout
+ x = x + self.dropout(self.multi_head_attention(self.ln1(x))) # Apply Dropout after multi-head attention
+
+ # Computation through feedforward network with LayerNorm and Dropout
+ x = x + self.dropout(self.feed_forward(self.ln2(x))) # Apply Dropout after feedforward network
return x
Placing dropout inside the residual branch regularizes what each sub-layer adds while leaving the residual path itself untouched. The model then takes the rate as a parameter and passes it to every block, so a single setting controls regularization throughout the depth. Step file: train_gpt_5_5.py.
--- train_gpt_5_4.py
+++ train_gpt_5_5.py
@@ -177,6 +177,6 @@
####################################################################################################
-# The Bigram Language Model with Final LayerNorm
+# The Bigram Language Model with Dropout
class BigramLanguageModel(nn.Module):
- def __init__(self, n_layer, num_heads, head_size):
+ def __init__(self, n_layer, num_heads, head_size, dropout=0.1):
super().__init__()
@@ -185,6 +185,6 @@
- # Stack multiple Transformer Blocks dynamically based on n_layer
- self.transformer_blocks = nn.Sequential(
- *[TransformerBlock(num_heads, head_size) for _ in range(n_layer)] # Stack n_layer blocks
- )
+ # Stack multiple Transformer Blocks with Dropout
+ self.transformer_blocks = nn.Sequential(*[
+ TransformerBlock(num_heads, head_size, dropout=dropout) for _ in range(n_layer)
+ ])
@@ -192,2 +192,3 @@
self.LM_head = nn.Linear(C, vocab_size) # Language Modeling Head
+
Dropout throughout the attention and feedforward layers
The walkthrough's sections 5.5 to 5.10 add dropout inside every component and set the final sizes; one step file, train_gpt_5_11.py, contains all of it. Its components, which are also those of the finished program, are shown here as they stand in train_gpt.py.
In each attention head, dropout is applied to the attention weights after the softmax and before they multiply the values. Each step, a random subset of the connections between positions is cut, so the model cannot depend on any one attention pattern being available:
class SelfAttentionHead(nn.Module):
def __init__(self, head_size, dropout_rate):
super().__init__()
self.head_size = head_size
self.query = nn.Linear(C, head_size, bias=False)
self.key = nn.Linear(C, head_size, bias=False)
self.value = nn.Linear(C, head_size, bias=False)
self.dropout = nn.Dropout(dropout_rate) # Dropout after attention weights
mask = torch.tril(torch.ones(block_size, block_size))
self.register_buffer('mask', mask)
def forward(self, x):
B, T, C = x.shape
Q = self.query(x)
K = self.key(x)
V = self.value(x)
affinities = Q @ K.transpose(-2, -1) / (self.head_size ** 0.5)
affinities = affinities.masked_fill(self.mask[:T, :T] == 0, float('-inf'))
attention_weights = F.softmax(affinities, dim=-1)
attention_weights = self.dropout(attention_weights) # Apply dropout to attention weights
out = attention_weights @ V
return out
A detail the walkthrough's text gets wrong: after dropout the weights in a row no longer sum to one. The surviving weights are scaled up by $1/(1-p)$ and the dropped ones are zero, so each row sums to one only on average.
In multi-head attention, dropout is applied to the output of the projection that combines the heads (the code's comment says "before projecting", but the call applies it after):
class MultiHeadAttention(nn.Module):
def __init__(self, num_heads, head_size, dropout_rate):
super().__init__()
self.heads = nn.ModuleList([SelfAttentionHead(head_size, dropout_rate) for _ in range(num_heads)])
self.projection = nn.Linear(num_heads * head_size, C)
self.dropout = nn.Dropout(dropout_rate) # Dropout after projection
def forward(self, x):
# Concatenate the outputs of all heads
out = torch.cat([head(x) for head in self.heads], dim=-1)
# Apply dropout before projecting back to original size
return self.dropout(self.projection(out))
In the feedforward layer, dropout is applied in the expanded $4C$-dimensional space, after the ReLU and before the second linear map. Since ReLU outputs are already zero wherever the input was negative, dropout here can only remove positive activations:
class FeedForwardLayer(nn.Module):
def __init__(self, C, dropout_rate):
super().__init__()
self.fc1 = nn.Linear(C, 4 * C) # Expand to 4x the embedding size
self.fc2 = nn.Linear(4 * C, C) # Project back down to embedding size
self.relu = nn.ReLU() # ReLU non-linearity
self.dropout = nn.Dropout(dropout_rate) # Dropout for regularization
def forward(self, x):
x = self.fc1(x) # First linear transformation
x = self.relu(x) # Apply ReLU
x = self.dropout(x) # Apply dropout after ReLU
return self.fc2(x) # Project back to original dimension
In the block, the dropout on each sub-layer's output from section 5.3 remains:
class TransformerBlock(nn.Module):
def __init__(self, num_heads, head_size, dropout_rate):
super().__init__()
self.multi_head_attention = MultiHeadAttention(num_heads, head_size, dropout_rate)
self.feed_forward = FeedForwardLayer(C, dropout_rate)
self.ln1 = nn.LayerNorm(C)
self.ln2 = nn.LayerNorm(C)
self.dropout = nn.Dropout(dropout_rate) # Apply dropout with the given rate
def forward(self, x):
# Multi-head attention with dropout
x = x + self.dropout(self.multi_head_attention(self.ln1(x)))
# Feedforward layer with dropout
x = x + self.dropout(self.feed_forward(self.ln2(x)))
return x
The placement differs from the lecture code. In Karpathy's gpt.py there are three dropout layers per block: on the attention weights, after the attention projection, and at the end of the feedforward layer, after its second linear map; the block itself adds none. Here the feedforward dropout sits before the second linear map instead, and the block applies its own dropout on top, so the attention output passes through two independent dropout masks in a row (each element survives with probability $0.8 \times 0.8 = 0.64$ at rate 0.2). The original Transformer applies dropout "to the output of each sub-layer, before it is added to the sub-layer input and normalized", and to the sums of embeddings and positional encodings, at a rate of 0.1 (Vaswani et al., 2017, Section 5.4). All of these are reasonable regularizers; the walkthrough's arrangement simply regularizes more heavily than the lecture's.
The final hyperparameters
The same step sets every size to its final value:
--- train_gpt_5_5.py
+++ train_gpt_5_11.py
@@ -8,7 +8,10 @@
-# Hyperparameters
-batch_size = 32
-block_size = 8
-C = 32 # Embedding dimensions
-learning_rate = 1e-4
+# Updated Hyperparameters
+batch_size = 64 # Increased batch size
+block_size = 256 # Increased block size for more context
+C = 384 # Increased embedding dimension
+learning_rate = 3e-4 # Lowered learning rate for stable training
+n_layer = 6 # Increased number of layers
+num_heads = 6 # Increased number of attention heads
+dropout_rate = 0.2 # Dropout rate for regularization
eval_interval = 100
The reasoning behind each choice, as the walkthrough gives it:
- Context and batch. A context of 256 characters instead of 8 lets the model see several lines at once, enough for the rhythm of verse and the alternation of speakers; 64 sequences per batch instead of 32 give steadier gradients. The cost of attention grows with the square of the context, since every position is compared with every earlier one.
- Width and heads. 384 dimensions split across 6 heads give each head 64 dimensions, the head size of the original Transformer (Vaswani et al., 2017, Section 3.2.2), enough for each head to learn a meaningful pattern.
- Depth. Six blocks instead of three.
- Learning rate and dropout. The learning rate is 3e-4 and the dropout rate 0.2, a stronger regularization for a model whose larger capacity makes memorization a real risk. The code's comment calls the learning rate "lowered"; it is lower than the 1e-3 of phases 2 and 3, but higher than the 1e-4 used since step 3.18.
These are exactly the settings of the lecture code (batch 64, context 256, 384 dimensions, 6 heads, 6 layers, dropout 0.2, learning rate 3e-4). The model has 10,788,929 parameters, the same as the lecture's, since the layer shapes are identical:
| Component | Parameters |
|---|---|
| Token embedding, 65 × 384 | 24,960 |
| Position embedding, 256 × 384 | 98,304 |
| Each block: query, key and value projections, 6 heads × 3 × 384 × 64 | 442,368 |
| Each block: attention output projection, 384 × 384 + 384 | 147,840 |
| Each block: feedforward, 384 × 1,536 + 1,536 + 1,536 × 384 + 384 | 1,181,568 |
| Each block: two layer normalizations, 2 × 2 × 384 | 1,536 |
| Six blocks, 6 × 1,773,312 | 10,639,872 |
| Final layer normalization | 768 |
| LM head, 384 × 65 + 65 | 25,025 |
| Total | 10,788,929 |
Nearly all the parameters, 98.6 percent, are in the blocks, and two thirds of each block is the feedforward layer. The embeddings and the LM head are small because the vocabulary has only 65 characters; with a subword vocabulary they grow large, as BPE Tokenization for the Shakespeare GPT shows.
The model class now also takes every hyperparameter as a constructor argument (walkthrough section 5.10). The configuration is not yet fully self-contained: the attention head still reads the global C and block_size, and forward and generate read the global block_size, so the arguments must match the globals. Packing the settings into a configuration object passed to every module, as nanoGPT does, would remove that coupling.
Saving and loading checkpoints
A run of 10,000 steps on a model of this size takes long enough that losing it to an interruption is costly. A checkpoint saves the model's weights and the optimizer's state to disk. The optimizer state matters because Adam keeps running averages of each parameter's gradients; restarting without them would change the course of training. Step file: train_gpt_5_12.py.
--- train_gpt_5_11.py
+++ train_gpt_5_12.py
@@ -304,2 +304,26 @@
####################################################################################################
+# Save the Model
+
+def save_checkpoint(model, optimizer, epoch, checkpoint_dir='checkpoints'):
+ if not os.path.exists(checkpoint_dir):
+ os.makedirs(checkpoint_dir)
+ checkpoint_path = os.path.join(checkpoint_dir, f'model_epoch_{epoch}.pth')
+ torch.save({
+ 'model_state_dict': model.state_dict(),
+ 'optimizer_state_dict': optimizer.state_dict(),
+ 'epoch': epoch,
+ }, checkpoint_path)
+ print(f"Checkpoint saved: {checkpoint_path}")
+
+####################################################################################################
+# Load the Model
+def load_checkpoint(model, optimizer, checkpoint_path):
+ checkpoint = torch.load(checkpoint_path)
+ model.load_state_dict(checkpoint['model_state_dict'])
+ optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
+ start_epoch = checkpoint['epoch']
+ print(f"Resuming from epoch {start_epoch}")
+ return start_epoch
+
+####################################################################################################
# Generating Text
The files are named model_epoch_<n>.pth, but n is the training step, not an epoch (a full pass over the data). Checkpoints make it possible to pause and resume training across sessions, to branch experiments from a given point, to share partly trained models, and to study how the model's output changes over training.
Resuming training from a checkpoint
The training loop now saves a checkpoint at every evaluation, every 100 steps, and the program tries to resume from a saved one before training. Step file: train_gpt_5_13.py.
--- train_gpt_5_12.py
+++ train_gpt_5_13.py
@@ -328,2 +308,31 @@
####################################################################################################
+# Training Loop
+
+optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
+
+def train_model(start_step=0):
+ for step in range(start_step, num_train_steps):
+ xb, yb = get_batch('train')
+ logits = model(xb)
+ loss = compute_loss(logits, yb)
+ optimizer.zero_grad()
+ loss.backward()
+ optimizer.step()
+
+ if step % eval_interval == 0:
+ losses = estimate_loss()
+ print(f"Step {step}, Train Loss: {losses['train']}, Val Loss: {losses['val']}")
+ # Save checkpoint at each evaluation interval
+ save_checkpoint(model, optimizer, step)
+
+# Attempt to load from a checkpoint
+start_epoch = 0
+checkpoint_path = 'checkpoints/model_epoch_9999.pth' # Replace with your checkpoint path
+if os.path.exists(checkpoint_path):
+ start_epoch = load_checkpoint(model, optimizer, checkpoint_path)
+
+# Start or resume training
+train_model(start_step=start_epoch)
+
+####################################################################################################
# Generating Text
The resume path is written by hand and must name a file that exists. Checkpoints are saved at steps 0, 100, ..., 9,900, so the path in this step, model_epoch_9999.pth, never exists and the program always starts afresh. The next step file names model_epoch_4100.pth, apparently a checkpoint from a real run, and the walkthrough's appendix names model_epoch_9900.pth. At most one interval of training, 100 steps, is lost to an interruption.
Watching the model learn through samples
Loss numbers say how well the model predicts, but not what it has learned. The next step generates text from three prompts at every checkpoint, so that its progress can be read. Step file: train_gpt_5_14.py.
--- train_gpt_5_13.py
+++ train_gpt_5_14.py
@@ -308,2 +308,21 @@
####################################################################################################
+# Generate Text
+
+def generate_text():
+ """Generates text starting from predefined prompts."""
+ scenarios = {
+ "Start with 'A'": "A",
+ "Start with 'The'": "The",
+ "Start with 'To be, or not to be, that is the '": "To be, or not to be, that is the "
+ }
+
+ for description, start_text in scenarios.items():
+ print(f"\n--- {description} ---")
+ idx = torch.tensor([encode(start_text)], dtype=torch.long).to(device)
+ generated_sequence = model.generate(idx, max_new_tokens=256)
+ generated_text = decode(generated_sequence[0].tolist())
+ print(generated_text)
+
+
+####################################################################################################
# Training Loop
@@ -312,2 +331,3 @@
+# Update the training loop to call the generate_text_scenarios function at each checkpoint
def train_model(start_step=0):
@@ -326,2 +346,4 @@
save_checkpoint(model, optimizer, step)
+ # Generate text after saving the checkpoint
+ generate_text()
@@ -329,3 +351,3 @@
start_epoch = 0
-checkpoint_path = 'checkpoints/model_epoch_9999.pth' # Replace with your checkpoint path
+checkpoint_path = 'checkpoints/model_epoch_4100.pth' # Replace with your checkpoint path
if os.path.exists(checkpoint_path):
@@ -336,8 +358,3 @@
-####################################################################################################
-# Generating Text
-
-idx = torch.zeros((1, 1), dtype=torch.long)
-generated_sequence = model.generate(idx, max_new_tokens=100)
-generated_text = decode(generated_sequence[0].tolist())
-print(generated_text)
+# Generate text after training
+generate_text()
The three prompts probe different things: A tests how the model begins a line; The tests common phrase openings; and To be, or not to be, that is the tests what the model does with the most famous line in Shakespeare. As the following sections show, that line is not in the training text at all.
Samples generated this way are produced while the model is in training mode, because estimate_loss switches it back with model.train() just before. Dropout is therefore active during sampling, and the samples are somewhat noisier than the trained model's best. Calling model.eval() before generating and model.train() after would fix this. The lecture code samples in training mode as well.
The finished program
The finished train_gpt.py replaces the fixed sequence of "load, train, generate" with a small interactive program: it loads the most recent checkpoint it finds and then accepts the commands train, load <step>, generate <prompt> and exit.
def get_path_to_latest_checkpoint():
"""Return the path to the latest checkpoint file"""
checkpoint_dir = 'checkpoints'
if not os.path.exists(checkpoint_dir):
return None
checkpoint_files = os.listdir(checkpoint_dir)
if not checkpoint_files:
return None
checkpoint_files.sort(key=lambda x: int(x.split('_')[2].split('.')[0]))
latest_checkpoint = checkpoint_files[-1]
return os.path.join(checkpoint_dir, latest_checkpoint)
def main():
# Attempt to load from a checkpoint
start_epoch = 0
checkpoint_path = get_path_to_latest_checkpoint()
if os.path.exists(checkpoint_path):
start_epoch = load_checkpoint(model, optimizer, checkpoint_path)
run()
Some rough edges remain in the finished program:
- On a first run with no
checkpointsfolder,get_path_to_latest_checkpointreturnsNone, andos.path.exists(None)raises aTypeErrorinstead of returningFalse, so the program stops before its menu. generatetakes only the first word of the prompt (command.split()[1]).- The menu lists a
helpcommand that the loop does not recognize. mainreads the step number of the checkpoint it loads intostart_epochbut never uses it: thetraincommand callstrain_model()with its defaultstart_step=0, so training after a resume runs a full 10,000 steps again from step 0 and overwrites the earlier checkpoint files.- Nothing calls
model.eval()outsideestimate_loss, so thegeneratecommand, like the samples during training, runs with dropout active.
The tokenized version fixes the first two; the other three remain in it.
What training looks like
The walkthrough describes what to expect over a run, in three stages:
- Early (about the first 20 percent). The loss drops quickly as the model learns basic character patterns; generated text begins to show structure, such as spacing and capitalization; simple common words start to appear.
- Middle (about 20 to 60 percent). The loss falls more slowly as the model works on subtler patterns; coherent phrases and basic grammar appear, along with Shakespeare's vocabulary.
- Late (the final 40 percent). Improvements in the loss are small; the text looks increasingly like Shakespeare's style and the layout of a play script, and longer-range patterns appear.
In the run described in the walkthrough, the validation loss fell from 2.07 to 1.48 (the walkthrough does not say at which steps these were measured). For comparison, the nanoGPT README reports a best validation loss of 1.4697 for its character-level model with the same depth, width and context on the same text (Karpathy, 2022), and the bigram models of The Bigram Language Model cannot go below about 2.45 even on their training text. In perplexity terms, 1.48 nats per character is $e^{1.48} \approx 4.4$: the model is, on average, about as uncertain as a choice among four or five equally likely characters, against 65 for a uniform guess.
Shakespeare-like text from the trained model
The walkthrough reports these samples from the trained model. The first:
GLOUCESTER:
These art but bleeding love, these words:
Welcome, I bid thee, Romeo
I mean to live as I am, drunk with thee,
And spelling the rest in honor
I know a perfect request, but I have not need
The model has learned the form of the text: a speaker's name in capitals followed by a colon, short lines, archaic pronouns ("thee"), and phrases with the sound of Elizabethan verse ("bleeding love"). It has also crossed plays: in the training text, Gloucester is a speaker in the history plays (he opens Richard III with "Now is the winter of our discontent"), while Romeo belongs to Romeo and Juliet. A second sample:
In suspicion'd applaud unpleased,
In rain, expectantly to steel death's bones;
And three fallows I leave and behold thy infection
Rid me of size, and not another dream
Several of these words do not occur anywhere in the training text: "suspicion'd", "fallows", "expectantly" and "spelling" are absent, and "honor" (from the first sample) appears there only as "honour". The model composed them from familiar pieces, for example the 'd ending of verbs such as "banish'd" and the plural s, rather than copying them. That is what a character-level model can do and a word-level model with a fixed vocabulary cannot. The lines do not, however, keep a consistent meter or a coherent meaning.
What interested me most personally was the famous prompt. The model continued it differently each time and never with Hamlet's actual words:
To be, or not to be, that is the earth
Which, shown now, I was in for hell!
To be, or not to be, that is the fruit
Given all who swears to hunger
To be, or not to be, that is the point
Of sword drawn and arrows fallen
The walkthrough first read this as a sign that dropout had kept the model from overfitting to the phrase. The training text cannot support that reading, because Hamlet is not in tiny-shakespeare: the words "To be, or not to be" and "that is the question" never occur in it, and no speaker is named Hamlet. The model could not reproduce a line it never saw, with or without dropout. What the samples do show is that it continues an unfamiliar prompt in the style it learned, and differently on each draw, which is the sampling at work. The phrase "that is the" itself occurs only three times in the training text.
Just as often, the model produces text that keeps the structure of dialogue while saying nothing:
BRUTUS:
Fawn, of 'tis not whom I
So doth I oft bride my heart to joys,
These state a tread in persuasion cut off.
Lord Mayor:
Into the shapes should be spoken by
Two on the been which of of the
The format is right: speaker names that occur in the training text, line lengths, punctuation. The content is not: "Two on the been which of of the" is a string of common words with no grammar.
The shape of meaning
"Words without thoughts never to heaven go," says Claudius in Hamlet (Act 3, Scene 3), the play the model never read. The technical side of building a language model proved surprisingly approachable: with free tools, public-domain text and the basic principles of machine learning, we built a system that generates text with much of the look of Shakespeare. As its output shows, though, not all of that text means anything. Much of it keeps the expected format and structure without conveying information; we might say it is information-shaped without actually saying anything. With incoherence disguised as real information, the harder problem is not building these models but interpreting their outputs reliably and usefully.
That problem grows with the models. If a small character-level model can produce output that makes us pause over its meaning, interpretation will be harder for systems whose outputs are far more convincing. And if models eventually reason better than people in some domains, it becomes a real question how we would tell when a model is being incoherent from when we have reached the limits of our own understanding. The technical barriers to building language models are lower than one might expect; the philosophical and practical challenges of understanding what they produce remain.
Beyond one sample at a time
This model generates text one character at a time along a single path. Large language models are often used differently. Chain-of-thought prompting has a model write out a series of intermediate reasoning steps before its answer, which Wei et al. (2022) showed improves large models' performance on arithmetic, commonsense and symbolic reasoning tasks. Tree of Thoughts (Yao et al., 2023) goes further: the model proposes several candidate intermediate steps, evaluates them itself, and searches over the resulting tree with lookahead and backtracking, rather than committing to one path. Extending a small generator like this one with such a search would be one way to experiment with more deliberate generation. Both move from plain next-token prediction towards structured inference built on top of it.
Final thoughts on training a GPT model
Training a character-level GPT on Shakespeare teaches more than its implementation. As the model learned Shakespeare's patterns one character at a time, building it meant breaking a complex architecture into its basic components, and each step, from bigram predictions to multi-head attention, showed something about language modeling. The gap between this model and industrial ones is vast: they use subword tokens rather than characters, and train on a large share of the world's written text rather than one author. But the principles are the same: attention to context, learned patterns, and probabilistic prediction. Natural next experiments are other tokenizations (the next article, BPE Tokenization for the Shakespeare GPT, tries one), larger datasets, and more deliberate decoding.
References
- Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580.
- Karpathy, A. (2022). nanoGPT [Software repository, MIT license]. https://github.com/karpathy/nanoGPT. Cited: README, validation loss of the shakespeare_char model.
- Karpathy, A. (2023). Let's build GPT: from scratch, in code, spelled out [Video lecture]. https://www.youtube.com/watch?v=kCc8FmEb1nY. Code: https://github.com/karpathy/ng-video-lecture
- PyTorch.
torch.nn.Dropoutdocumentation (scaling by $1/(1-p)$ during training, identity during evaluation). https://docs.pytorch.org/docs/stable/generated/torch.nn.Dropout.html - Shakespeare, W. Hamlet, Act 3, Scene 3. Folger Shakespeare Library edition, https://www.folger.edu/explore/shakespeares-works/hamlet/
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30. arXiv:1706.03762. Cited: Section 3.2.2 (head size 64), Section 5.4 (residual dropout).
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903.
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36. arXiv:2305.10601.
- Code for this article: gpt_text_model/steps/5_scaling the_model and train_gpt.py in github.com/mdb42/ml-demos.