← back to lessons
A visual guide · no PhD required

How does an LLM
actually learn?

From a blank model that knows nothing, all the way to a model that you can have a conversation with. No equations. Just clear thinking.

① The brain analogy
② Making a prediction
③ Measuring the mistake
④ Backpropagation
⑤ Pre-training
⑥ RLHF
scroll
CH · 01

The network of dials

Forget everything you know about how a computer normally works — rules, if-statements, lookup tables. An LLM works nothing like that. It is, at its core, a gigantic collection of numbers.

Imagine a wall covered in millions of physical dials. Each dial can be turned to any value — positive or negative, big or small. Those dials are called weights (or parameters). A large model like GPT-4 has around 1.8 trillion of them.

Think of the model as a city-sized mixing console. Every dial affects the sound. At the start, all dials are set to random positions. The music it produces is pure noise. Training is the process of turning all those dials — slowly, carefully — until the music sounds right.

— The Weights Metaphor

These weights are organized into layers. Each layer takes in some numbers, multiplies them by its weights, adds them up, and passes the result to the next layer. That's it. That simple operation, repeated thousands of times across hundreds of layers, is what produces language.

// Interactive — A tiny neural network

Hover over a node to see how it connects to the next layer. This tiny net has 3 layers. A real LLM has hundreds.

Key idea
The model has no rules hard-coded into it. Everything it "knows" — grammar, facts, reasoning — is encoded as patterns in those billions of weight values.
CH · 02

Making a prediction

When you type a prompt, the model's job is to predict: "what token should come next?" A token is roughly a word (or part of a word). The model doesn't pick one deterministically — it produces a probability distribution over the entire vocabulary.

Imagine asking the model to complete: "The sky is ___". The model assigns a probability to every word in its vocabulary — maybe blue gets 42%, falling gets 8%, beautiful gets 6%, and millions of other words share the rest. Then one is sampled.

// Interactive — probability distribution

Context: "The sky is ___" — drag the temperature slider to see how it affects confidence.

Temp
0.8

Low temperature = model is more decisive. High = more creative (and unpredictable).

Why probabilities?
Because language is not deterministic. "The sky is" can be followed by dozens of valid words. The model captures this uncertainty as a distribution, not a single answer.
CH · 03

Measuring the mistake

Before a model can learn, it needs to be able to measure how wrong it was. That measurement is called the loss (or cost). Think of it as a single number that tells you: "how bad was this prediction?"

If the correct next word was "blue" and the model only gave it 2% probability, the loss is high. If the model gave it 85% probability, the loss is low. The specific formula used is called cross-entropy loss, but all you need to know is: higher loss = worse prediction.

Imagine you're a student taking a test where you don't just pick one answer — you say "I'm 60% sure it's A, 30% B, 10% C." If the answer was A, you did well. If the answer was C and you said 10%... you're in trouble. The loss quantifies exactly how in trouble you are.

— The Confident Student Metaphor

During training, the loss is computed for millions of predictions. The goal of training is to minimize this loss — to turn those dials until the model assigns high probability to correct tokens.

// A loss curve over time

As training progresses, the loss falls. The model goes from random noise to meaningful predictions.

CH · 04

Backpropagation — the magic dial-turner

Here is the key question: once you know the loss, how do you know which dials to turn, and in which direction?

The answer is backpropagation — and it's one of the most elegant ideas in all of computer science. Here's the narrative:

Step 1 — The forward pass. The input flows forward through all the layers. Every single multiplication, every addition is recorded. The model builds a kind of "receipt" of every computation it did.

Step 2 — The loss is computed. At the end of the forward pass, we compare the prediction to the correct answer and get a loss number.

Step 3 — The backward pass. Now we go backwards through that receipt, using calculus (the chain rule). For each weight, we compute a gradient — a number that says: "if you increase this weight by a tiny amount, does the loss go up or down, and by how much?"

Step 4 — Update the weights. Each weight is nudged in the direction that reduces the loss, by a tiny amount called the learning rate. That's one step of training. Now repeat this billions of times.

Imagine the loss is a ball rolling down a hilly landscape. The gradient tells you which direction the hill goes up from where you're standing. You take a small step in the opposite direction — downhill. Repeat millions of times. Eventually you settle in a valley. That valley is the trained model.

— Gradient Descent as a hiker
The crucial insight
Backprop doesn't need to "guess" which weights matter. The mathematics guarantees that every weight receives a gradient that precisely reflects how much it contributed to the error. The network doesn't know it's learning language — it's just following math down a hill.
// How a single weight gets updated

Each training step, a weight is pushed in the direction that reduces the loss. Click to simulate one step.

Weight value: 0.34
Gradient (slope): +0.18  →  step down
CH · 05

Pre-training — reading the internet

Now you know the mechanics. Here's how they're applied at scale.

During pre-training, the model is given a simple task: predict the next token. That's it. The training data is an enormous corpus of text — books, articles, code, websites. Trillions of tokens.

For each sentence in that corpus, the model is shown the beginning and asked to predict what comes next. It predicts. The actual next word is revealed. Loss is computed. Backprop runs. Weights are updated. Then the next sentence. Then the next. For months, on thousands of chips running in parallel.

Why does this produce "intelligence"?
Because to predict language well, you have to understand it. To predict the next word in a medical text, you have to learn medicine. To predict the next line of code, you have to learn programming. The simple task of next-token prediction forces the model to compress a huge understanding of the world into its weights.

At the end of pre-training you have a base model: a model that is extraordinary at predicting and generating text, but has no concept of being helpful, honest, or having a conversation. It would complete "How do I make a bomb?" as naturally as "The sunset was beautiful." It's a text predictor, not an assistant.

// Scale of pre-training
Training tokens Trillions. A trillion tokens is roughly 750 billion words — more than all books ever written, many times over.
Training steps Hundreds of billions of individual weight updates, each one a tiny gradient descent step.
Hardware Thousands of specialized chips (GPUs/TPUs) running in parallel for weeks or months.
Result A base model — brilliant text predictor, but not yet a useful assistant.
CH · 06

RLHF — teaching it to be good

The base model is powerful but wild. To turn it into a helpful assistant, we use Reinforcement Learning from Human Feedback (RLHF). Here's how it works, step by step.

Step 1 — Supervised Fine-Tuning. First, humans write examples of ideal conversations: a question and a perfect answer. The model is trained on these examples using standard backprop. It begins to learn the format and tone of being helpful.

Step 2 — Train a Reward Model. Now humans are shown pairs of answers and asked: which one is better? These preferences train a separate small model — the reward model — whose sole job is to score an answer from 0 to 10 based on how good it is. Think of it as a judge that has been calibrated on thousands of human opinions.

Step 3 — Reinforcement Learning. Now the main model generates an answer. The reward model scores it. And here is where your original question lives:

The key mechanism
A high reward score becomes a training signal — just like the "correct answer" in pre-training. It gets fed back through backpropagation. The math of backprop traces exactly which weights contributed to generating those tokens, and nudges each one to make those tokens more probable next time. A low reward does the reverse.

There's one more piece: a KL penalty. During RL, the model could learn to "cheat" — to generate nonsense that happens to fool the reward model. The KL penalty punishes the model if it drifts too far from the original base model, keeping it grounded.

// Reward signal → weight update

Same two responses to the same question. Different rewards. Different effects on the weights.

High reward response

"The capital of France is Paris, which has been the country's capital since the 10th century..."

+8.4
↑ Weights adjusted to make this more likely
Low reward response

"France is a country in Europe and the answer to your question can be found by searching online..."

+1.2
↓ Weights adjusted to make this less likely

Backprop doesn't need to "find" which weights are responsible. The forward pass recorded every computation. The gradient flows back through that same computational graph automatically.

RLHF is like a chef being trained by a food critic. The critic doesn't explain why a dish is good or bad — they just rate it. But the chef, who knows exactly how they made every dish, can work backwards from the rating to understand which ingredients, techniques, and proportions to adjust. Backprop is that "working backwards."

— The Chef & Critic Metaphor
SUMMARY

The full picture

Weights Billions of numbers. Everything the model "knows" lives here.
Forward pass Input flows through layers, producing a probability over next tokens.
Loss A number measuring how wrong the prediction was.
Backpropagation The chain rule flows the error signal backwards. Each weight learns exactly how much it contributed to the mistake.
Gradient descent Each weight is nudged in the direction that reduces the loss. Repeat billions of times.
Pre-training Predict the next token across trillions of words. Forces the model to understand the world to do well.
RLHF Replace the "correct token" signal with a human preference score. Backprop still does the work. Same mechanism, different teacher.
The one sentence to remember
An LLM learns by making predictions, measuring how wrong they were, and using mathematics to trace that error back through every single weight — adjusting each one by a tiny amount in the direction that would have made the prediction better. RLHF just changes what "better" means.