Transformers
3Blue1Brown on transformers; write five sentences on attention in LEARNING.md.
Let's build GPT, first half.
Keys, once you've clicked the player: Space play or pause · ← → 5 s · J L 10 s · C captions · < > speed · M mute · F full screen
Read the transcript
0:00Twelve tokens back
Yesterday's toy model was handed the answer. Paris is the capital of France, said its prompt. The capital of France is: a database, and deletes when sources change. The answer sat twelve tokens back, and the toy could see two. Grounding, yesterday's first fix, only works if a model can use what's in its prompt. So something inside a real model has to reach back and pull out the one token that matters. That something is called attention. Today you watch a small model learn it from nothing, in under three seconds. Then you find out what it costs you on every request.
By the end of today, three things. You can say what a token's vector starts as, before it has read anything. You can explain what attention lets a token take from the other tokens, and why one pass of it isn't enough. And you can link all of it to a real request: what comes out at the end, and why cost and waiting time grow with every token you send. Fast lane: if you can already explain queries, keys and values out loud, and say why a long prompt delays the first token, skip to the last chapter and go straight to today's video.
Here's yesterday's toy as a pipeline. Take the last two tokens, look up their row of counts, turn the counts into odds. Everything earlier in the prompt is dropped before the model even starts. A transformer, the kind of model behind every API you'll call, keeps the last box and replaces the rest. Nothing is dropped. Every token in the context becomes a vector, which is just a list of numbers. Then comes the new step. In attention, tokens read other tokens, and each one takes in what it needs. That step repeats, layer after layer. Only then does the model produce odds for the next token. Same output as yesterday, and a very different way of getting there. Today's task is a twenty-seven-minute video on the full-size version, and then five sentences of your own. This lesson takes the same boxes on a model small enough to read every number, so the video's diagrams have somewhere to land.
2:37What a vector starts as
Day two ended with text cut into tokens. Here are three tokens from today's toy model. In this model every word is one token, and the whole vocabulary is thirty-nine of them. A token is an ID. Lima is eight, Oslo is nine, Friday is twenty. Don't read meaning into those: eight and nine are neighbors only because of the order the vocabulary was written in. The ID is a row number, in a table the model owns. One row per token, and in this toy each row holds twenty-four numbers. From here on, as far as the model is concerned, that row is Lima. It's called the embedding table, and a token's row is its embedding. The rows start out random. Training nudges them, the same nudging you watched yesterday, until they're useful for the model's task. Can you read them? Not really. No single number means city or weekday; whatever the model knows about Lima is spread across the whole row, in a code that only the rest of the model uses. One thing is missing. Lima gets the same row whether it's the second word or the tenth, and the step that comes next has no built-in sense of order. So the model adds position: here, a second row, looked up by where the token sits. This toy uses the simplest version, a second learned table. Most current models encode position in a different way, for the same reason. That sum is what the first of your five sentences has to say: a token's vector starts as its own learned row, plus where it sits. Notice what it doesn't hold yet. Lima knows nothing about a pilot or a question; every token is still alone.
4:53One token reads the rest
Here's the toy's whole world. A sentence says who went where, and when: the pilot flew to Lima on Friday. Then one more token asks a question: 'Where?' The model's job is yesterday's job, predict the next token, and here the right one is Lima. The prediction is read off the last token's vector. And the last token is the question. So look at the problem from that token's seat. Its vector says: a where-question, in ninth position. The answer is four tokens back, inside a different vector, and nothing connects the two yet. Attention is the connection. The question token gives every token it can see a share of its attention, and the shares add up to a hundred percent. Before training, with every number in the model still random, the shares come out nearly flat. And so does the guess. Its top pick is 'nurse', at just over four percent, with Monday and the word 'to' right behind. It's reading everything equally, which is the same as reading nothing. Now train it: three thousand small corrections, under three seconds on a laptop. Same sentence, same question. Watch where the attention goes. All of it lands on one token: a hundred percent on Lima. Now look at the panel underneath, because this is the part people skip. Attention doesn't just point. What Lima offers is added into the vector of the token that's reading. So the question's vector now holds something it didn't hold a moment ago. The prediction is read from that updated vector, and out comes Lima, at better than ninety-nine point nine percent. That's the second of your five sentences, in the model's own behavior. Attention lets a token take in information from other tokens, weighted by how much each one matters to it.
So here's a prediction for you. Keep the sentence, keep the trained model, and change only the last token: the question becomes 'When?'. Where does the attention go now?
To Friday. The sentence is identical, and so is every number in the model. The only thing that changed is which token is doing the reading.
Here's the trained model as you left it, with the where-question reading Lima. Now swap the last token. All of the attention swings to Friday, and Friday comes out. Nobody told this model that Friday is a day. It worked out which tokens answer a when-question from examples alone. Ask who, and it swings again, to the pilot. One sentence, three readers, three different places to look. The sentence doesn't decide where the attention goes; the token doing the reading does. Maybe it only learned that the place is always the fifth word. So move the words around: on Friday the pilot flew to Lima. The place is seventh now, and the model still finds it. And it isn't reciting its training data. One in ten of the possible sentences was held back from training, these two included: eight hundred and eight sentences it never saw. It answers all three questions about every one of them.
Today's video has names for three parts of this, so put them on the picture now. What the reading token is looking for is its query. For this token, the query amounts to: is there a place back there? Every token also puts out a description of what it holds. That's its key. A query is scored against every key in sight, and the attention shares are just those match scores, squeezed so they add up to a hundred percent. And what a token hands over once it's picked is its value. All three are computed from the token's vector, using tables of numbers the model learned. Nobody wrote a rule that says: look for a place. If you've used a hash map, you already own most of this. A hash map takes one key, demands an exact match, and returns one value. Attention scores every key for how well it matches, and returns a blend of all the values.
9:47Why models have layers
Now give it something harder. The pilot flew from Lima to Oslo on Friday. Two places, and a new question token: 'From?' The right answer is Lima. Same design as before, one pass of attention, trained for twice as long. And it does find the places: a third of its attention on Lima, a third on Oslo. Then it can't choose. Oslo, forty-nine point six percent; Lima, forty-nine point three: wrong, by a hair. On held-out sentences it gets this question right forty-eight percent of the time, which is a coin flip. Why? Look at what it has to work with: Lima's vector says a place, in fifth position, and Oslo's says a place, in seventh. The one fact that settles it, which of them comes right after the word 'from', isn't in either vector. And position can't stand in for it, because the training sentences put the two places in both orders. The word 'from' is sitting right there, one token to the left. But Lima never read it.
So let the tokens read first. This second model runs attention twice, in two layers, and in the first layer every token reads, not just the last one. Watch Lima. Seventy-eight percent of its attention goes to the token just before it: 'from'. So after this layer, Lima's vector no longer says just 'a place'. It says 'a place, with from mixed in'. Notice the dark half of the sentence, too. Lima can read itself and what came before it, never what comes after. That one rule is what lets a model write left to right: a token's vector never depends on text that doesn't exist yet. Oslo does the same with its own neighbor: eighty-eight percent on 'to'.
Now the second layer, where the question token reads. But it isn't reading the original tokens anymore. It's reading the updated ones: Lima carrying 'from', and Oslo carrying 'to'. This time there's something to match on. Nearly all of its attention goes to Lima, and Lima comes out. No coin flip. Change the question to 'To?', and it flips to Oslo. Change it back to 'From?', and the attention returns to Lima. Now turn the sentence around: to Oslo, from Lima. The attention follows the word, not the position. On the held-out test this model answers every question correctly, including the two where the one-layer model flipped a coin. Same data, same number of training steps. The difference is that the tokens could read each other before the question read them.
That's what depth buys. Each layer lets every token rewrite itself using the others, and the next layer reads the rewritten versions. The toy needed two layers for a two-step question; the models you'll call stack dozens. Real models also run several attention steps side by side inside each layer. They're called heads, and each can track something different. One might follow which word came just before, the way the first layer did here. And every layer has a second part, a feed-forward network, which the toy doesn't have. For today, just know where it sits: attention moves information between tokens, and feed-forward works on each token alone. None of that changes the outline. Vectors in, layers of tokens reading tokens, and at the top, one vector that has to predict the next token.
13:59What comes out, and what it costs
Back to the first model for one last experiment. Take the place out of the sentence, the pilot flew on Friday, and ask where. There's no answer in there to find. It looks, and nothing matches. The attention scatters: nearly half on the period, a fifth on the word 'the', a fifth on the question itself. And then it answers anyway. Tokyo, at forty-six percent, then Oslo at twenty-four. Neither city is anywhere in the sentence. Here's why. The model's last step is always the same: score every token in the vocabulary, all thirty-nine, and turn the scores into odds that add up to a hundred percent. 'Not stated' isn't one of the thirty-nine, so some token has to win. That's yesterday's lesson, seen from the inside: no data, no error, and an answer anyway. The architecture always produces odds. Abstaining, saying that the source doesn't say, is a behavior a model has to learn or be asked for, and one you have to test. Those final scores are also where tomorrow starts: temperature and top-p are two ways of reshaping them before a token is drawn. For today, that's your fourth sentence. What the model outputs at the end is a score for every token in its vocabulary, turned into odds.
Now the bill. In each layer, every token reads itself and every token before it. Draw one cell per read for a prompt of eight tokens, and you get a triangle: thirty-six reads. Double the prompt to sixteen tokens, and the reads don't double. A hundred and thirty-six: nearly four times as many. Double it again: five hundred and twenty-eight. Twice the tokens, about four times the work, every time. That's what people mean when they say attention grows with the square of the length.
So predict the bill. Your prompt grows from a thousand tokens to ten thousand, and the reads inside each layer go from half a million to fifty million, a hundred times more. What happens to what you pay for input?
Ten times, not a hundred. Providers price input per token, in a straight line. The squared part is real, but it's their compute problem, not a line on your invoice.
Where you do feel it is the wait. Before the first token of the answer can appear, the model has to process every prompt token: the whole triangle. That phase is called prefill, and it's why a longer prompt means a longer wait. Then the answer starts. Does a new token mean redoing the triangle? No: earlier tokens never read ahead, so nothing in their rows can change, and the model keeps that work and adds one row. And one more row for each token after that. What it keeps is every earlier token's key and value, which is where the name comes from: the KV cache. If you've ever memoized a function, this is that. Prompt caching stretches the same idea across requests. When your next request starts with exactly the same tokens, the provider can reuse the stored work for that prefix: a faster first token, and a discount on those input tokens.
Which sets up the mistake to avoid. Context windows are large now, so it's tempting to paste in every document you have. It all fits, so what's the harm? Rewind to the attention shares. They always add up to a hundred percent, so every token you add is one more candidate for them. Guide two says it in one line: every irrelevant token competes for attention. So send what the answer needs, and not much else. Fewer tokens cost less and answer sooner, and quality can fall in a long context full of distractors, even when everything fits.
18:35Five sentences
Today's task is to write five sentences on attention, without looking. They're yours to write, so here's only what each one has to cover, and you've already watched all five happen. One: what a token's vector starts as, before it has read anything. Two and three: what attention lets a token take from the other tokens, and what that changes about its vector. Add why that helps predict the next token, and you've said more than most candidates do. Four: what the model outputs at the end. And five, the link to the API. The final scores become the odds that temperature and top-p reshape tomorrow, and every token in the context is processed on every request, so cost and latency grow with tokens.
One trap, and it's the one the site warns about. You watch the video, you pick up the vocabulary, and a sentence comes out like this: attention uses queries, keys and values to weigh the tokens. Rewind, and play the interviewer: so what does attention do? The sentence names three parts, and says nothing about what any of them is for. So describe the change. Say what attention does to a token's vector, and why that helps predict the next token. You watched both today: a question's vector took in Lima, and the odds went from noise to the right answer.
Your turn. Today's video is linked on the day's page: watch it once, straight through. Then rewatch the attention section, and pause at each diagram until you can say what it shows. At each diagram, ask what you asked the toy today: which token is reading, which tokens it can see, and what changes in its vector afterwards. Then close the video, open LEARNING.md, and write your five sentences without looking. Make the fifth one the link to the API. Read them aloud, as if to a colleague. Rewrite any you couldn't defend in an interview, then commit. And on the student track there's one extra today: the first half of Let's build GPT, also linked on the day's page.
On the map, it's LEARNING.md once more: five sentences under Week one, committed.
Done when, exactly as the site puts it. LEARNING.md has five committed sentences on attention in your own words, and you can say them aloud without notes in under two minutes. Time yourself once. It's a short test, and it's the same one an interviewer gives you.
Self-check one. In a model that writes left to right, which tokens can a token read: every token in the context, itself and the tokens before it, or only the token just before it?
Itself and the tokens before it. Nothing reads ahead, and that's what lets the model write one token at a time.
Self-check two. You make a prompt ten times longer. What changes on the API: input cost times a hundred, because attention grows with the square; nothing, as long as it fits the context window; or input cost times ten, and the first token takes longer?
Input cost goes up ten times, and the first token takes longer. Pricing is linear per token, and prefill has to process every input token before the first output token.
Self-check three. Which sentence about attention would hold up in an interview: a token's vector takes in what it needs from earlier ones; it finds the one token that matches and returns its value; or it lets the model focus on the most important words?
The first one: it says what attention changes, a token's vector. The second describes a hash map, and attention returns a blend, not one value. The third never says what changes.
That's Day four. Tomorrow is Day five, sampling and reasoning: what temperature and top-p do to the scores you saw today, and why many reasoning models won't let you set them. This lesson was narrated with an AI voice.
How to do it
- 1Watch the video once straight through, then rewatch the attention section and pause at each diagram until you can say what it shows.
- 2Write five sentences on attention in
LEARNING.mdwithout looking: what a token's vector starts as, what attention lets it take from the other tokens, and what the model outputs at the end. - 3Link it to the API: the final scores become the next-token odds that temperature and top-p reshape on Day 5, and every token in the context is processed on every request, so cost and latency grow with tokens.
- 4Read the five sentences aloud as if to a colleague. Rewrite any you couldn't defend in an interview, then commit.
LEARNING.md has five committed sentences on attention in your own words, and you can say them aloud without notes in under two minutes.
Stopping at vocabulary. A sentence that only names queries, keys and values explains nothing; say what attention changes about a token's vector and why that helps predict the next token.
Create a free account to mark steps done and keep your progress. Week 1 is free.
Start free