Sampling and reasoning
Skim HF LLM Course ch. 1. On a models page, note two models' context and output limits.
Let's build GPT, second half.
Keys, once you've clicked the player: Space play or pause · ← → 5 s · J L 10 s · C captions · < > speed · M mute · F full screen
Read the transcript
0:00Same prompt, different answer
Yesterday ended on a question that had no answer in it: the pilot flew on Friday, where? The model answered anyway. Tokyo, at forty-six percent. But forty-six percent isn't an answer. It's odds. So send that exact request twenty times, and watch what comes back.
Twenty identical requests, six different answers. Tokyo ten times, Oslo five, chef twice. And once each: Monday, Rome and Tuesday. Nothing in the model changed between those requests. Same weights, same prompt, same scores. The only difference was one random number per request, drawn after the model had finished its work. That last step has settings, and today you learn what they do. You also find out why turning the randomness all the way off still doesn't give you the same answer twice, and why many of the latest models won't let you touch these settings at all.
By the end of today, three things. You can say what temperature and top-p each change, in a line apiece. You can explain why temperature zero still isn't fully reproducible, and what to test instead of an exact string. And you can read a model's label: its context window, its maximum output, and the controls it accepts. Then you pick the model you'll develop on. Fast lane: if you can already say why a reasoning model rejects a temperature setting, and what a context window caps, skip to the last chapter and do today's task.
Here's where today sits. Everything up to yesterday happened inside the model: your prompt goes in, the layers do their reading, and out comes one score for every token in the vocabulary. Then a much simpler piece of code takes over. It turns the scores into odds, draws one token, appends it, and the whole loop runs again. That piece is the sampler. Temperature and top-p live here, and nowhere else. They reshape the odds once the model has finished. Keep this picture, because it settles a lot of confusion. No sampling setting makes the model know more. It only changes how the model's existing opinion gets turned into a token.
2:35Temperature
Start with what the model actually hands over. It isn't percentages. It's a raw score for every token in the vocabulary, and the name for those scores is logits. Tokyo scored four point seven three, and Oslo four point oh nine. On their own, those numbers don't tell you much. A small function called softmax turns scores into odds. It exponentiates each score, so bigger scores pull further ahead, and then divides by the total so everything adds up to a hundred percent. Here's the one property to hold on to: only the gaps between scores matter. Tokyo leads Oslo by zero point six four, and that gap alone is why Tokyo's odds are one point nine times Oslo's. Add ten to every score, and not one percentage would change. Stretch or squeeze the gaps, though, and all of them do. That's the whole idea behind temperature.
So predict. Divide every score by one half, which doubles each of them and doubles every gap. Tokyo has forty-six percent now: does it go up, go down, or stay put?
Up, to seventy-four percent. The gap doubled, so Tokyo's lead over Oslo grew from one point nine times to three point six times. And the number you divided by has a name: temperature.
Here's the dial, with the twenty requests from the start of the lesson. At temperature one, the scores are used exactly as the model gave them: six different answers. Turn it down to zero point five. Every score doubles, Tokyo climbs to seventy-four percent, and the same twenty random numbers now land on Tokyo sixteen times. Four different answers now, not six. Notice the long shots haven't gone. They've only become rarer. Now turn it up to two. Every score is halved, the gaps shrink, and Tokyo falls to twenty percent. Same random numbers, eleven different answers. And look at what's in there: sailed, judge, even the question word itself. Tokens the model rated as poor are now getting drawn. That gives you the working rule. Extraction, classification and tool arguments run at a low temperature, because you want the likely answer every time. Creative work can run higher, because there a surprise is the point.
What about temperature zero? You can't divide by zero, so APIs treat it as a special case: skip the draw, and take the highest score every time. Twenty requests, twenty Tokyos. You've seen this before. On Day three, the toy with the dice taken away wrote the same sentence on every run. It looks like the off switch for randomness. Hold on to that claim, because it doesn't survive the chapter after next.
Here's the mistake that follows. The answers vary too much, so you set temperature to zero, and now the output is the same every time. It feels reliable. Rewind to the prompt: the pilot flew on Friday. There's no place in that sentence. Tokyo, twenty times out of twenty, is a confident answer to a question that doesn't have one. Day three asked you to remember this. Temperature controls how often you get the model's top answer, and has no say in whether that answer is right. A wrong top answer at low temperature is the same wrong answer, every time.
6:42Top-p
Back at temperature one, look at the bottom row. Thirty-two more tokens, none of them above two percent. Together, though, they hold about eight and a half percent. That's roughly one request in twelve landing on something the model itself rated a long shot. It's where Monday, Rome and Tuesday came from. Top-p is built for exactly this. Walk down the list, keeping a running total of the odds, and stop as soon as the total reaches p. With p at zero point nine, that takes seven tokens. Everything below the line is dropped: all thirty-two long shots. The seven survivors are scaled back up to a hundred percent, so Tokyo goes from forty-six to fifty. Their shape, relative to each other, stays the same. Same twenty random numbers again. Monday, Rome and Tuesday can't come up any more. Those three requests now land on tokens inside the cut. Here's what a fixed cutoff can't do. Give the model a prompt it's sure about, where the sentence says Lima, and one token is already past ninety percent. Top-p keeps exactly one. Seven tokens when the model was unsure, one when it was sure. The cut follows the model's confidence.
So, side by side. Temperature reshapes: it stretches or squeezes every gap, and no token is ever ruled out. Top-p cuts: it removes the tail, and leaves the shape of what's left alone. And there's a third you'll meet, top-k. It's the fixed cutoff: keep the top k tokens, whether the model is sure or not.
The guide's advice is to tune one knob, not all three. Some APIs enforce that for you, and reject a request that sets temperature and top-p together. This is a good moment for the first part of today's task. Two lines in LEARNING.md: what temperature changes, and what top-p changes. You've watched both happen, so write what you saw.
9:04Zero isn't a promise
Back to Day three's toy for a minute, with the draw switched off. It always takes the candidate with the highest count. Here's what it writes, and you may recognize the ending. And here's the same prompt again. Same model, same counts, still no random draw. The first six tokens match. Then one run says It, the other says Retrieval, and from there they're two different texts. No randomness anywhere, and two different answers. Something else made that choice, and it's worth finding out what.
Here are the counts at that seventh token. After the word confidence and a full stop, the toy had seen eight different continuations, each exactly once. An eight-way tie. With no draw, something still has to pick, and code always has a rule. The first run took the candidate with the lowest token ID. For the second run, that rule was flipped by hand to take the highest. So yes, this demo is rigged. The real question is whether a real model ever gets close enough to a tie for something that small to decide.
Here's something that isn't rigged. Add zero point one, zero point two and zero point three in Python, and the result has a stray one, sixteen places down. Add the same three numbers in the other order, and it's gone. Floating-point addition rounds at every step, so the order of the additions changes the last digit.
A score is a sum of a great many of these. And your request shares a GPU with other traffic, so how it's batched can change which routine runs, and the order it adds in. So the same prompt, sent twice, can come back with scores that differ in their last digits.
Does a last digit ever matter? Here's a prompt from yesterday, given to the one-layer model that couldn't tell from and to apart. The pilot flew from Lima to Oslo: from where? Oslo scores six point oh six, Lima six point oh five. The gap is under a hundredth, so the odds are almost level, and at temperature zero Oslo wins by that sliver. To be straight about the sizes: this gap is still far bigger than rounding noise, so this exact case wouldn't flip. It shows you the shape of the problem, not the scale. But a long answer makes this kind of choice hundreds of times. It only takes one flip, because every token after it is written from a different text, exactly as the toy just showed you. Add one more fact: providers update the models behind a name. So read temperature zero as less random. Don't read it as reproducible.
Which brings you to the mistake this causes in real code. You set temperature to zero, copy today's answer into a test, and assert that the output matches it, character for character. Rewind. A near-tie flips, or the model behind the name is updated, and the test fails without anyone touching the code. That's a flaky test, and you already know what those cost a team. So test properties, not exact strings: does it parse, does it name a source, is the number right. Pin the model version and log the full request. And if a borderline case flips one run in ten, run it ten times and gate on the pass rate.
13:03Models that think first
A quick one, and no paper. Seventeen times twenty-four: what's the first digit of the answer? Say it straight away.
It's four: the answer is four hundred and eight. If you got it, you worked something out first. A model that writes left to right has to commit to that first digit before it has written any working at all.
That's a plain model's position. Your prompt goes in, and the first token of the answer comes straight out, with nowhere to work. A reasoning model gets scratch paper. Before the answer, it writes thinking tokens: working, drafts, checks. They're generated one at a time, the same way as everything else you've seen this week. Then it writes the answer, and attention does what you watched yesterday: the answer's tokens can read the working. The first digit no longer has to be a guess. You often don't get to read that working. Many APIs return only a summary of it, or nothing at all. It was still written, though, so it still took time, and it's still on the bill.
Start with the time. A plain request: half a second for the model to read the prompt, then a three-hundred-token answer at sixty tokens a second. The numbers are made up; the method is the point. The first word shows up after half a second, and the whole answer takes five and a half. Now the same answer from a model that thinks first: six hundred tokens of thinking, at the same speed. This one is playing at real speed, so watch the clock. Nothing has reached the screen yet. Whoever sent this request is looking at a spinner.
There's the first word, ten and a half seconds in. Six hundred tokens at sixty a second is ten seconds of writing that nobody sees. And you pay for it. Thinking tokens are billed as output tokens, usually the most expensive kind. Same visible answer, three times the output on the bill.
Try a bigger one, and it's the guide's own example. The answer you see is three hundred tokens, and before it the model wrote three thousand tokens of thinking. How many output tokens are you billed for?
Three thousand three hundred: eleven times what you can see. On a reasoning model, the length of the answer tells you very little about its cost. Tomorrow you start logging token counts on every call, and this is why.
So how do you steer a model like this? Thinking gets its own dial. You set a reasoning effort, from low to high, or a thinking budget in tokens, and it decides how much working the model does before it answers. And the old dials often stop working. Many reasoning models fix their sampling settings, or reject them outright. Reject means an error with status four hundred: the request itself is wrong, so retrying fails the same way every time. Report it in one clear line, and use the control the model does support. Tomorrow you trigger that error on purpose. As for why: on these models the provider has fixed the sampling itself, and the dial it hands you is effort. The docs usually state that without giving a reason, so don't invent a mechanism for your notes. Write what your model's docs say, and what follows from it: your code has to know which controls each model accepts.
So when is thinking worth buying? When the work has steps: math and logic, code changes, planning, and agent steps where one wrong move costs more than the extra tokens. And when usually not: classification, extraction, routing, short answers, and anything where the wait itself is the problem. There, low effort or a small model clears the bar at a fraction of the cost. The rule is to pick the lowest effort that holds quality. The guide's method is a sweep: run your own examples at low, medium and high, and stop where the answers stop getting better.
17:49Read the label
Time to pick a model, which means reading its label. Two numbers on it look as if they describe the same thing, and they don't. The first is the context window: say, two hundred thousand tokens. Suppose your prompt, with all its documents, comes to a hundred and fifty thousand. That leaves fifty thousand, because the window caps input plus output together, on every request. The second number is the maximum output: the most the model will write in one response. Here it's thirty-two thousand, and it's a separate, smaller cap. So the room for the answer is whichever runs out first. Fifty thousand left in the window, thirty-two thousand allowed: you get thirty-two thousand.
Your turn. Same model, but now the prompt is a hundred and eighty thousand tokens. How long can the answer be?
Twenty thousand tokens. The output cap says thirty-two, but the window only has twenty thousand left, and the window wins.
That's the pitfall the site warns about: reading the context window as room for input alone. Fill it with input, and the answer gets cut short. There's a second way to run out, even with a short prompt. On a reasoning model, thinking tokens are output, so they count against the output cap. Here the thinking took twenty-four thousand of the thirty-two, and left eight thousand for the answer. Tomorrow you set a cap of your own on each request. Too low, and the thinking can use all of it: the answer comes back cut off, or empty. So leave room for the answer, and room for any thinking.
Now do it for real. Open the models page linked on today's page, and pick two models: a small, fast one, and a larger one. For each, record the context window and the maximum output. Copy the numbers from the page, not from memory and not from a video, because they change with every release. Then the controls each one's docs say it supports: effort, extended thinking, temperature. That third line is the one that saves you from tomorrow's four-hundred error.
Last decision: which of the two to develop on. Guide zero's advice is the small, fast one. Evaluation runs get repeated again and again, and they're the biggest cost, so make them cheap. The starter repo already has a line for it in .env.example: LLM_MODEL. Check the ID against the models page, then set the model you chose in both .env and .env.example. From now on, your code reads the model from there and never hard-codes it. Changing models later is a one-line edit, with no hunting through files.
A quick check from the repo root: grep for LLM_MODEL in both files. Each one should show a single line, with the same model ID.
One reminder while you're in there. .env.example is committed, and .env never is. The model ID goes in both; your API key only ever goes in .env.
21:35Your turn
Your turn. Start with the course chapter linked on today's page. Skim it for two things: how models generate text one token at a time, and its section on limitations. Then close it, and write your two lines in LEARNING.md, if you haven't yet: what temperature changes, and what top-p changes. Add two notes under them. Why many reasoning models reject sampling settings and offer an effort control instead, and why even temperature zero isn't fully reproducible. Then the models page: two models, and for each one the context window, the maximum output, and the controls its docs say it supports. Choose the small one to develop on, and set its ID as LLM_MODEL in .env and .env.example. Commit LEARNING.md and .env.example. .env stays out of git, as always. And on the student track there's one extra: the second half of Let's build GPT, linked on the day's page.
On the map, it's LEARNING.md again, plus one line in .env.example, which sits at the root beside it. Tomorrow the warm-ups folder gets its first file of real code.
Done when, exactly as the site puts it. LEARNING.md lists two models with context window, maximum output and supported controls, and .env.example has an LLM_MODEL line for the model you'll develop on.
Self-check one. You lower temperature from one to zero point five. What happens to the odds: tokens below a cutoff are removed; every token ends up with the same odds; or the top token's lead over the others grows?
The top token's lead grows. Dividing by a half doubles every gap between scores, so the leader pulls ahead. Nothing is removed: removing is top-p's job.
Self-check two. At temperature zero, a test that compares the exact output string fails about one run in ten. Why: most models ignore a temperature of zero; the random seed changes on every call; or near-ties can flip between identical calls?
Near-ties can flip. Tiny numeric differences on the server can tip two almost-level scores the other way, and every token after that is different. Test properties, and gate on a pass rate.
Self-check three. The context window is two hundred thousand tokens, the maximum output is thirty-two thousand, and your input is a hundred and ninety thousand. How much room is left for output: ten thousand tokens, thirty-two thousand, or forty-two thousand?
Ten thousand. The window caps input plus output, so a hundred and ninety thousand in leaves ten thousand, and that's under the output cap. If the model thinks first, that room has to cover the thinking too.
That's Day five. Tomorrow is Day six, and you build: your first real call to a model, with stop reasons, token counts, a log row for every request, and that four-hundred error caused on purpose. This lesson was narrated with an AI voice.
How to do it
- 1Skim the course chapter for how models generate text one token at a time, and for its section on limitations. Write two lines on what temperature and top-p each change.
- 2Note in
LEARNING.mdwhy many reasoning models reject sampling settings and offer a reasoning-effort control instead, and why even temperature 0 isn't fully reproducible. - 3On the models page, pick a small, fast model and a larger one. Record each one's context window, maximum output tokens, and which controls (effort, extended thinking, temperature) its docs say it supports.
- 4Choose the small model to develop on, as Guide 00's cost advice suggests, and set its ID as
LLM_MODELin.envand.env.example. Your code will read it from there, never hard-code it.
LEARNING.md lists two models with context window, maximum output and supported controls, and .env.example has an LLM_MODEL= line for the model you'll develop on.
Reading the context window as room for input alone. It caps input plus output per request, and maximum output is a separate, smaller cap, so always leave room for the answer and any thinking.
Create a free account to mark steps done and keep your progress. Week 1 is free.
Start free