Experiment
Write EXPERIMENTS.md, push, tag v0.1.
Chart tokens and latency against max_tokens and effort.
Keys, once you've clicked the player: Space play or pause · ← → 5 s · J L 10 s · C captions · < > speed · M mute · F full screen
Read the transcript
0:00A hint isn't a finding
Here's yesterday's run again: one question, asked three times. Look at the first status line and the last. Same question, same model, and the only thing that changed was the effort setting. Ninety-three tokens out at the default, and seventy-nine at low effort. That's about fifteen percent fewer, and it would make a tidy line in a README: low effort cuts output by fifteen percent. But you watched that file run three more times yesterday. So before anyone signs that sentence, put all four runs on one picture.
Each dot is one run, placed by how many tokens came out. Here's run one: ninety-three at the default, and seventy-nine at low effort. The gap between them is your fifteen percent. Now the default's other three runs. Nothing about that request changed, and it came back at eighty-four, then seventy-nine, then seventy-eight. So one request, sent four times, landed anywhere from seventy-eight to ninety-three. That's Day five: the model samples, so the same request can come back at a different length. Give that band a name: the spread. It's how far a number moves when nothing changes, and here it's fifteen tokens wide. And low effort's other three: eighty-four, eighty-four, seventy-nine. All four sit inside the spread. On these numbers, you can't tell low effort from changing nothing at all. So when does a difference count? How far does a number have to move, and how often, before you can say the setting did it?
That's today's job: turn a hint into a finding, with numbers you can defend.
By the end of today, three things. You can run a fair experiment: one prompt, one setting changed at a time, and something to compare it with. Every run is written down in EXPERIMENTS.md, with numbers copied from your log, not from memory. And you've proved that a fresh clone of your repo runs, and tagged it v0.1. That tag is what the Week one checkpoint looks for. Fast lane: if you already test from a fresh clone before you tag a release, skip to chapter eleven and do today's task.
It's a longer lesson again, in three parts. In the first six chapters you run the experiments properly, and learn to read what the numbers can and can't tell you. In the middle three, you write the results down, and then test whether your repo works anywhere other than your own machine. That test usually has a surprise in it. And in the last two, you tag the week's work, and then it's your turn. Keep a terminal open.
3:06One change at a time
Say you'd like shorter answers. You lower the effort, and while you're in there, you reword the prompt to say briefly. The next answer is shorter, so which change did it? Rewind. Two things changed and one number moved, so you can't pin the result on either. It's the same reason you don't change two things at once when you're hunting a bug. So run it once with nothing changed. That run is your baseline: the result every other run gets compared with. Then change one setting per run, and put everything else back.
Here's that idea as a file: experiments.py, in your warm-ups folder. Line seven is yesterday's question, and it never changes. Under it is a dictionary of runs, each with a name. Each entry holds only what's different from the baseline. The first is empty: nothing changed. The second is empty too, and that's on purpose, so hold that thought until the next chapter. Then the cap, max_tokens, set low and set high. Then a system prompt. That's a block of instructions that sits apart from the messages and applies to the whole conversation. On this API it's a top-level setting called system. And effort, low and high, through output_config, exactly as yesterday's third call set it. Seven runs, where the site asks for at least four. Each differs from the baseline by a single key, so when a number moves, you know what moved it.
The loop is short. For each run, print its name, then call ask with the question and that run's one change. Yesterday's two stars are what make this work: any setting passes straight through. Now look where the try sits: inside the loop. Yesterday you saw that a try block ends at the first failure, so around the whole loop, one rejected request would cost you every run after it. Inside, a bad request prints one line, and the loop moves on. And if your ask raises on a cut-off answer, which was one of yesterday's choices, catch that here too, and check that your client logs the call before it raises.
One thing before you run it. Here's yesterday's log: three rows, one for each call. Which of them was the low-effort run?
You can work it out, because you remember the order. In a month you won't, and with seven rows it's already a puzzle. The log records what came back, and nothing about what was asked for.
So the client gets one small change. log_call takes a third argument: settings. And line twenty-six writes it into the row, so every row carries what was asked for, right next to what came back.
Down in ask, line forty-one hands them over: the cap, plus whatever extra options arrived. The site doesn't ask for this, and it's two minutes of work. The rule behind it: record the conditions next to the result. A number you can't trace back to its settings is a number you can't use.
Pause here, and write yours: one prompt, at least four runs, the baseline twice, and the try inside the loop. Add the settings to your log row. Then come back, and let's run it.
7:02Baseline first
Run it as a module, from the root of your repo, the same way as yesterday. Here's the board for the next two chapters: seven runs, queued. Each row fills in from your log: the stop reason, tokens in, tokens out, and the seconds it took. These are real calls, made for this recording, and the badge says so. Your numbers will differ, and yours are the ones that go in your table. The baseline comes back first. Twenty tokens in, seventy-nine out, and it stopped by itself: end_turn. Next is the same request again, with nothing changed. So what do you expect: seventy-nine?
Eighty-three. Same question, same settings, four tokens more. It's what yesterday's four runs did: the model samples, so identical requests don't give identical answers. And that gap is the most useful number on the board. It's what changes when nothing changes. Any other run has to move by clearly more than this before you can credit the setting. To be straight about it, two runs make a thin yardstick. Yesterday's four runs of this same request spread over fifteen tokens, and these two show a gap of four. Keep that doubt: it comes back in chapter five. Now the cap, set to forty. It comes back with exactly forty tokens out, and a stop reason of max_tokens. The model didn't finish: it was cut off, like yesterday.
So here's a prediction. Raise the cap from four hundred to two thousand: five times the room. How many tokens come out: about eighty, about four hundred, or about two thousand?
About eighty. The cap is a ceiling, and the model was never near it.
Here it is on the board: eighty-two. That's inside the gap between your two baselines, so on this evidence, raising the cap changed nothing. Which tells you what max_tokens is. It doesn't ask for a longer answer, or a shorter one. It says where to cut the answer off, if it gets that far. And here's the mistake that follows. People lower the cap to get shorter answers, and what they get is cut-off answers. If you'd like brevity, ask for it in the prompt, and prompts are next week. So why set it at all? Because it's a guard. It bounds what one request can cost you, and how long it can take, whatever the model decides to write.
9:56What a setting costs
Three runs left, and the next is the system prompt: explain it to a ten-year-old. Will the answer come back shorter, or longer? Take a guess.
A hundred and twenty-three tokens out. That's forty above your higher baseline, when identical runs differed by four. So it looks like a clear result: explaining to a ten-year-old takes more words. And a second number moved, in a column you probably weren't watching: tokens in. Thirty-four, up from twenty. A system prompt is input. Those fourteen extra tokens are sent, and paid for, on every request that carries it. But don't write that down yet. A hundred and twenty-three is a count of output tokens, and yesterday showed you that output isn't always answer. So open two of these responses up.
This recording kept each response whole, so open the first baseline up. One block came back, of type text. Seventy-nine tokens of output, and every one of them is answer. Now the run with the system prompt. Two blocks this time, and the first one is thinking. It's yesterday's surprise again: nobody asked for it. So split the count. A hundred and twenty-three tokens of output: fifty-two of thinking, and seventy-one of answer. Now read that against the baseline. The answer itself got shorter: seventy-one tokens, where the unchanged request wrote seventy-nine and eighty-three. The count went up because thinking is billed as output. And the board would have had you write the opposite: a system prompt makes answers longer. One output count per row can't tell you what the count is made of. So before you write a finding about length, look at what came back. Print the answer, or log each block's type next to the count. Did it make the answer simpler? Read it: how much stuff an AI can keep in mind, like pages of a book. For this one answer it looks that way, and judging it across many answers takes evals, which arrive in Week five.
Back to the board, with two runs left. Effort low: eighty-five tokens out. That's two above your higher baseline, when identical runs differ by four. So you can't tell it from changing nothing. And effort high: eighty-three tokens out. That's inside the gap. So on this board, low effort wrote more than high effort. Would you report that?
You'd have no more behind it than yesterday's fifteen percent had. Both runs land within two tokens of your baselines, so on this board, effort changed nothing you can see. One more column before you leave this board: seconds. The cut-off run was the quickest, at zero point eight one seconds, and that's tempting to write down: fewer tokens, less time. But it's a fifth of a second under your faster baseline, and your two baselines differ by a quarter. So that's a hint, like the others. It goes in the table as a number, not as a claim.
Now look at the log, with a grep that prints only the settings part of each row. Seven rows, and each one says what it asked for. The last two are the effort runs, and this time you don't have to remember the order. That's the change from chapter two, paying for itself. So that's the first pass: seven runs, and a tidy story. The cap cuts, the system prompt costs input and thinking, and effort does nothing you can see. Which of those would you bet on, if you ran all seven again?
14:09Run it again
There's a cheap way to find out: run the file again. Before you do, add one more run from the site's list: temperature, where supported. It's one more entry. As yesterday, temperature goes through extra_body, because current versions of the SDK no longer take it as a keyword. If your model rejects it, the loop prints rejected, with the API's message, and moves on to the next run. That's the try from chapter two doing its job.
Run it again: the same command, and eight runs this time. Start at the bottom. Temperature: rejected, with a four hundred and the API's own message. It's the answer yesterday's warm-up got, and here it's one line among eight. Now the top two. Ninety, and ninety. The first pass gave you seventy-nine and eighty-three for that very request. So your yardstick just moved, and every other row has to be read again. First, though, a quick question about the log. Fifteen requests have gone out now: seven in the first pass, and eight in the second. So how many rows does the log hold?
Count the lines that have settings in them. Fourteen. A rejected request never produced a response, so the client had nothing to write. The log only knows about calls that came back. So that row, you write by hand, from what the terminal printed: the setting, the status code, and the message. A failed request is still a result.
Here are both passes on one picture. Each dot is a run, placed by its tokens out, and the first pass is on it already. The purple band is the gap between your two baselines: seventy-nine to eighty-three. Now the second pass, baseline first: ninety, and ninety. So the band has to stretch: seventy-nine to ninety. That's eleven tokens of spread, where the first pass showed you four. The yardstick was too short, which is what yesterday's four runs were telling you. Five settings are still to come in, and each has a first-pass story. Which of those stories do you expect to see again?
The cap at forty first. Forty again, and cut off again: two of two. That one came back. Its head start on the clock didn't. This run took one point oh six seconds, level with the two baselines beside it. So that hint from the last chapter stays a hint. The cap at two thousand: eighty. Inside the band both times, so that one came back too: a higher cap changed nothing. The system prompt: a hundred and nineteen. Over the band both times, by about thirty tokens. Effort low: eighty-two. And look at its first run, the one that sat two above your old yardstick. The wider band takes it in: inside both times. And effort high: a hundred and nine. Nineteen over the band, where its first run sat inside. Once in, once out. Now open the counts up, the way you did in the last chapter. The system prompt's replies had thinking in them both times: fifty-two tokens, then forty-six. The answers alone were seventy-one and seventy-three, both shorter than any baseline run. And effort high? Its long run had thirty-five tokens of thinking in it, around an answer of seventy-four. Its other run had none. So here's what the picture says. The count went over the band on exactly three runs, and they're the three where the model thought. Leave out the two that were cut, and every answer here is between seventy-one and ninety tokens long. Which stories came back? The cap cut both times, and the system prompt cost more both times, in and out. For effort, one run in four moved: a hint again, and you know what to do with a hint. So keep this one: a finding is a result that comes back when you run it again. And even then, this is one prompt and two runs per setting: observations, not rates.
19:01A longer list
On Day two you took something on trust: a chat re-sends its whole history with every request. Today you watch it happen in your own log. And yesterday left you a clue: a conversation is just a longer list. So the client grows a second function, chat. Where ask took one prompt, chat takes the list of messages, and line thirty-nine sends it as it is. Line forty-two adds one thing to the settings: how many messages went out. You'll want that in the log in a minute.
And ask doesn't go away. It becomes one line: a conversation that has a single message in it. Yesterday's warm-up and today's experiments run exactly as before.
Here's the script that uses it. Three questions, each one following from the last, and an empty list called messages. For each question, line thirteen adds it to the list, as the user's turn. Line fourteen sends the whole list. And line fifteen adds the model's answer to the same list, as the assistant's turn. Read that line twice, because it is the chat's entire memory. The API keeps nothing between requests. If the model seems to remember your first question, that's because your code just sent it again.
Before you run it, a prediction. The first request is the question you've sent all day: twenty tokens in, about eighty back. Each new question is one short sentence, so what does the third request take: about twenty, about sixty, or over a hundred and forty?
Over a hundred and forty. The questions are tiny. What grows is everything sent along with them.
Run it. Three answers, each with its status line. Read only the input counts. Twenty, then a hundred and twenty-two, then a hundred and ninety. Your questions never got longer. The requests did. Let's draw them to scale.
Request one: a single message, your first question. Twenty tokens. Request two carries three messages: the first question again, the model's answer to it, and the new question. A hundred and twenty-two tokens. Split it up. Twenty of those went out before, in request one. A hundred and two are new: the answer, and your second question. Request three: five messages, a hundred and ninety tokens. And a hundred and twenty-two of them are request two, sent all over again. Add up the input you were billed for. Twenty, plus a hundred and twenty-two, plus a hundred and ninety: three hundred and thirty-two tokens, for three short questions. And a hundred and forty-two of those tokens had already been sent once. That's over four in ten, after only three turns. Do the columns add up? Here they do: of those hundred and two new tokens, eighty-two are the answer the model wrote. That leaves twenty for your second question, which is what your first one took. They add up because no reply in this chat had thinking in it. When one does, they won't: thinking is billed as output, and this script sends back only the text. So don't expect your columns to add up exactly. When they don't, ask what was actually sent. The rule to keep: in a chat, input grows with every turn, even when your messages don't. And the mistake is costing a conversation by its latest message, when you pay for the whole list every time. It's also why real products trim or summarise old turns, as Day two said. You now have the log that shows when that starts to matter.
Pause here, and send your own three-turn conversation through your client, so it's logged. Yours can be built differently. Write down the three input counts, and compare them with your prediction.
23:36Write it down
Runs you don't write down are runs you'll repeat. So step three is a file at the root of your repo: EXPERIMENTS.md. And it starts with no results at all. It starts with the conditions: the prompt, word for word; the model and the date; what the baseline was; and the spread. That last line is your yardstick, written down: the baseline's four runs gave seventy-nine to ninety tokens out. Every finding below is read against it. And the model line carries a date for a reason. Models change under their names, so a result without a model and a date can't be checked later.
Then the table. One row per run, with the columns the site lists: the setting, the stop reason, tokens in, tokens out, the latency, and a finding in one line. Where do the numbers come from? From the log, every one. Copy them by hand, or write ten lines of Python to do it: what matters is that each number here has a row in calls.jsonl behind it. Now read the findings. A good one says how far something moved, and against what: fourteen more tokens in, and thirty-three over the spread. And look at row four: inside the spread. That's a finding too. Leave out the runs where nothing happened, and your table tells a rosier story than your log does.
The second pass gets its own table, and its heading is the question you asked of it: which results came back? And the findings answer it, row by row: again cut at forty, again inside the spread, again over it. And for effort high: this time, nineteen over. The last row is the one you wrote by hand. Temperature: rejected, a four hundred, and no row in the log.
The conversation gets its own small table, because step two asks you to record the input tokens of each request. Twenty, a hundred and twenty-two, a hundred and ninety, and one line under it with the total.
Here's the pitfall the site names for today: writing impressions. The site's example is answers got shorter. Yours, after today's runs, might be the one on screen: the system prompt made answers longer. Rewind, and ask it two questions: longer than what, and by how much, on how many runs? And you know from chapter four that it isn't even true: the answers got shorter. Now the version with the log behind it: a hundred and twenty-three and a hundred and nineteen tokens out, against seventy-nine to ninety at baseline, on two runs. And notice it says tokens out, not answers, because that's what the log measured. It's Day one's rule again: no claim without a number.
Add the warm-ups folder and the new file, and commit. And push. The push is marked example, because the address on screen stands in for yours. Three files changed. The experiments are written down and pushed, so everything's on GitHub now. Is it?
27:17A fresh clone
There's one way to find out, and it's step four. Make a scratch folder somewhere outside your repo, and clone into it from GitHub, the way a stranger would. Go in, and run uv sync. There's no .venv folder in a clone, so uv builds one, and it installs exactly the versions in uv.lock. That's why you committed that file on Day one. Now run yesterday's warm-up. No key is set yet, so it can't finish. The question is how far it gets. Not far. Module not found: no module named llm.client. And that's a strange one, because on your machine this exact command worked ten minutes ago. Read the name closely. It doesn't say there's no llm. The folder arrived, with the template's own file in it; it's client that's missing.
To see why, put the two folders side by side. On the left, your working folder, where everything runs. A clone copies what's been committed, and nothing else. So these files make the trip: the project file, the lockfile, the example env file, your experiments, your warm-ups. These don't, and they shouldn't. Your .env file, the virtual environment, the logs: git was told to ignore them, and it does. And then there's this one. client.py is in your folder, it isn't ignored, and it was never added. Every commit you made named other files. So two kinds of file stayed behind, and only one is a problem. What git ignores is either rebuilt, like the environment uv sync just made, or yours to supply, like the key. Nothing rebuilds a file you never committed. Here's the honest part: this recording's repo was left with that gap on purpose. Yesterday's commit added the warm-ups folder and the ignore file by name, and the llm folder was never on the list. Yours may pass first time. The test is still the only way to know, because your own machine can't tell you: there, every file is present.
Back in your working folder, git status says it in one line. Two question marks mean untracked: git can see the file, and has never been asked to keep it. Add the file by name, and commit it. One file changed, and it's the one that was missing. Then push, so that GitHub has the file too. Here's a habit to take away: before every commit, read git status, and treat every line it prints as a decision. Commit the file or ignore it, and never leave it undecided.
Over to the scratch clone again, and pull. One file arrives: client.py. Run it again. Another traceback, and this time it's good news. Read the last line: TypeError, could not resolve authentication method. It's yesterday's first error. The code is all here, and the key isn't, which is exactly right. Your .env file never left your machine. So set the one thing a stranger would have to set: export the key, in this terminal. Yours goes where the placeholder is. And run it once more. Three real replies, each with its stop reason and token counts, and then yesterday's rejected request, caught by name. A stranger's copy of your repo works. Now notice what you didn't set: LLM_MODEL. There's no .env file here, so the client fell back to the default in the code. That's yesterday's fallback, and it's why the site can say one environment variable. One more thing, since you now read these lines differently. The third reply, at low effort: seventy-one tokens. That's under your whole spread, for the first time in seven low-effort runs across two days. Is that the setting, at last, or a spread wider than four runs showed? You can't tell from one run. So it goes in the log like the rest: write the number down, and leave the claim until it comes back.
Here's where things stood: one file that never left your machine. Committed, pushed, and now it arrives. Both folders run. Keep the rule that sits under this: your working folder is not your repo. Your folder holds whatever you've ever made, and your repo holds only what you committed. A clone is the one test that can tell them apart. Then delete the scratch folder. Two copies of a project is how you end up editing the wrong one.
32:40What git remembers
One check is left in step four, and it's about the key. Here's a throwaway repo, with two commits in it. It made yesterday's mistake on purpose, with a made-up key: the first commit pasted the key into the code. The second took it out again, and reads it from the environment. Search the files for it: git grep. Nothing comes back: the key isn't in any file. So is it gone?
Now search the history. This is git log with the p flag, which prints every change ever committed, piped through grep. Two lines, newest first: the key coming out, with a minus, and under it the key going in, with a plus.
Here are those two commits, newest on top. A commit is a snapshot of the whole project, as it was at that moment. The older one has the key on line three, and the newer one doesn't. git grep reads your files as they are now, which means the latest snapshot. No match. The history search reads every snapshot. And there it is, in the first commit, where it stays. Because a new commit never changes an old one. It's added on top. And a clone copies every commit, so anyone who clones your public repo gets that key too.
Now run that search on your own repo, with the first few characters of your key. Only the first few, so the full key stays out of your shell history. The count is zero: no commit has ever contained it. Zero is the only acceptable answer. And one more: list the files git tracks, and filter for env. Only .env.example. That's the done-when's no .env anywhere. And if your count isn't zero? Then the order matters, and it's the order from Day one: revoke the key at your provider first. Then make a new one, and put it in .env only. Once a key is revoked, the copy in your history is just a string.
35:02v0.1
Last step. A stranger who lands on your repo needs three short documents, and each answers a different question. The README: what is this, and how do you run it? The changelog: what changed in this release? One entry per tag, and the starter's template adds a sharp instruction: say which number it moved. And the learning log, which you've been writing all week. Its three lines: what you built, one decision with its evidence, and one thing that broke.
The README's centre is a section called run it, and you already know what goes in it, because you just did it: clone, sync, set one variable, run. Under it, a line worth earning: tested from a fresh clone, with the date. Yours can say that, as of ten minutes ago. And the starter's own guide stays underneath for now. Its README says it's temporary until Day forty-three, so add your short section on top and leave the rest.
The changelog entry has three parts. Added: the client that logs every call, and the two scripts.
Measured: the numbers you'd stand behind. The template asks for each number's n, so the first line gives it: two runs each, and four of the unchanged request. And known issues: one prompt and two runs per setting, so these are observations, not rates. And nothing here measures answer quality yet. It can feel odd to list weaknesses in your own release notes. But a reader trusts the numbers above because you said what they don't show.
And the week's three lines in LEARNING.md. For one thing that broke, today handed you a good one: the fresh clone failed, and git status showed why. For the decision, pick one you made and say what backed it. Here it's the settings in each log row, and the evidence is yesterday's log, where nothing said which run was which.
Before the tag, the usual two: ruff, and the tests. Both pass. Then a third, from the starter: the precheck script, for v0.1, with a flag that makes it check your working tree: what you'd commit right now. It looks for the paths the website checks: a README, EXPERIMENTS.md, .env.example, and no .env. All four are fine. Those path checks are the floor. They'd pass with an empty EXPERIMENTS.md. The checkpoint's criteria are about what's inside, and that part is on you.
Now ship them. Add the three documents by name, commit, and push: the same three commands as before. Now the tag: git tag v0.1. It prints nothing, and that's normal. Run the precheck again, without the flag this time, so it reads the files inside the tag and not the ones in your folder. Same four lines. And push, as usual. Everything up to date? You just made a tag, and git says there's nothing to send.
That's the catch: a plain push sends commits, and it doesn't send tags. Until you push the tag itself, GitHub has never heard of v0.1, and neither has the checkpoint. So push the tags as well, with the command on screen. New tag: v0.1, and now it's public.
So what did you just make? Here's your week as git has it: seven commits, oldest on the left. The label called main sits on the newest. A tag is a second label, on the same commit. Nothing was copied, and nothing was frozen. It's a name for one snapshot. The difference shows up with your next commit. main moves forward to it. The tag stays where you put it. So v0.1 will always mean your repo exactly as it was when you called Week one done. That's what the checkpoint reads, and it's what you can point someone to in six weeks. A tag can be moved, but on a shared project that's an exception, because people rely on a tag never changing. On your own repo this week, the two commands on screen do it: one moves the tag, and one pushes the move.
40:09Your turn
Your turn, and it's the last build of the week. Run at least four experiments through your client, on one prompt, changing one setting at a time. Run your baseline twice, and then run the whole file a second time. Send one three-turn conversation, and note the input tokens of each request. Write EXPERIMENTS.md: one row per run, every number copied from your log. Then the fresh-clone test: a scratch folder, uv sync, only your key, and the warm-up. Search your history for the start of your key while you're there. Add the short README, the changelog entry and the week's learning log. Push, tag v0.1, and push the tag. On the student track, chart tokens and latency against max_tokens and effort. Two small charts, from your own log.
On the map, Week one is in place. The llm folder has a client that logs, the warm-ups folder has three scripts, and the root has a README, a learning log and your experiments.
Done when, exactly as the site puts it. GitHub shows tag v0.1 with README.md, .env.example and an EXPERIMENTS.md of four or more runs with numbers, no .env anywhere, and a fresh clone runs after uv sync plus one environment variable.
Self-check one. Two identical runs gave seventy-nine and eighty-three tokens out, and a changed setting gave eighty-five. What can you say: the setting makes answers a little longer; nothing yet, because identical runs already differ by more; or the setting has no effect on the output?
Nothing yet. Identical runs differ by four tokens, and this run sits two above the higher one, so the setting hasn't beaten the gap. It isn't proof of no effect either: a few runs can't show that.
Self-check two. In a chat, every question is ten tokens long, yet request twenty takes far more input than request two. Why: each request re-sends every earlier turn; the API stores the chat, and bills you for storage; or the model thinks harder late in a conversation?
Each request re-sends every earlier turn. The API keeps nothing between requests, so your code sends the whole list, every time.
Self-check three. You committed a key, then removed it in a new commit. Is it out of your repo: yes, because the latest commit doesn't contain it; yes, once the new commit is pushed; or no, because the old commit still holds it?
No. A new commit never changes an old one, and a clone copies them all. So revoke the key first, and then replace it.
That's Day seven, and that's Week one: a client, a log, experiments with numbers, and a tag. Next is the Week one checkpoint, where you tick four criteria against your repo, and explain four ideas without notes. This lesson was narrated with an AI voice.
How to do it
- 1Run at least four experiments through
llm/on the same prompt, changing one setting at a time:max_tokenslow and high, with and without a system prompt, effort low and high, andtemperaturewhere supported. - 2Send one three-turn conversation and record the input tokens of each request. They grow because every request re-sends the whole history.
- 3Write
EXPERIMENTS.md: one row per run with the setting, stop reason, input and output tokens, latency and a one-line finding. Copy the numbers from your JSONL log, not from memory. - 4Test a fresh clone: clone into a scratch folder, run
uv sync, set only your API key, and runuv run python -m warmups.hello_llm. Searchgit log -pfor your key's prefix to confirm it was never committed. - 5Add a short
README.md(what it is, how to run it), aCHANGELOG.mdentry and this week'sLEARNING.mdentry. Push, rungit tag v0.1andgit push --tags, then tick the Week 1 checkpoint.
GitHub shows tag v0.1 with README.md, .env.example and an EXPERIMENTS.md of four or more runs with numbers, no .env anywhere, and a fresh clone runs after uv sync plus one environment variable.
Writing impressions such as "answers got shorter". Every row needs the numbers from your log (tokens, latency, stop reason), because the rule is no claim without a number.
your first logged calls
Move the client and logger into llm/. Vary max_tokens, the system prompt, and effort and temperature where supported. Responses can hold several blocks (thinking, text), so keep only the text; at 40 tokens it may be cut off or empty. Try temperature only where supported: current Python SDKs no longer take it as a keyword, so pass extra_body={"temperature": 0.2}, and the latest Claude models return a 400 error for it, so catch anthropic.BadRequestError. On OpenAI, use responses.create(...) and resp.output_text.
import os
import anthropic
MODEL = os.getenv("LLM_MODEL", "claude-haiku-5-5") # set to a current model ID
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment
def ask(prompt: str, max_tokens: int = 400, **options) -> str:
resp = client.messages.create(
model=MODEL, max_tokens=max_tokens,
messages=[{"role": "user", "content": prompt}], **options)
u = resp.usage
print(f"[{resp.stop_reason}] in={u.input_tokens} out={u.output_tokens}")
return "".join(b.text for b in resp.content if b.type == "text")
q = "In two sentences, what is a context window?"
print(ask(q))
print(ask(q, max_tokens=40)) # watch stop_reason
print(ask(q, output_config={"effort": "low"})) # effort, where supportedCreate a free account to mark steps done and keep your progress. Week 1 is free.
Start free