Tokens
Paste code, English and another language into a tokenizer.
Count tokens for one record as JSON, YAML and prose.
Keys, once you've clicked the player: Space play or pause · ← → 5 s · J L 10 s · C captions · < > speed · M mute · F full screen
Read the transcript
0:00What the model actually reads
Here's a question a seven-year-old gets right: how many r's are in strawberry? For a long time, some very capable language models answered two. They weren't being careless. Look at what a model actually receives: your sentence, cut into eight pieces. And each piece arrives as a number. Strawberry is this one: a single number, with no letters inside it to count. The model has to recall how the word is spelled, because it can't look. So if a model never sees your text, what does it see, and what does that do to your bill, your limits and your bugs? By the end of today you'll have measured it yourself, in a small table in your learning log.
By the end of today, three things. You can explain what a token is, and why models read tokens instead of text. You've measured what the same content costs as English, as code, and in a second language, with your own numbers. And you can say why exact strings like IDs break, and why a long chat costs more with every turn. Fast lane: if you can already explain why a ten-turn chat costs more than ten separate questions, and you've measured characters per token for your own language, skip to the last chapter and fill in the table.
Before the real answer, design it yourself for a moment. A model is arithmetic; it needs numbers, so you have to cut text into pieces and give each piece a number. What should a piece be? The obvious cut is words. But then your list needs an entry for every word in every language, every name, every typo. Type something that isn't on the list, and the model has nothing to read. So go the other way: single characters. Now the list is tiny and nothing is ever missing, but this one short sentence is already thirty-seven pieces. Every piece the model reads costs you time and money. Real tokenizers land in between. Common words stay whole, rare ones get built out of smaller parts, and this sentence comes to nine tokens. Nine tokens, for six words. Keystrokes is the rare one, so it arrives in three parts. The full stop is a token of its own. Now look where the spaces went: each one rides at the front of the next word. So a token isn't a word; it's a piece that was worth having on the list.
2:50Where tokens come from
So who decides which pieces make the list? Nobody. The real one has about two hundred thousand entries, and it was learned by counting; you can watch the whole idea on five words. Start with single letters, and count neighbours. B followed by E turns up in all five words, and nothing beats that. So glue them: B E is now one piece, and it goes on the list. Count again, with the new piece in play. B E followed by R: five times. Glue those too. Do it twice more: add the second R, then the Y. And there it is: berry, as a single piece, in all five words. Nobody told the algorithm that berry means anything; it was just frequent. Keep going, and it picks up rarer pairs: R and A, three times. Run this a couple of hundred thousand times over a huge pile of text, and you get a real vocabulary. This is byte-pair encoding. Byte, because real tokenizers start from raw bytes rather than letters, so any text at all can be spelled from pieces. The upshot: strings that were common in the training text earn a token of their own, and everything else is assembled from fragments.
That was a toy; here's the real tokenizer on the same fruit. Blueberry splits into blue and berry, and cranberry into cran and berry. It found the same piece the toy did. And strawberry? Three pieces: S T, raw, berry. That's not syllables and it's not meaning; it's just which fragments were frequent enough to make the list. Now put a space in front, the way the word appears in the middle of a sentence. One token. To you it's the same word; to the model it's a different input.
It goes further than that. Take a word that's already a single token: Paris, with a capital P. One token, one ID. Add a space in front. It's still one token, but look at the number underneath: a completely different ID. Lower case: two tokens, par and is. All capitals: two tokens again, and different ones. Four ways to write one city, and the model gets four different inputs. So casing and stray spaces change what the model receives. The same word isn't always the same tokens, and tokens are all the model ever gets.
Fine, text becomes pieces. Why would you care about the exact count? Because the token is the unit for three things you'll answer for as an engineer. Price. You're billed per token: for everything you send in, and everything the model writes back. Limits. A model's context window, the most it can read and write in one request, is measured in tokens; not characters, not pages. And speed. Output arrives one token at a time, which is why speed is quoted in tokens per second: more tokens out, longer wait. So think of tokens as your bytes, which raises a practical question: how many tokens is a piece of text? Time to measure.
6:38Same content, different bill
Here's a paragraph about what Sourcebook does, in plain English: three hundred and two characters. The usual rule of thumb says four characters per token, so expect about seventy-five tokens. Hold that number. Run it through the tokenizer, and it's fifty-seven. That's about a quarter fewer than the rule predicted. So this paragraph gets five point three characters per token. The rule of four is a rough average, and for prose it errs on the safe side: clean English in common words does better. And you can see why. Every colour block here is a whole word with its space attached, or a punctuation mark. The only word that got cut is the invented one: Sourcebook, into source and book.
Now predict. Here's the same paragraph translated into Turkish, and it's shorter: two hundred and eighty-eight characters against three hundred and two. How many tokens: forty-five, fifty-seven, or seventy-seven?
Seventy-seven. Fewer characters, and thirty-five percent more tokens. If you guessed lower, good: that's the instinct to retire today, because characters don't tell you the cost.
Here's what happened. In the English, a colour block was a word. Here the colour changes every three or four letters, because more than half the words arrive in pieces. Take these two words. Together they mean your company's documents, and each one is a stem with endings stacked on it. The tokenizer cuts both into three: six tokens, where the English took three. So Turkish gets three point seven characters per token, against five point three for English. Same content, thirty-five percent more tokens to pay for, and you hit the context window that much sooner. And nothing about Turkish is harder. The tokenizer learned its pieces from text where English dominates, so English words earned whole tokens, and Turkish words get assembled from fragments. Remember the berries: frequent wins.
Now code. This is a real function from the pre-check script in your repo: eleven lines, five hundred and seventy characters. Your task asks for about twenty lines; this one's shorter so it fits on screen. A hundred and sixty-nine tokens. Look how fine the colours get: brackets, quote marks and colons keep cutting the line into small pieces, and ten of these tokens are nothing but indentation. Zoom in on one argument, the string rev-parse. Quote, rev, hyphen, parse, quote and comma: five tokens for one short string. And the function's own name: files, underscore at, underscore tag. Three tokens. Names you invent are rare by definition, so they never earn a token of their own. Three point four characters per token. Character for character, this code costs about one and a half times what the English prose did.
Put the languages side by side: one paragraph, one tokenizer. English is the baseline, at fifty-seven tokens. Spanish comes to sixty-six, sixteen percent more than English. German comes to seventy-five, about a third more. And Turkish, at seventy-seven. If your product answers in German or Turkish, every request costs about a third more, and the same context window holds about a quarter less. Careful what you take from this, though. It's one paragraph and one tokenizer, so read it as a direction, not a constant. That's why today's task has you measure your own.
Put those three measurements together and you have the table today's task asks for: characters, tokens, and characters per token, for English, for code, and for your second language. The last column is the one to remember. It's your exchange rate, from text you can see to tokens you pay for. On screen it's five point three, three point four and three point seven; yours will differ, and yours is the one to trust.
One more measurement; on the student track, it's your extra for today. Take one small record: an ID, a title, an owner, a date, an access list and a page count. As pretty-printed JSON, it's sixty-two tokens: about two point six characters each.
Now write the same record three more ways, starting from those sixty-two tokens. Strip the whitespace, same data: thirty-eight. Those line breaks and indents were twenty-four tokens of nothing. As YAML, forty. As a plain sentence, thirty-six. Structured output pays for every key, quote and brace, so whenever you ask a model for JSON, you're paying for the punctuation too.
12:23Strings that break models
So common text is cheap, which raises a problem: much of what engineers care about isn't common text. Take a UUID, the kind of ID your documents and users carry. Thirty-six characters; guess the tokens. Thirty-four. Almost one token per character, because random hex has no frequent pairs to merge. And when a model writes one back to you, it has to get all thirty-four right, in order, with nothing to check against. Here's a twenty-digit number: seven tokens, cut three digits at a time from the left. But place value counts from the right, so these chunks don't even line up with the thousands. That's what the model is doing arithmetic on. And a short error code, E-404, comes to three tokens: a letter, a hyphen and a number. No single piece of it is the code. Keep that one in mind. In Week three you'll watch vector search miss exact identifiers like this, which is why Sourcebook runs a keyword search alongside it. Exact strings stay fragile all the way up the stack.
Now you can explain the strawberry. In a sentence, the word arrives as one token, with one ID. Counting the r's in that is like counting the r's in a customer number. The letters aren't in what the model was handed. Now spell it out, with a space between the letters. Ten tokens, one per letter. Look at the IDs underneath. This r is token four hundred and twenty-eight. And so are these two. Three identical numbers, sitting right there in the input, so now there's something to count. That's why asking a model to spell a word first makes it better at counting letters.
Here's the mistake all this leads to, and nearly everyone makes it once. You hand the model something exact: add up these invoice totals. A fluent, confident answer comes back, one digit is wrong, and nothing flags it. Rewind. Before you send a task like that, split it in two: which part is language, and which part has to be exact? Let the model do the language: read the messy invoice and pull out the values. Then let code do the sum. Code is exact every time, and it doesn't bill you per token. For letters, have the model spell the word out first, or give it a tool. The rule of thumb to keep: models for language, code for exactness.
15:23Why long chats cost more
Tokens explain one more thing, and it's the one that surprises people on their first bill. A chat feels like talking to someone who remembers: you say something, it answers, you follow up. But the model keeps nothing between requests; like plain HTTP with no session, every request stands alone. So how does it know what you said a minute ago? Your client sends it again. Every earlier message and every earlier reply goes back in with each new request, and the model reads the whole transcript from the top, every time. The memory is in your payload.
So predict. Say every exchange, your message plus the reply, averages two hundred tokens. How much old history does the tenth request carry: two hundred, eighteen hundred, or two thousand?
Eighteen hundred. Nine earlier exchanges at two hundred tokens each, all sent again before the model reads a single new word from you.
Watch it stack up. Each request carries everything before it, so the history climbs in a straight line: two hundred, four hundred, six hundred, all the way to eighteen hundred by request ten. Now add the column up. Across ten requests you sent nine thousand tokens of history, for a conversation with two thousand tokens of actual content in it. You paid for the old words four and a half times over. And it accelerates. The cost of each turn grows with the length of the chat, so the cost of the whole chat grows roughly with its square: double the turns, and you send about four times the history. Latency goes the same way, because the model processes every token on every request. And all of it counts against the context window, which caps input plus output; that's Day five. It's why real products trim or summarise old turns. You'll see this in your own numbers on Day seven, when you send a three-turn conversation and log the input tokens of each request. For today, write two sentences in your learning log on why long chats cost more per turn, in your own words.
17:56Measure your own
Your turn. Pause here, and open the tokenizer linked on today's page. Paste an English paragraph, then a Python function of about twenty lines, then the same paragraph in another language you read. Write down characters and tokens for each. Then paste the awkward ones: a UUID, a long number, E-404, and the word strawberry. Watch how each one splits, and come back when you have your numbers.
Got your numbers? One warning before you trust them, because this mistake makes budgets wrong. Every number in this lesson came from one tokenizer: here's the Turkish paragraph again at seventy-seven tokens, with the English, fifty-seven, underneath. Now the same two paragraphs through two older tokenizers. The Turkish goes to a hundred and eight, then a hundred and thirty-two: the same text, and about seventy percent more tokens. The English barely moves: fifty-seven, fifty-nine, fifty-nine. So the gap between languages isn't a fact about the languages; it's a fact about one tokenizer's vocabulary. And tokenizers differ between model families. The one linked on today's page belongs to one provider, so use it to build intuition. Budget from your own provider's counts: the usage field you start logging on Day six.
Where does all this go? LEARNING.md told you on Day one: notes the days ask for, such as the Day two token table, go under that week's three lines. So under Week one, add your table and your two sentences on long chats, then commit. A measurement that isn't in the repo didn't happen.
Back to the map. Nothing new lit up today, and that's fine: today was a measurement, and it lives in LEARNING.md. But it sets up the first code you write. On Day six, your client goes into llm and warmups, and it logs input and output tokens for every call. Now you know what those numbers mean.
Done when, exactly as the site puts it. LEARNING.md has a small table of characters, tokens and characters per token, for English, code and your second language. Plus two sentences on why long chats cost more per turn. If both are in your repo, Day two is done.
Self-check one. Using today's measurements, about how many tokens is six thousand characters of Python: about eleven hundred, about fifteen hundred, or about eighteen hundred?
About eighteen hundred: six thousand divided by three point four. The English rate gives you eleven hundred and the rule of four gives fifteen hundred, and both would leave your budget short.
Self-check two. Why does a model miscount the r's in strawberry? It receives token IDs and not letters, the tokenizer drops repeated letters, or it only reads the first token of a word?
It receives token IDs, not letters. The letters aren't in the input, so the model has to recall the spelling instead of reading it; spelling the word out first puts them there.
Self-check three. Each exchange averages two hundred tokens. How much history does request six re-send: two hundred tokens, one thousand, or twelve hundred?
One thousand. Five earlier exchanges at two hundred tokens each; the sixth request's own message comes on top of that.
That's Day two. Tomorrow is Day three, how models are built: where the training text comes from, what a model predicts at each step, and why it can be confidently wrong. This lesson was narrated with an AI voice.
How to do it
- 1Take one English paragraph, a 20-line Python function, and the same paragraph in another language you read. Paste each into the tokenizer and record its characters and tokens.
- 2Work out characters per token for each. English lands near four; code and many other languages need more tokens for the same content, so they cost more and fill the context window sooner.
- 3Paste strings that cause bugs: a UUID, a long number, an error code such as
E-404, and the word strawberry. Note how each one splits, and why counting letters or doing arithmetic is unreliable for a model. - 4Do the chat arithmetic: if every exchange (your message plus the reply) averages 200 tokens, the tenth request re-sends about 1,800 tokens of history. Write two sentences in
LEARNING.mdon why long chats cost more per turn.
LEARNING.md has a small table of characters, tokens and characters per token for English, code and your second language, plus two sentences on why long chats cost more per turn.
Treating one tokenizer's counts as universal. Tokenizers differ between model families, so budget from your own provider's counts: the usage field you start logging on Day 6.
Create a free account to mark steps done and keep your progress. Week 1 is free.
Start free