How your sentence becomes numbers: a trip through the tokenizer

The question I could not answer

I had been building on top of these models for a while when it occurred to me that I could not explain the most basic thing about them. Not how they reason, not how they are trained. Something simpler: how does a model understand a word?

Because it plainly does not, in any direct sense. I type letters. A neural network multiplies numbers. Somewhere between those two facts there is a translation, and I had been shipping systems for months without ever looking at it.

So I went and looked. This post is the trip: eight stops between the text you type and the numbers a transformer actually consumes. Everything below runs in your browser, no network calls, no pre-baked results. Press the buttons. The widgets are the post; the writing is just me pointing at things along the way.

First, build the machine

Here is the first surprise, and it reframes everything after it. The thing that cuts your text into tokens is not part of the model. It is a separate artifact, trained on its own pile of text, finished and frozen before the model's first training step.

Which means it does not exist yet on this page. Let us make one.

Three words first, because the widget uses all of them.

A token is one entry in the model's fixed vocabulary: a chunk of text it is able to recognise as a single thing. A symbol is whatever piece of a word we happen to be holding while that vocabulary is being built, starting out as single characters. And a merge is one rule, of the form wherever symbol A is immediately followed by symbol B, treat AB as a single symbol from now on. Every merge adds exactly one new entry to the vocabulary, and merges are numbered, because they have to be replayed in the same order later on.

Training a tokenizer means finding a good ordered list of merges. That is genuinely all it is.

That is a real byte pair encoder, learning a real merge table by counting character pairs and gluing the most frequent one together, over and over. Watch the vocabulary climb and the corpus token count fall. That gap is the entire value proposition of tokenization.

Scale it up and nothing about the procedure changes. Swap 1,100 words for a few trillion characters, run 200,000 merges instead of a few hundred, and you have GPT-4o's tokenizer: a table in which " the", "ing", and " function" are each one entry, and your surname almost certainly is not.

Watch your sentence get cut

Now feed it something. Whatever the merge table learned upstairs is what does the cutting here.

Four things I stopped and stared at.

Spaces belong to the word after them. " the" and "the" are different tokens with different IDs. A stray trailing space at the end of your prompt is not cosmetic. You have handed the model a token it rarely sees in that position.

Capitalisation changes everything. Hit the "capitals count" preset. "Tokenizers" and "tokenizers" share no tokens at all. Two strings that look nearly identical to you can be entirely disjoint sequences on the inside.

Rare words shatter. "the" is one token; "antidisestablishmentarianism" is a pile of fragments. How cheap a word is to write down is a direct measurement of how common it was in the training text.

Numbers are a mess. Change one digit of "1234567" and watch the boundaries jump around. The model is not seeing a number, it is seeing an arbitrary chunking of digits whose shape depends on the value. A good part of why arithmetic feels so inconsistent is sitting right there.

That was the first thing on this trip that changed how I write prompts.

A detour through bytes

The next surprise was one level below characters.

Modern tokenizers do not work on characters at all. They work on UTF-8 bytes, which is how they guarantee they can never meet something they cannot encode. That guarantee has a price, and it is not evenly distributed.

An ASCII character is one byte. Greek or Cyrillic, two. Sinhala, Devanagari, Japanese, most emoji: three or four. Then the second penalty lands on top of the first: these scripts are thin on the ground in tokenizer training corpora, so they earn few merges and fall back to per-byte tokens far more often. The costs multiply.

Which quietly explains something I had noticed and never chased down: the same meaning, written in a non-Latin script, costs more tokens. More money, a context window that fills faster, less room for retrieved documents. If you ship to a multilingual audience, measure tokens per language before you size anything.

Where the vocabulary comes from, in slow motion

At stop 00 a few hundred merges went by in about nine seconds, which is far too fast to see what actually happened. So here is the identical algorithm on a corpus small enough to check by hand.

The corpus is five made-up words, repeated. low five times, lower twice, new six times, newest six times, wider three times. Twenty-two words in total, in the style of the example BPE is usually taught with. The words were picked because they deliberately overlap: low hides inside lower, new inside newest, er sits at the end of both lower and wider. Those shared pieces are precisely what the algorithm has to find on its own, with nobody telling it that English has a suffix called -er.

Two things to know before you press anything. In the left panel, ×6 means that word appears six times in the corpus, which matters because a pair inside a common word is worth more than the same pair inside a rare one. And _ is a marker for the end of a word, so the standalone word low stays distinguishable from the low buried inside lower.

How to read it. Every word starts fully spelled out, one character per symbol: l o w _. The right panel counts every adjacent pair of symbols across the entire corpus, weighted by those frequencies, and ranks them. The top pair wins, gets glued into one new symbol everywhere it appears, and the counting starts again from scratch. That is one merge. Press it ten times.

Watch the two numbers at the bottom move in opposite directions. Vocabulary climbs by exactly one per merge, because each merge invents one new symbol. Tokens in corpus falls, because the corpus now needs fewer symbols to write itself down. Trading a slightly bigger dictionary for shorter text is the entire deal, and it is the same trade GPT-4o makes 200,000 times.

Notice the order it finds things in. The first four merges build new and low into single units, because those letters keep turning up side by side. Merges five through eight then assemble newest one character at a time, so by merge eight a six-letter word costs exactly one token. After ten merges, three of the five words are single symbols, and the pair queued up next is e+r, the ending shared by lower and wider. Nobody wrote a rule about suffixes. It is all counting.

Start with individual symbols. Count every adjacent pair. Glue the most frequent pair into a new symbol. Repeat. That is the whole of it:

for _ in range(num_merges):
    pairs = Counter()
    for word, freq in vocab.items():          # word: a tuple of symbols
        for pair in zip(word, word[1:]):      # every adjacent pair
            pairs[pair] += freq               # weighted by how often the word occurs
 
    best = pairs.most_common(1)[0][0]         # the most frequent pair
    vocab = apply_merge(vocab, best)          # glue it everywhere it appears
    merges.append(best)                       # the ordered list IS the tokenizer

So how does that table cut a word it has never seen?

By replaying the merges in the order they were learned. Take newer, which never appears in the corpus above. It starts fully spelled out, and each learned merge is tried in turn, lowest number first:

n e w e r _        the word, one symbol per character
ne w e r _         merge 1:  n + e
new e r _          merge 2:  ne + w
newe r _           merge 5:  new + e
                   no further merge applies

Three tokens: newe, r, _. Note that this is not the new + er split you would probably have drawn by hand. Merge 5 fires before the e+r merge exists, and once new and e have been welded together there is no going back. This is why real token boundaries so often look arbitrary: they are the residue of a fixed replay order, not a judgement about where a word divides.

But the word encoded. That is the property that matters, and it holds for anything you type: unseen words, names, typos, code, all of it falls back to smaller pieces and always lands somewhere.

The ordered list of merges is the tokenizer. Nobody wrote that vocabulary. It fell out of counting. (BPE is one of two dominant families; the other, SentencePiece with a unigram model, picks the segmentation that maximises likelihood instead of merging greedily. The results are similar enough that callers rarely notice.)

Numbers, at last

Halfway. We have integers, but an integer is just a name tag. ID 4,102 is not "twice" ID 2,051, and nothing about it says what the token means.

I expected the next step to be where the cleverness lived. It is the least clever thing in the entire transformer: a matrix of shape [vocab_size × d_model], and embedding a token means reading row number id. No computation. An array index.

For a 200,000-token vocabulary at 4,096 dimensions, that table alone is about 800 million parameters, spent entirely on "what does each token mean before any context arrives". Many models then reuse the same matrix, transposed, to turn the final hidden state back into next-token probabilities. The input dictionary and the output dictionary are one object, read in two directions.

The values start as noise. They become meaningful only because gradient descent spent trillions of tokens nudging them in whatever direction made the next prediction less wrong.

The map

Which raises the question I had actually come down here for: what do those numbers mean?

Not "what does dimension 12 stand for"; the answer to that is usually nothing. The meaning is in the geometry. Similar words end up pointing in similar directions, and the measure that matters is cosine similarity: the angle between two vectors, ignoring their lengths.

The map below uses toy vectors I wrote by hand in eight named dimensions, so the geometry is readable. Real embeddings have hundreds or thousands of dimensions and none of them have names. Every operation here is the one you would run on the real thing.

Click mouse. It sits stranded between the animals and the machines, because one word carries two meanings and a single static row has to average them. That averaging is a real limitation of the embedding table, and it is exactly what the attention layers exist to repair. By the time a token has climbed the stack, its representation has been reshaped by everything around it, so "mouse" among the cheese and "mouse" among the USB ports end up far apart. The embedding table is context-free. The model is not.

Walking from one word to another

Once meaning is direction, you can walk between meanings. Subtract the "male" direction from king, add "female", and you arrive next to queen.

Try to break it. sleep − run + jump returns nonsense, because no consistent direction connects those words. That is the honest state of the trick: analogies hold for relations the training data expressed consistently and thousands of times, and fall apart quietly everywhere else. The famous result is real, published analogy benchmarks land well short of perfect, and part of the effect comes from the convention (used here too) of excluding the input words from the answer.

The souvenir is not the party trick. It is that direction is the unit of meaning, the assumption every vector database, semantic search, and clustering job silently runs on.

The thing that was still missing

One stop left, and it is the one I did not see coming. Everything so far gives the same vector for dog no matter where dog appears, and attention has no built-in sense of sequence. Left alone, "dog bites man" and "man bites dog" reach the model as the same unordered bag of three vectors.

So position gets injected straight into the representation.

The original transformer used those fixed sinusoids, chosen so that relative offsets look the same anywhere in the sequence. Most current models use rotary embeddings (RoPE), which rotate the query and key vectors by an angle proportional to position instead of adding anything up front. Different mechanism, identical job: tell the model that this token came third.

What I carried home

Five habits came back up with me:

1. Count tokens with the real tokenizer, per language. Not characters ÷ 4. That rule of thumb holds for English prose and quietly fails on code, JSON, digits, and every non-Latin script, which is to say on production traffic.
2. Keep the stable prefix genuinely stable. Prompt caching matches exact token prefixes, and one changed character early on reshuffles every boundary after it.
3. Never hand a model character-level work. Counting letters, reversing strings, hard character limits: give it a tool or expect a guess. It does not have the characters.
4. Match the embedding model exactly between indexing and querying. Different model, different space, meaningless angles, and re-embedding the corpus is the only migration there is.
5. Treat formatting as a cost. Deeply indented JSON, long opaque IDs, heavy Unicode: reformatting the same information can cut a payload by a third.

Two translations sit in front of every model. Text becomes integers, by a frequency table discovered through counting. Integers become vectors, by a lookup into a matrix where direction carries meaning. Neither step is intelligent, and both were finished before your prompt existed.

So does a model understand a word? It receives an integer, reads a row of numbers that were shaped by every sentence it was ever trained on, and gets told where in the line that word was standing. Whether that adds up to understanding is a question for someone else. But it stopped being a mystery to me, and a surprising number of things I used to find strange about these models turned out to live down here rather than in the reasoning above it.