Tagged: Tokenization

How your sentence becomes numbers: a trip through the tokenizer

Eight stops between the words you type and the grid of numbers a model actually receives. Build the merge table yourself, cut a sentence apart, follow it down to raw bytes, and come out the other side in vector space. Every stop is playable in the page.