VectorMindX.

Latest Posts

Notes on building AI agents, LLM systems, and software that holds up in production.

A six-stage diagram: raw text, a tokenizer splitting it into pieces, token IDs, an embedding lookup, rows of vectors, and points scattered in a 3D vector space
FeaturedAILLMsTokenizationEmbeddings

How your sentence becomes numbers: a trip through the tokenizer

Eight stops between the words you type and the grid of numbers a model actually receives. Build the merge table yourself, cut a sentence apart, follow it down to raw bytes, and come out the other side in vector space. Every stop is playable in the page.

August 12, 202613 min read