Programming Fundamentals · Data Structures Inside the LLM

Vocabulary & Trie: How Tokenizers Cut Words

Remember that detail from LLM fundamentals—the tokenizer cuts “五花肉” into one whole token, not three characters. We said “common combos stay whole”; now the underbelly: how does a tokenizer spot, among tens of thousands of words, that “五花肉” should leave as one chunk? The answer is a way of organizing called a Trie (prefix tree)—hang the vocabulary by shared prefixes into a tree.

First, the organizing · how a vocabulary hangs as a tree

Suppose the vocabulary has: 五, 五月, 五花肉, 今天, 天, 天气, 吃, 花, 好. Hang them by first character, then second… shared prefixes share branches—“五月” and “五花肉” crowd the same “五” branch. A green ✓ means “a complete word ends here”; note “五→花” has no ✓—it’s only a waypoint (“五花” isn’t a word).

Pick a sentence:

Hit “Start tokenizing”—the cursor walks the tree character by character. Watch two things: at a ✓ it doesn’t cut yet (greedy—try going longer); when stuck it backs up to the nearest ✓ and cuts.

Tokens:

About 15 seconds, with narration each step

This walk is “greedy longest match”: go as deep as you can; when stuck, back up to the nearest word end and cut. Why greedy? Cutting “五花肉” as one chunk beats “五 / 花 / 肉” on tokens and meaning. Trie’s trick: each step only asks “does this node have a branch for this character?”—no need to rescan the whole vocabulary. Tens of thousands of words, one lookup per step. Again: organize well, look up fast.

Under the hood · real LLMs use BPE—same idea

🧩 BPE: repeatedly merge the “most co-occurring character pairs” into chunks

Real tokenizers (GPT and DeepSeek both use BPE) build vocabularies more wildly: shatter text into tiny fragments, count which two fragments sit next to each other most, glue them into the vocab; count again, glue again, tens of thousands of times. Common combos “grow” into big tokens—same idea as Trie’s “common words stored whole.”

的，五花肉 → high frequency, each a whole chunk

“饕餮” → 饕(frag1) 饕(frag2) 餮(frag1) 餮(frag2) → rare, shattered into byte fragments

So the phenomena you’ve seen make sense: “的” and “，” always cost one token; rare characters get split into several. That’s also why Chinese usually costs more tokens than English—most vocabularies train on English-heavy corpora, so English common words earn whole tokens while Chinese gets fewer, forcing more cuts. Same sentence, Chinese bill often higher—roots in who that “dictionary” vocabulary organized.

What this lesson wants to share

A vocabulary is a dictionary: it organizes all tokens; tokenization is “look up the dict and cut the sentence into chunks”
Trie organizes by shared prefixes: longest-match becomes one lookup per step—no backtracking the whole list
Greedy longest match: go as deep as you can; when stuck, back up to the nearest word end and cut
BPE shares the same idea: repeatedly merge the most co-occurring pairs into chunks—common stays cheap, rare gets shattered
Why Chinese costs more tokens: English-leaning vocabularies, fewer whole Chinese tokens, so more cuts