How Tokens Are Counted: BPE and the Hidden “Token Tax”

Early NLP swung between two poles. Word-level tokenization is semantically clear, but English's hundreds of thousands of words plus morphology (look / looks / looking / looked) explode the vocabulary—and still can't handle unseen words (the OOV problem). Character-level tokenization can split anything, but sequences get too long, each character carries little information, and inference is painfully inefficient.

Modern LLMs took the middle road—subword tokenization: keep common words as whole Tokens, split rare ones into meaningful pieces. Vocabularies stay between 32k and 200k—no explosion, yet everything is representable.

A Token isn't simply a character or a word—it's a “subword” built from statistical patterns. By balancing vocabulary size and semantic expression, BPE became the foundation of modern LLM training and inference. (Figure: from the author's original share deck)

GPT-series models and Llama 3 all use BPE (Byte-Pair Encoding). The vocabulary isn't hand-crafted—it's “merged” out of the corpus:

At inference, look it up in reverse: a common word like "hug" is one Token; a rarer "bug" splits into ["b", "ug"]. The larger the vocabulary and the more “common” your text, the fewer Tokens—and the thinner the bill. That's why after GPT-4 moved to a 100k vocabulary, it specifically merged multi-level indent space sequences into single Tokens—code-generation efficiency jumped several times over.

Left: during BPE training, iteratively merge the highest-frequency byte pairs to build the vocabulary. Right: at inference, pre-tokenize by regex, then greedily merge by merge priority. (Figure: from the author's original share deck)

Mainstream model tokenizers train mostly on English, so BPE merge rules lean hard toward English vocabulary. Result: non-Latin scripts (Chinese, Japanese, Korean) face a severe efficiency disadvantage at tokenization—the industry calls it the “Token tax.”

For the same meaning, Chinese users burn more than 2× the Tokens of English users: they pay more API money and effectively shrink the context window.

BPE favors English: for the same meaning, Chinese is sliced finer, and the context window effectively gets shorter. (Figure: from the author's original share deck)

Pick a sentence and compare how many Tokens the Chinese and English versions need (estimated at typical ratios). For exact splits, try OpenAI Tokenizer yourself.

English has a natural guardrail: before BPE merges, text is pre-tokenized on spaces, so merges stay inside words and Token boundaries mostly match linguistic intuition. Chinese has no spaces, so BPE must rely entirely on statistical co-occurrence for boundaries—if a span (even a nonsense ad slogan) appears millions of times in the corpus, BPE will merge the whole thing into one “indivisible minimal semantic unit.”

The most famous example is “给主人留下些什么吧” (“leave something for the host”): a high-frequency blog-era guestbook line that was forcibly merged into a standalone Token in GPT's vocabulary—the famous “glitch Token.” Not a joke; an inevitable product of BPE's statistics.

English uses space pre-tokenization—boundaries feel intuitive; Chinese uses co-occurrence frequency—high-frequency text gets forcibly merged into a single Token. (Figure: from the author's original share deck)

Source: Adapted from Part 1 “What Makes Up a Token” of the author's internal team share “AI Token Cost Engineering Strategies.” For BPE details see OpenAI Tokenizer and the open-source tiktoken library. To review the underlying principles of tokenization and vocabulary, revisit LLM Fundamentals · Vocabulary & Training.