Subword Tokenization — BPE, WordPiece, Unigram, SentencePiece

Your vocabulary has 50,000 words, and a user types "untokenizable". Your tokenizer returns [UNK], and the model now has no signal about the word. Worse: the 90th-percentile document in your corpus has 40 rare words, which means 40 bits of dropped information per document.

Subword tokenization fixes this. Common words stay single tokens, and rare words decompose into meaningful pieces: untokenizable → un, token, izable. Because any string is ultimately a sequence of bytes, the training data ends up covering everything.

Every frontier LLM in 2026 ships on one of three algorithms (BPE, Unigram, WordPiece), wrapped in one of three libraries (tiktoken, SentencePiece, HF Tokenizers). Ship a language model and you have to pick one.

Every algorithm below solves the same tradeoff: keep frequent words as single tokens, and break the long tail into pieces small enough that nothing ever falls out of vocabulary. In code, whatever algorithm builds it, a trained vocabulary boils down to two objects: a token-to-ID table, and (for BPE) the ordered list of merge rules used to build it.

vocab = {"the": 0, "ing": 1, "un": 2, "token": 3, "izable": 4}  # token string -> id
merges = [("t", "h"), ("th", "e"), ("i", "n")]                     # BPE's ordered merge rules, applied in order

Analogy: it's like spelling an unfamiliar name over the phone. Nobody spells out "Anderson"; everyone already knows that one. "Kowalczyk" gets broken into chunks the listener has heard before: K, O, WAL, CZYK. Subword tokenization does the same to text, keeping common words whole and spelling out anything rare using pieces the model already recognizes.

BPE (Byte-Pair Encoding). Start with a character-level vocabulary. Count every adjacent pair. Merge the most frequent pair into a new token. Repeat until you hit the target vocabulary size. Dominant algorithm: GPT-2/3/4, Llama, Gemma, Qwen2, Mistral.

Byte-level BPE. Same algorithm but over raw bytes (256 base tokens) instead of Unicode characters. Guarantees zero [UNK] tokens: any byte sequence encodes. GPT-2 uses 50,257 tokens (256 bytes + 50,000 merges + 1 special).

Unigram. Start with a huge vocabulary. Assign each token a unigram probability. Iteratively prune tokens whose removal least increases the corpus log-likelihood. Probabilistic at inference: can sample tokenizations (useful for data augmentation via subword regularization). Used by T5, mBART, ALBERT, XLNet, Gemma.

WordPiece. Merge pairs that maximize likelihood of the training corpus rather than raw frequency. Used by BERT, DistilBERT, ELECTRA.

SentencePiece vs tiktoken. SentencePiece is the library that trains vocabularies (BPE or Unigram) directly on raw Unicode text, encoding whitespace as ▁. tiktoken is OpenAI's fast encoder against pre-built vocabularies; it does not train.

Rule of thumb:

• Training a new vocabulary: SentencePiece (multilingual, no pre-tokenization) or HF Tokenizers.
• Fast inference against GPT vocab: tiktoken (cl100k_base, o200k_base).
• Both: HF Tokenizers, one library that handles training and serving.

Step 1: BPE from scratch

See code/main.py. The loop:

def train_bpe(corpus, num_merges):
    vocab = {tuple(word) + ("</w>",): count for word, count in corpus.items()}
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for symbols, freq in vocab.items():
            for a, b in zip(symbols, symbols[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = pairs.most_common(1)[0][0]
        merges.append(best)
        vocab = apply_merge(vocab, best)
    return merges

Three facts the algorithm encodes. </w> marks word end so "low" (suffix) and "lower" (prefix) stay distinct. Frequency weighting makes high-frequency pairs win early. The merge list is ordered, and inference applies merges in that same training order.

Step 2: encode with the learned merges

def encode_bpe(word, merges):
    symbols = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(symbols) - 1:
            if symbols[i] == a and symbols[i + 1] == b:
                symbols = symbols[:i] + [a + b] + symbols[i + 2:]
            else:
                i += 1
    return symbols

Naive O(n·|merges|). Production implementations (tiktoken, HF Tokenizers) use merge-rank lookup with priority queues and run in near-linear time.

Step 3: SentencePiece in practice

import sentencepiece as spm
 
spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="my_tokenizer",
    vocab_size=8000,
    model_type="bpe",          # or "unigram"
    character_coverage=0.9995, # lower for CJK (e.g. 0.9995 for English, 0.995 for Japanese)
    normalization_rule_name="nmt_nfkc",
)
 
sp = spm.SentencePieceProcessor(model_file="my_tokenizer.model")
print(sp.encode("untokenizable", out_type=str))
# ['▁un', 'token', 'izable']

Notice: no pre-tokenization required, space encoded as ▁, character_coverage controls how aggressively rare characters are preserved vs mapped to <unk>.

Step 4: tiktoken for OpenAI-compatible vocabs

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(enc.encode("untokenizable"))        # [127340, 101028]
print(len(enc.encode("Hello, world!")))   # 4

Encoding-only. Fast (Rust backend). Exact match with GPT-4/5 tokenization for byte-counting, cost estimation, context-window budgeting.

• Tokenizer drift. Training on vocab A, deploying against vocab B. Token IDs differ; model outputs garbage. Check the tokenizer.json hash in CI.
• Whitespace ambiguity. BPE "hello" vs " hello" produce different tokens. Always specify add_special_tokens and add_prefix_space explicitly.
• Multilingual undertraining. English-heavy corpora produce vocabularies that split non-Latin scripts into 5-10x more tokens. Same prompt costs 5-10x more in Japanese/Arabic on GPT-3.5. o200k_base partially fixed this.
• Emoji splits. A single emoji can take 5 tokens. Checkpoint emoji handling when budgeting context.

The 2026 stack:

Vocabulary size is a scaling decision, not a constant. Rough heuristic: 32k for <1B params, 50-100k for 1-10B, 200k+ for multilingual/frontier.

Save as outputs/skill-bpe-vs-wordpiece.md:

---
name: tokenizer-picker
description: Pick tokenizer algorithm, vocab size, library for a given corpus and deployment target.
version: 1.0.0
phase: 5
lesson: 19
tags: [nlp, tokenization]
---
 
Given a corpus (size, languages, domain) and deployment target (training from scratch / fine-tuning / API-compatible inference), output:
 
1. Algorithm. BPE, Unigram, or WordPiece. One-sentence reason.
2. Library. SentencePiece, HF Tokenizers, or tiktoken. Reason.
3. Vocab size. Rounded to nearest 1k. Reason tied to model size and language coverage.
4. Coverage settings. `character_coverage`, `byte_fallback`, special-token list.
5. Validation plan. Average tokens-per-word on held-out set, OOV rate, compression ratio, round-trip decode equality.
 
Refuse to train a character-coverage <0.995 tokenizer on corpora with rare-script content. Refuse to ship a vocab without a frozen `tokenizer.json` hash check in CI. Flag any monolingual tokenizer under 16k vocab as likely under-spec.

1. Easy. Train a 500-merge BPE on code/main.py's tiny corpus. Encode three held-out words. How many produced exactly 1 token vs >1 token?
2. Medium. Compare token counts on 100 English Wikipedia sentences between cl100k_base, o200k_base, and a SentencePiece BPE you train with vocab=32k. Report the compression ratio of each.
3. Hard. Train the same corpus with BPE, Unigram, and WordPiece. Measure downstream accuracy when using each on a small sentiment classifier. Does the choice move the needle by more than 1 point F1?

• Sennrich, Haddow, Birch (2015). Neural Machine Translation of Rare Words with Subword Units: the BPE paper.
• Kudo (2018). Subword Regularization with Unigram Language Model: the Unigram paper.
• Kudo, Richardson (2018). SentencePiece: A simple and language independent subword tokenizer: the library.
• Hugging Face: Summary of the tokenizers: concise reference.
• OpenAI tiktoken repo: cookbook + encoding list.