Tokenizers: BPE, WordPiece, SentencePiece

Your LLM does not read English. It does not read any language. It reads numbers.

The gap between "Hello, world!" and [15496, 11, 995, 0] is the tokenizer. Every word, every space, and every punctuation mark gets converted into an integer before the model sees it. That conversion is not neutral, because it bakes assumptions into the model that you cannot undo later.

Get it wrong and the model spends capacity on common words that take several tokens each, and your context window holds less text than it should. Get it right and the same window holds more meaning. Much of the gap between "this model handles code well" and "this model chokes on Python" comes down to how the tokenizer was trained.

API calls to GPT-4 or Claude are priced per token, and every token a model generates costs compute. The fewer tokens it takes to represent an output, the faster inference finishes end to end. Tokenization is not preprocessing. It is architecture.

Three Approaches That Failed (and One That Won)

There are three obvious ways to convert text to numbers. Two of them break down at scale.

Word-level tokenization splits on spaces and punctuation. "The cat sat" becomes ["The", "cat", "sat"]. It is simple, but what about "tokenization"? Or "GPT-4o"? Or a German compound word like "Geschwindigkeitsbegrenzung"? Word-level needs a huge vocabulary to cover every word in every language. Miss a word and you get the [UNK] token, the model's way of saying "I have no idea what this is." Add code, URLs, scientific notation, and many other languages, and no fixed vocabulary keeps up.

Character-level tokenization goes the other direction. "hello" becomes ["h", "e", "l", "l", "o"]. The vocabulary is tiny (a few hundred characters) and there are no unknown tokens. The cost is sequence length. A sentence that is 10 word-level tokens can run to 50 character-level tokens, and the model has to learn that "t", "h", "e" together mean "the", which spends attention on something a word-level model gets for free.

Subword tokenization sits between the two. Common words stay whole, so "the" is one token. Rare words break into pieces, so "unhappiness" becomes ["un", "happi", "ness"]. The vocabulary stays manageable (30K to 128K tokens), sequences stay short, and unknown tokens mostly go away because any word can be built from smaller pieces.

Nearly every modern LLM uses subword tokenization, including GPT-2, GPT-4, BERT, and Llama 3. The real question is which algorithm.

BPE: Byte Pair Encoding

BPE is a greedy compression algorithm repurposed for tokenization. The whole idea fits on an index card.

Start with individual characters. Count every adjacent pair in the training corpus. Merge the most frequent pair into a new token. Repeat until you reach your target vocabulary size.

Here is BPE running on a tiny corpus with the words "lower", "lowest", and "newest":

Corpus (with word frequencies):
  "lower"  x5
  "lowest" x2
  "newest" x6

Step 0 -- Start with characters:
  l o w e r       (x5)
  l o w e s t     (x2)
  n e w e s t     (x6)

Step 1 -- Count adjacent pairs:
  (e,s): 8    (s,t): 8    (l,o): 7    (o,w): 7
  (w,e): 13   (e,r): 5    (n,e): 6    ...

Step 2 -- Merge most frequent pair (w,e) -> "we":
  l o we r        (x5)
  l o we s t      (x2)
  n e we s t      (x6)

Step 3 -- Recount after the merge. The pairs that matter now:
  (l,o): 7   (o,we): 7   (we,r): 5   (we,s): 8
  (s,t): 8   (n,e): 6    (e,we): 6

Step 4 -- (we,s) and (s,t) tie at 8. Pick the first one seen, (we,s) -> "wes":
  l o we r        (x5)
  l o wes t       (x2)
  n e wes t       (x6)

Step 5 -- Merge (wes,t) -> "west":
  l o we r        (x5)
  l o west        (x2)
  n e west        (x6)

...continue until target vocab size reached.

The merge table is the tokenizer. To encode new text, apply the merges in the order they were learned. The training corpus decides which merges exist, and that choice shapes everything the model sees afterward.

Byte-Level BPE (GPT-2, GPT-3, GPT-4)

Standard BPE operates on Unicode characters. Byte-level BPE operates on raw bytes (0-255), which gives a base vocabulary of exactly 256 and handles any language or encoding without an unknown token.

GPT-2 introduced this approach. The base vocabulary covers every possible byte, and BPE merges build on top of it. OpenAI's tiktoken library implements byte-level BPE with these vocabulary sizes:

• GPT-2: 50,257 tokens
• GPT-3.5/GPT-4: ~100,256 tokens (cl100k_base encoding)
• GPT-4o: 200,019 tokens (o200k_base encoding)

WordPiece (BERT)

WordPiece looks like BPE but picks merges differently. Instead of raw frequency, it maximizes the likelihood of the training data:

BPE merge criterion:      count(A, B)
WordPiece merge criterion: count(AB) / (count(A) * count(B))

BPE asks which pair appears most often. WordPiece asks which pair appears together more often than chance would predict. The two questions produce different vocabularies, because WordPiece favors merges where co-occurrence is surprising and not just frequent.

WordPiece also uses a "##" prefix for continuation subwords:

"unhappiness" -> ["un", "##happi", "##ness"]
"embedding"   -> ["em", "##bed", "##ding"]

The "##" prefix marks a piece that continues the previous token. BERT uses WordPiece with a vocabulary of 30,522 tokens. Not every BERT descendant follows it: RoBERTa, despite the name, uses BPE.

SentencePiece (Llama, T5)

SentencePiece treats the input as a raw stream of Unicode characters, including whitespace. There is no pre-tokenization step and no language-specific rule about word boundaries, so it works on Chinese, Japanese, Thai, and other languages where spaces do not separate words.

SentencePiece supports two algorithms:

• BPE mode: same merge logic as standard BPE, applied to raw character sequences
• Unigram mode: starts with a large vocabulary and repeatedly removes the tokens that least affect the overall likelihood. It is BPE run backwards: prune instead of merge.

Llama 2 uses SentencePiece BPE with a vocabulary of 32,000 tokens. T5 uses SentencePiece Unigram, also with 32,000 tokens. Llama 3 switched to a tiktoken-based byte-level BPE tokenizer with 128,256 tokens.

Vocabulary Size Tradeoffs

Vocabulary size is a real engineering decision with measurable consequences.

Take 4,096-dimensional embeddings. With a 128K vocabulary the embedding matrix alone is 128,000 x 4,096 = 524 million parameters. With a 32K vocabulary it is 131 million. The tokenizer choice by itself moves roughly 390 million parameters.

Larger vocabularies also compress text harder. The same English paragraph takes fewer tokens with a 128K vocabulary than with a 32K one, and fewer tokens means fewer forward passes during generation. For a model serving heavy traffic, that is a direct cut in compute cost.

Vocabulary sizes keep growing. GPT-2 used 50,257, GPT-4 uses ~100K, Llama 3 uses 128K, and GPT-4o uses 200K.

The Multilingual Tax

Tokenizers trained mostly on English treat other languages badly. Korean text in GPT-2's tokenizer can average 2-3 tokens per word, and Chinese can be worse. A Korean user then gets a context window that holds far less text than an English user's, at the same price per token.

That is one reason Llama 3 quadrupled its vocabulary from 32K to 128K. More tokens for non-English scripts means more even compression across languages.

Step 1: Character-Level Tokenizer

Start at the foundation. A character-level tokenizer maps each character to its Unicode code point. It needs no training and has no unknown tokens.

class CharTokenizer:
    def encode(self, text):
        return [ord(c) for c in text]
 
    def decode(self, tokens):
        return "".join(chr(t) for t in tokens)

"hello" becomes [104, 101, 108, 108, 111]. Every character is its own token, and that is the baseline we improve on.

Step 2: BPE Tokenizer from Scratch

Now the real thing. We train on raw bytes (like GPT-2), count pairs, merge the most frequent, and record every merge in order. The merge table is the tokenizer.

from collections import Counter
 
class BPETokenizer:
    def __init__(self):
        self.merges = {}
        self.vocab = {}
 
    def _get_pairs(self, tokens):
        pairs = Counter()
        for i in range(len(tokens) - 1):
            pairs[(tokens[i], tokens[i + 1])] += 1
        return pairs
 
    def _merge_pair(self, tokens, pair, new_token):
        merged = []
        i = 0
        while i < len(tokens):
            if i < len(tokens) - 1 and tokens[i] == pair[0] and tokens[i + 1] == pair[1]:
                merged.append(new_token)
                i += 2
            else:
                merged.append(tokens[i])
                i += 1
        return merged
 
    def train(self, text, num_merges):
        tokens = list(text.encode("utf-8"))
        self.vocab = {i: bytes([i]) for i in range(256)}
 
        for i in range(num_merges):
            pairs = self._get_pairs(tokens)
            if not pairs:
                break
            best_pair = max(pairs, key=pairs.get)
            new_token = 256 + i
            tokens = self._merge_pair(tokens, best_pair, new_token)
            self.merges[best_pair] = new_token
            self.vocab[new_token] = self.vocab[best_pair[0]] + self.vocab[best_pair[1]]
 
        return self
 
    def encode(self, text):
        tokens = list(text.encode("utf-8"))
        for pair, new_token in self.merges.items():
            tokens = self._merge_pair(tokens, pair, new_token)
        return tokens
 
    def decode(self, tokens):
        byte_sequence = b"".join(self.vocab[t] for t in tokens)
        return byte_sequence.decode("utf-8", errors="replace")

The training loop is the core of BPE: count pairs, merge the winner, repeat. Each merge shrinks the token list, and after num_merges rounds the vocabulary has grown from 256 base bytes to 256 + num_merges.

Encoding applies merges in the exact order they were learned, and the order matters. On the corpus above, the third merge builds "he " and the fourth builds "the " from "t" + "he ". Encoding has to apply the third merge first, or the fourth has nothing to build on.

Decoding runs the other way. Look up each token ID in the vocabulary, concatenate the bytes, and decode them as UTF-8.

Step 3: Encode and Decode Roundtrip

corpus = (
    "The cat sat on the mat. The cat ate the rat. "
    "The dog sat on the log. The dog ate the frog. "
    "Natural language processing is the study of how computers "
    "understand and generate human language. "
    "Tokenization is the first step in any NLP pipeline."
)
 
tokenizer = BPETokenizer()
tokenizer.train(corpus, num_merges=40)
 
test_sentences = [
    "The cat sat on the mat.",
    "Natural language processing",
    "tokenization pipeline",
    "unhappiness",
]
 
for sentence in test_sentences:
    encoded = tokenizer.encode(sentence)
    decoded = tokenizer.decode(encoded)
    raw_bytes = len(sentence.encode("utf-8"))
    ratio = len(encoded) / raw_bytes
    print(f"'{sentence}'")
    print(f"  Tokens: {len(encoded)} (from {raw_bytes} bytes) -- ratio: {ratio:.2f}")
    print(f"  Roundtrip: {'PASS' if decoded == sentence else 'FAIL'}")

The compression ratio tells you how well the tokenizer is doing. A ratio of 0.50 means it produced half as many tokens as raw bytes, and lower is better. On the training corpus the ratio will look good. On text the corpus never saw, like "unhappiness", it will be worse, because unseen patterns fall back to raw bytes.

Step 4: Compare with tiktoken

import tiktoken
 
enc = tiktoken.get_encoding("cl100k_base")
 
texts = [
    "The cat sat on the mat.",
    "unhappiness",
    "Hello, world!",
    "def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)",
    "Geschwindigkeitsbegrenzung",
]
 
for text in texts:
    our_tokens = tokenizer.encode(text)
    tiktoken_tokens = enc.encode(text)
    tiktoken_pieces = [enc.decode([t]) for t in tiktoken_tokens]
    print(f"'{text}'")
    print(f"  Our BPE:   {len(our_tokens)} tokens")
    print(f"  tiktoken:  {len(tiktoken_tokens)} tokens -> {tiktoken_pieces}")

tiktoken uses the same core algorithm, trained on a vastly larger corpus with roughly 100K merges. The difference is the training data and the number of merges. A tokenizer trained on one paragraph with 40 merges will not keep up with that, but the mechanism is the same.

Step 5: Vocabulary Analysis

def analyze_vocabulary(tokenizer, test_texts):
    total_tokens = 0
    total_chars = 0
    token_usage = Counter()
 
    for text in test_texts:
        encoded = tokenizer.encode(text)
        total_tokens += len(encoded)
        total_chars += len(text)
        for t in encoded:
            token_usage[t] += 1
 
    print(f"Vocabulary size: {len(tokenizer.vocab)}")
    print(f"Total tokens across all texts: {total_tokens}")
    print(f"Total characters: {total_chars}")
    print(f"Avg tokens per character: {total_tokens / total_chars:.2f}")
 
    print(f"\nMost used tokens:")
    for token_id, count in token_usage.most_common(10):
        token_bytes = tokenizer.vocab[token_id]
        display = token_bytes.decode("utf-8", errors="replace")
        print(f"  Token {token_id:4d}: '{display}' (used {count} times)")
 
    unused = [t for t in tokenizer.vocab if t not in token_usage]
    print(f"\nUnused tokens: {len(unused)} out of {len(tokenizer.vocab)}")

This shows the Zipf-style skew in your vocabulary. A few tokens dominate, starting with the space, and most tokens are rarely used. In this code, the earliest merges get the lowest new IDs, so the patterns learned first are the ones that were most frequent.

Your scratch BPE works. Here is what the production tools look like.

tiktoken (OpenAI)

import tiktoken
 
enc = tiktoken.get_encoding("cl100k_base")
 
text = "Tokenizers convert text to integers"
tokens = enc.encode(text)
print(f"Tokens: {tokens}")
print(f"Pieces: {[enc.decode([t]) for t in tokens]}")
print(f"Roundtrip: {enc.decode(tokens)}")

tiktoken is written in Rust with Python bindings. It runs the same BPE algorithm with an implementation built for speed.

Hugging Face tokenizers

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel
 
tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = ByteLevel()
 
trainer = BpeTrainer(vocab_size=1000, special_tokens=["<pad>", "<eos>", "<unk>"])
tokenizer.train(["corpus.txt"], trainer)
 
output = tokenizer.encode("The cat sat on the mat.")
print(f"Tokens: {output.tokens}")
print(f"IDs: {output.ids}")

The Hugging Face tokenizers library is also Rust under the hood, and it is built to train BPE on large corpora quickly. Reach for it when you train your own model.

Loading Llama's Tokenizer

from transformers import AutoTokenizer
 
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
 
text = "Tokenizers are the unsung heroes of LLMs"
tokens = tokenizer.encode(text)
print(f"Token IDs: {tokens}")
print(f"Tokens: {tokenizer.convert_ids_to_tokens(tokens)}")
print(f"Vocab size: {tokenizer.vocab_size}")
 
multilingual = ["Hello world", "Hola mundo", "Bonjour le monde"]
for text in multilingual:
    ids = tokenizer.encode(text)
    print(f"'{text}' -> {len(ids)} tokens")

Llama 3's 128K vocabulary should compress non-English text better than GPT-2's 50K vocabulary. Check it yourself by encoding the same sentence in several languages and counting the tokens.

This lesson produces outputs/prompt-tokenizer-analyzer.md, a reusable prompt that analyzes tokenization efficiency for a text and model combination. Feed it a text sample and it tells you which model's tokenizer handles it best.

1. Modify the BPE tokenizer to print the vocabulary at each merge step. Watch how "h" + "e " becomes "he ", then "t" + "he " becomes "the ". Track how common English words get assembled piece by piece.

2. Add special tokens (<pad>, <eos>, <unk>) to the BPE tokenizer. Assign them IDs 0, 1, 2 and shift all other tokens accordingly. Implement a pre-tokenization step that splits on whitespace before running BPE.

3. Implement the WordPiece merge criterion (likelihood ratio instead of frequency). Train both BPE and WordPiece on the same corpus with the same number of merges. Compare the resulting vocabularies. Which one produces more linguistically meaningful subwords?

4. Build a multilingual tokenizer efficiency benchmark. Take 10 sentences in English, Spanish, Chinese, Korean, and Arabic. Tokenize each with tiktoken (cl100k_base) and measure the average tokens per character. Quantify the "multilingual tax" for each language.

5. Train your BPE tokenizer on a larger corpus (download a Wikipedia article). Tune the number of merges to achieve a compression ratio within 10% of tiktoken on that same text. This forces you to understand the relationship between corpus size, merge count, and compression quality.

• Sennrich et al., 2016, "Neural Machine Translation of Rare Words with Subword Units": the paper that introduced BPE for NLP, turning a 1994 compression algorithm into the foundation of modern tokenization
• Kudo & Richardson, 2018, "SentencePiece: A simple and language independent subword tokenizer": language-agnostic tokenization that made multilingual models practical
• OpenAI tiktoken repository: production BPE implementation in Rust with Python bindings, used by GPT-3.5/4/4o
• Hugging Face Tokenizers documentation: production-grade tokenizer training with Rust performance