Building a Tokenizer from Scratch

Your BPE tokenizer from Lesson 01 works on English text. Now give it Japanese, or emoji, or Python code with mixed tabs and spaces.

It breaks.

BPE isn't the problem. The implementation is incomplete. A production tokenizer reads raw bytes whatever the encoding, normalizes Unicode before splitting, keeps special tokens out of the merge step, chains pre-tokenization with subword splitting, and runs fast enough that it doesn't bottleneck a training pipeline processing 15 trillion tokens.

GPT-2's tokenizer has 50,257 tokens. Llama 3 has 128,256. GPT-4 has roughly 100,000. The merge tables behind those vocabularies were trained on hundreds of gigabytes of text. The machinery around them (normalization, pre-tokenization, special token injection, chat template formatting) is what separates a tokenizer that handles "hello world" from one that handles the web.

You are going to build that machinery.

The Full Pipeline

A production tokenizer is a pipeline of five stages, and each one solves a separate problem.

Here is what each stage does:

Byte-Level BPE

Lesson 01's tokenizer operated on UTF-8 bytes, which was the right call. It left a gap, though: what happens when those bytes are not valid UTF-8?

Byte-level BPE closes it by treating each byte value (0-255) as a valid token, so the base vocabulary is exactly 256 entries. A text file, a binary, or a corrupted file can be tokenized without producing an unknown token.

GPT-2 added a trick: map each byte to a printable Unicode character so the vocabulary stays human-readable. Byte 0x20 (space) becomes the character "Ġ" in their mapping. This is purely cosmetic, and the algorithm doesn't care.

The bigger payoff is coverage. Chinese characters are 3 UTF-8 bytes each, and Japanese can take 3-4. Arabic, Devanagari, and emoji are also just byte sequences. BPE finds patterns in them the same way it finds patterns in English ASCII bytes.

Pre-Tokenization

Before BPE sees your text, split it into chunks. That stops the merge algorithm from creating tokens that span word boundaries.

GPT-2 uses this regex:

'(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

It splits on contractions ("don't" becomes "don" + "'t"), words with an optional leading space, numbers, punctuation, and whitespace. The leading space stays attached to the word, so "the cat" becomes ["the", " cat"], not ["the", " ", "cat"].

Llama uses SentencePiece, which skips the regex. It treats the raw byte stream as one long sequence and lets BPE find the boundaries. That is simpler, but BPE gets more freedom to create tokens that cross words.

The choice matters. GPT-2's regex blocks merges across a word boundary. SentencePiece allows them, which can compress better at the cost of less interpretable tokens.

Special Tokens

Production tokenizers reserve token IDs for structural markers:

BPE does not split special tokens. They are matched as exact strings before the merge algorithm runs and replaced with their fixed ID, and the surrounding text is tokenized normally.

Chat Templates

This is where most people get confused and most implementations break.

When you send messages to a chat model, the API takes a list of messages:

[
  {"role": "system", "content": "You are helpful."},
  {"role": "user", "content": "Hello"},
  {"role": "assistant", "content": "Hi there!"}
]

The model does not see JSON. It sees a flat token sequence, and the chat template converts messages into that sequence using special tokens. Each model does it differently:

Llama 3:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are helpful.<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hi there!<|eot_id|>

ChatGPT:
<|im_start|>system
You are helpful.<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there!<|im_end|>

Get the template wrong and the model produces garbage, because it was trained on one exact format. A missing newline, a swapped token, or an extra space puts the input outside the training distribution.

Speed

Pure Python is too slow for production tokenization.

tiktoken (OpenAI) is written in Rust with Python bindings. HuggingFace tokenizers is also Rust, and SentencePiece is C++. These get 10-100x speedups over pure Python.

For perspective: tokenizing 15 trillion tokens for Llama 3 pre-training at 1 million tokens per second (fast Python) would take 174 days. At 100 million tokens per second (Rust), it takes 1.7 days.

You build in Python here to understand the algorithm. In production you'd use a compiled implementation and touch only the Python wrapper.

Step 1: Byte-Level Encoding

The foundation: turn a string into a list of byte values, and turn a list of byte values back into a string.

def bytes_to_tokens(text):
    return list(text.encode("utf-8"))
 
def tokens_to_text(token_bytes):
    return bytes(token_bytes).decode("utf-8", errors="replace")

Test on multilingual text to see the byte counts:

texts = [
    ("English", "hello"),
    ("Chinese", "你好"),
    ("Emoji", "🔥"),
    ("Mixed", "hello你好🔥"),
]
 
for label, text in texts:
    b = bytes_to_tokens(text)
    print(f"{label}: {len(text)} chars -> {len(b)} bytes -> {b}")

"hello" is 5 bytes. "你好" is 6 bytes (3 per character). The fire emoji is 4 bytes. The byte-level tokenizer doesn't care what language it is. Bytes are bytes.

Step 2: Pre-Tokenizer with Regex

Split text into chunks with the GPT-2 regex. BPE tokenizes each chunk independently.

import re
 
try:
    import regex
    GPT2_PATTERN = regex.compile(
        r"""'(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""
    )
except ImportError:
    GPT2_PATTERN = re.compile(
        r"""'(?:[sdmt]|ll|ve|re)| ?[a-zA-Z]+| ?[0-9]+| ?[^\s\w]+|\s+(?!\S)|\s+"""
    )
 
def pre_tokenize(text):
    return [match.group() for match in GPT2_PATTERN.finditer(text)]

The regex module supports Unicode property escapes (\p{L} for letters, \p{N} for numbers). The standard library re module does not, so the code falls back to ASCII character classes. For multilingual text, install regex.

Try it:

print(pre_tokenize("Hello, world! Don't stop."))
# ['Hello', ',', ' world', '!', ' Don', "'t", ' stop', '.']

The leading space stays attached to every word after the first, contractions split at the apostrophe, and punctuation becomes its own chunk. BPE can't merge across those boundaries.

Step 3: BPE on Byte Sequences

The core algorithm from Lesson 01, now run on each pre-tokenized chunk independently.

from collections import Counter
 
def get_byte_pairs(chunks):
    pairs = Counter()
    for chunk in chunks:
        byte_seq = list(chunk.encode("utf-8"))
        for i in range(len(byte_seq) - 1):
            pairs[(byte_seq[i], byte_seq[i + 1])] += 1
    return pairs
 
def apply_merge(byte_seq, pair, new_id):
    merged = []
    i = 0
    while i < len(byte_seq):
        if i < len(byte_seq) - 1 and byte_seq[i] == pair[0] and byte_seq[i + 1] == pair[1]:
            merged.append(new_id)
            i += 2
        else:
            merged.append(byte_seq[i])
            i += 1
    return merged

Step 4: Special Token Handling

Special tokens need exact matching and fixed IDs, and they bypass BPE.

class SpecialTokenHandler:
    def __init__(self):
        self.special_tokens = {}
        self.pattern = None
 
    def add_token(self, token_str, token_id):
        self.special_tokens[token_str] = token_id
        escaped = [re.escape(t) for t in sorted(self.special_tokens.keys(), key=len, reverse=True)]
        self.pattern = re.compile("|".join(escaped))
 
    def split_with_specials(self, text):
        if not self.pattern:
            return [(text, False)]
        parts = []
        last_end = 0
        for match in self.pattern.finditer(text):
            if match.start() > last_end:
                parts.append((text[last_end:match.start()], False))
            parts.append((match.group(), True))
            last_end = match.end()
        if last_end < len(text):
            parts.append((text[last_end:], False))
        return parts

Step 5: Full Tokenizer Class

Chain the stages: normalize, split on special tokens, pre-tokenize, BPE merge, map to IDs.

import unicodedata
 
class ProductionTokenizer:
    def __init__(self):
        self.merges = {}
        self.vocab = {i: bytes([i]) for i in range(256)}
        self.special_handler = SpecialTokenHandler()
        self.next_id = 256
 
    def normalize(self, text):
        return unicodedata.normalize("NFKC", text)
 
    def train(self, text, num_merges):
        text = self.normalize(text)
        chunks = pre_tokenize(text)
        chunk_bytes = [list(chunk.encode("utf-8")) for chunk in chunks]
 
        for i in range(num_merges):
            pairs = Counter()
            for seq in chunk_bytes:
                for j in range(len(seq) - 1):
                    pairs[(seq[j], seq[j + 1])] += 1
            if not pairs:
                break
            best = max(pairs, key=pairs.get)
            new_id = self.next_id
            self.next_id += 1
            self.merges[best] = new_id
            self.vocab[new_id] = self.vocab[best[0]] + self.vocab[best[1]]
            chunk_bytes = [apply_merge(seq, best, new_id) for seq in chunk_bytes]
 
    def add_special_token(self, token_str):
        token_id = self.next_id
        self.next_id += 1
        self.special_handler.add_token(token_str, token_id)
        self.vocab[token_id] = token_str.encode("utf-8")
        return token_id
 
    def encode(self, text):
        text = self.normalize(text)
        parts = self.special_handler.split_with_specials(text)
        all_ids = []
        for part_text, is_special in parts:
            if is_special:
                all_ids.append(self.special_handler.special_tokens[part_text])
            else:
                for chunk in pre_tokenize(part_text):
                    byte_seq = list(chunk.encode("utf-8"))
                    for pair, new_id in self.merges.items():
                        byte_seq = apply_merge(byte_seq, pair, new_id)
                    all_ids.extend(byte_seq)
        return all_ids
 
    def decode(self, ids):
        byte_parts = []
        for token_id in ids:
            if token_id in self.vocab:
                byte_parts.append(self.vocab[token_id])
        return b"".join(byte_parts).decode("utf-8", errors="replace")
 
    def vocab_size(self):
        return len(self.vocab)

Step 6: Multilingual Test

The real test: English, Chinese, emoji, and code.

corpus = (
    "The quick brown fox jumps over the lazy dog. "
    "The quick brown fox runs through the forest. "
    "Machine learning models process natural language. "
    "Deep learning transforms how we build software. "
    "def train(model, data): return model.fit(data) "
    "def predict(model, x): return model(x) "
)
 
tok = ProductionTokenizer()
tok.train(corpus, num_merges=50)
 
bos = tok.add_special_token("<|begin|>")
eos = tok.add_special_token("<|end|>")
 
test_texts = [
    "The quick brown fox.",
    "你好世界",
    "Hello 🌍 World",
    "def foo(x): return x + 1",
    f"<|begin|>Hello<|end|>",
]
 
for text in test_texts:
    ids = tok.encode(text)
    decoded = tok.decode(ids)
    print(f"Input:   {text}")
    print(f"Tokens:  {len(ids)} ids")
    print(f"Decoded: {decoded}")
    print()

Chinese characters produce 3 bytes each and the emoji produces 4. Neither crashes the tokenizer or produces an unknown token. That is what byte-level BPE buys you.

Comparing Real Tokenizers

Load the real tokenizers from Llama 3, GPT-4, and Mistral and see how each handles the same multilingual paragraph.

import tiktoken
 
gpt4_enc = tiktoken.get_encoding("cl100k_base")
 
test_paragraph = "Machine learning is powerful. 机器学习很强大。 L'apprentissage automatique est puissant. 🤖💪"
 
tokens = gpt4_enc.encode(test_paragraph)
pieces = [gpt4_enc.decode([t]) for t in tokens]
print(f"GPT-4 ({len(tokens)} tokens): {pieces}")

from transformers import AutoTokenizer
 
llama_tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
mistral_tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
 
for name, tok in [("Llama 3", llama_tok), ("Mistral", mistral_tok)]:
    tokens = tok.encode(test_paragraph)
    pieces = tok.convert_ids_to_tokens(tokens)
    print(f"{name} ({len(tokens)} tokens): {pieces[:20]}...")

The token counts differ for the same text. Llama 3's 128K vocabulary merges common patterns more aggressively, GPT-4's 100K sits in the middle, and Mistral's 32K produces more tokens but has a smaller embedding layer.

The tradeoff stays the same: a larger vocabulary gives shorter sequences and more parameters.

The output is a prompt for building and debugging production tokenizers: outputs/prompt-tokenizer-builder.md.

1. Easy: Add a get_token_bytes(id) method that shows the raw bytes for any token ID. Use it to inspect what your most common merged tokens actually represent.
2. Medium: Implement the Llama-style pre-tokenizer that splits on whitespace and digits but keeps leading spaces. Compare its vocabulary with the GPT-2 regex approach on the same corpus.
3. Hard: Add a chat template method that takes a list of {"role": ..., "content": ...} messages and produces the correct token sequence for the Llama 3 chat format. Test it against the HuggingFace implementation.

• OpenAI tiktoken source -- Rust BPE implementation used by GPT-3.5/4
• HuggingFace tokenizers -- Rust tokenizer library supporting BPE, WordPiece, Unigram
• Llama 3 paper (Meta, 2024) -- details on 128K vocabulary and tokenizer training
• SentencePiece (Kudo & Richardson, 2018) -- language-agnostic tokenization
• GPT-2 tokenizer source -- the original byte-to-Unicode mapping