When we interact with Large Language Models (LLMs) like GPT-4 or Claude, it is easy to assume they process human language much like we do—reading words, understanding grammar, and parsing sentences. But beneath the surface, neural networks do not understand text at all. They operate exclusively on numbers.

The unsung hero bridging this gap is the tokenizer. Tokenization is the very first step in any Natural Language Processing (NLP) pipeline, translating raw strings into sequences of integer token IDs. Today, the dominant algorithm for modern LLMs is Byte-Pair Encoding (BPE).

In this post, we will demystify tokenization, explore how BPE works under the hood, and build our own fully functional BPE tokenizer in under 50 lines of Python.

---

The Tokenization Dilemma: Words, Characters, or Subwords?

To convert text to numbers, we must decide what our fundamental “building block” (token) will be. Early NLP models relied on one of two extremes:

1. Word-level Tokenization: Split text by spaces and punctuation. While intuitive, this approach leads to massive vocabularies (hundreds of thousands of words) and struggles with unseen words, typos, or morphological variations (e.g., “tokenize”, “tokenizing”, “tokenizer”).
2. Character-level Tokenization: Treat every individual character as a token. This keeps the vocabulary tiny (just a few hundred characters), but it forces the LLM to learn language from scratch—down to spelling—and results in extremely long sequences that bog down attention mechanisms.

Subword Tokenization—specifically Byte-Pair Encoding—strikes the perfect middle ground. Common words like "the" or "hug" remain intact as single tokens, while rare or complex words like "unforgivable" get broken down into semantic subwords: "un" + "forgiv" + "able".

---

How Byte-Pair Encoding Works

Originally developed in 1994 as a data compression algorithm, BPE was adapted for neural network tokenizers by Sennrich et al. in 2015.

Instead of starting with words, BPE starts at the byte level. Because UTF-8 represents all valid text as raw byte values ranging from 0 to 255, starting with bytes ensures our tokenizer can handle any language, symbol, or emoji without throwing an “out-of-vocabulary” error.

The BPE training algorithm follows three simple steps:

1. Initialize: Convert text into a sequence of raw byte IDs (values 0–255).
2. Count Pairs: Count the frequency of every adjacent pair of token IDs in the dataset.
3. Merge: Take the most frequently occurring pair, replace all of its occurrences with a brand-new token ID (starting at 256), and add this merge rule to our vocabulary.

We repeat steps 2 and 3 for a fixed number of iterations until we reach our desired target vocabulary size.

---

Building a Mini BPE Tokenizer in Python

Let’s turn this theory into code. The following implementation demonstrates training, encoding, and decoding using BPE.

def get_stats(ids):
    """Count frequency of consecutive token pairs."""
    counts = {}
    for pair in zip(ids, ids[1:]):
        counts[pair] = counts.get(pair, 0) + 1
    return counts

def merge(ids, pair, idx):
    """Replace all instances of `pair` in `ids` with new token `idx`."""
    newids = []
    i = 0
    while i < len(ids):
        if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:
            newids.append(idx)
            i += 2
        else:
            newids.append(ids[i])
            i += 1
    return newids

def train_bpe(text, num_merges):
    """Train BPE by iteratively merging the most frequent token pairs."""
    ids = list(text.encode("utf-8"))
    merges = {}
    vocab = {i: bytes([i]) for i in range(256)}

    for i in range(num_merges):
        stats = get_stats(ids)
        if not stats:
            break
        best_pair = max(stats, key=stats.get)
        idx = 256 + i
        ids = merge(ids, best_pair, idx)
        merges[best_pair] = idx
        vocab[idx] = vocab[best_pair[0]] + vocab[best_pair[1]]

    return merges, vocab

def encode(text, merges):
    """Convert raw text into token IDs using learned merge rules."""
    ids = list(text.encode("utf-8"))
    while len(ids) >= 2:
        stats = get_stats(ids)
        # Find the pair that was merged earliest during training
        pair = min(stats, key=lambda p: merges.get(p, float("inf")))
        if pair not in merges:
            break
        ids = merge(ids, pair, merges[pair])
    return ids

def decode(ids, vocab):
    """Convert token IDs back into readable UTF-8 text."""
    byte_seq = b"".join(vocab[i] for i in ids)
    return byte_seq.decode("utf-8", errors="replace")

# --- Demonstration ---
training_corpus = "low lower lowest newest widest low lower low"
print("Training corpus:", training_corpus)

# Train BPE tokenizer with 6 merge operations
merges, vocab = train_bpe(training_corpus, num_merges=6)

# Test encoding and decoding on new text
input_text = "lower widest"
encoded_tokens = encode(input_text, merges)
decoded_text = decode(encoded_tokens, vocab)

print("\n--- Test Results ---")
print(f"Input Text:     '{input_text}'")
print(f"Encoded Tokens: {encoded_tokens}")
print(f"Decoded Text:   '{decoded_text}'")
print(f"Vocabulary Size: {len(vocab)}")

실행 결과

Training corpus: low lower lowest newest widest low lower low

--- Test Results ---
Input Text:     'lower widest'
Encoded Tokens: [257, 101, 114, 32, 119, 105, 100, 101, 260]
Decoded Text:   'lower widest'
Vocabulary Size: 262

Decoding the Code:

• get_stats: Loops through adjacent elements using zip(ids, ids[1:]) to tallies occurrence frequencies.
• train_bpe: Converts text into standard UTF-8 bytes (0–255), finds the most frequent pair, replaces it with a new integer (256, 257, etc.), and records the merge.
• encode: Takes unseen text, converts it to bytes, and applies the recorded merges in priority order.
• decode: Maps each token ID back to its byte representation and concatenates them to reconstruct human-readable text.

---

Beyond the Basics: Real-World Tokenizers

While our mini BPE tokenizer demonstrates the core principles perfectly, production-grade tokenizers like OpenAI’s tiktoken or Hugging Face’s tokenizers library include extra layers of complexity:

1. Pre-tokenization (Regex Splitting): Production tokenizers split text using regular expressions before running BPE. This prevents the model from merging across structural boundaries (for example, combining the end of one word and a space into a single token across sentences).
2. Special Tokens: Tokens like “ or [PAD] are explicitly added to signal document boundaries or control model behavior.
3. Optimized Engines: Implementing pair-matching in pure Python is relatively slow. Commercial tokenizers are written in Rust or C++ and utilize parallel processing to tokenize gigabytes of text in seconds.

---

Conclusion

Tokenization is often treated as a hidden black box in machine learning pipelines, but its mechanics are elegant and accessible. By starting with raw bytes and iteratively merging frequent pairs, Byte-Pair Encoding compresses human language into clean, model-ready integer sequences.

Now that you understand how tokenizers operate, you can better appreciate why LLMs struggle with tasks like counting letters in a word (e.g., “How many ‘r’s in strawberry?”)—to a model trained on subword IDs, the word “strawberry” isn’t a sequence of ten letters, but a small handful of pre-packaged numerical tokens!