Text Processing — Tokenization, Stemming, Lemmatization

A model cannot read "The cats were running." It reads integers.

Every NLP system opens with the same three questions: where does a word start, what's its root, and when should "run", "running", and "ran" count as the same thing versus three different ones?

Get tokenization wrong and the model learns from garbage. If your tokenizer treats don't as one token but do n't as two, the training distribution splits. If your stemmer collapses organization and organ to the same stem, topic modeling dies. If your lemmatizer needs part-of-speech context but you don't pass it, verbs get treated as nouns.

This lesson builds the three preprocessing steps from scratch, then shows how NLTK and spaCy do the same work so you can see the tradeoffs.

Three operations, each with its own job and its own failure mode.

Tokenization splits a string into tokens. "Token" is deliberately vague because the right granularity depends on the task: word-level for classical NLP, subword for transformers, character for languages without whitespace.

Stemming chops suffixes with rules. Fast, aggressive, dumb. running -> run. organization -> organ. That second one is the failure mode.

Lemmatization reduces a word to its dictionary form using grammar knowledge. Slower, accurate, needs a lookup table or morphological analyzer. ran -> run (needs to know "ran" is past tense of "run"). better -> good (needs to know comparative forms).

Rule of thumb: stem when speed matters and you can tolerate noise (search indexing, rough classification); lemmatize when meaning matters (question answering, semantic search, anything the user will read).

Step 1: a regex word tokenizer

The simplest useful tokenizer splits on non-alphanumeric characters while keeping punctuation as its own tokens. Not perfect, not final, but it runs in one line.

import re
 
def tokenize(text):
    return re.findall(r"[A-Za-z]+(?:'[A-Za-z]+)?|[0-9]+|[^\sA-Za-z0-9]", text)

Three patterns compete in order of precedence: words with an optional inner apostrophe (don't, it's), pure numbers, and any single non-whitespace, non-alphanumeric character standing alone as punctuation.

>>> tokenize("The cats weren't running at 3pm.")
['The', 'cats', "weren't", 'running', 'at', '3', 'pm', '.']

Watch for the failure modes: 3pm splits to ['3', 'pm'] because the regex alternates between letter runs and digit runs, and URLs, emails, and hashtags break the same way. Good enough for most tasks, but for production, add specific patterns before the general ones.

Step 2: a Porter stemmer (step 1a only)

The full Porter algorithm has five phases of rules. Step 1a alone covers the most frequent English suffixes and teaches the pattern.

def stem_step_1a(word):
    if word.endswith("sses"):
        return word[:-2]
    if word.endswith("ies"):
        return word[:-2]
    if word.endswith("ss"):
        return word
    if word.endswith("s") and len(word) > 1:
        return word[:-1]
    return word

>>> [stem_step_1a(w) for w in ["caresses", "ponies", "caress", "cats"]]
['caress', 'poni', 'caress', 'cat']

Read the rules top-down: the ies -> i rule is why ponies -> poni instead of pony, and only step 1b in the real algorithm would fix it. Rules compete. Earlier rules win. The order matters more than any single rule.

Step 3: a lookup-based lemmatizer

Lemmatization proper needs morphology. A tractable teaching version uses a small lemma table and a fallback.

LEMMA_TABLE = {
    ("running", "VERB"): "run",
    ("ran", "VERB"): "run",
    ("runs", "VERB"): "run",
    ("better", "ADJ"): "good",
    ("best", "ADJ"): "good",
    ("cats", "NOUN"): "cat",
    ("cat", "NOUN"): "cat",
    ("were", "VERB"): "be",
    ("was", "VERB"): "be",
    ("is", "VERB"): "be",
}
 
def lemmatize(word, pos):
    key = (word.lower(), pos)
    if key in LEMMA_TABLE:
        return LEMMA_TABLE[key]
    if pos == "VERB" and word.endswith("ing"):
        return word[:-3]
    if pos == "NOUN" and word.endswith("s"):
        return word[:-1]
    return word.lower()

>>> lemmatize("running", "VERB")
'run'
>>> lemmatize("cats", "NOUN")
'cat'
>>> lemmatize("better", "ADJ")
'good'
>>> lemmatize("watched", "VERB")
'watched'

The last case is the key teaching moment. watched is not in our table and our fallback only handles ing. Real lemmatization covers ed, irregular verbs, comparative adjectives, plurals with sound changes (children -> child). This is why production systems use WordNet, spaCy's morphologizer, or a full morphological analyzer.

Step 4: pipe them together

def preprocess(text, pos_tagger=None):
    tokens = tokenize(text)
    stems = [stem_step_1a(t.lower()) for t in tokens]
    tags = pos_tagger(tokens) if pos_tagger else [(t, "NOUN") for t in tokens]
    lemmas = [lemmatize(word, pos) for word, pos in tags]
    return {"tokens": tokens, "stems": stems, "lemmas": lemmas}

The missing piece is a POS tagger. Phase 5 · 07 (POS Tagging) builds one. For now, default everything to NOUN and acknowledge the limitation.

NLTK and spaCy ship the production versions, in a few lines each.

NLTK

import nltk
nltk.download("punkt_tab")
nltk.download("wordnet")
nltk.download("averaged_perceptron_tagger_eng")
 
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk import pos_tag
 
text = "The cats were running."
tokens = word_tokenize(text)
stems = [PorterStemmer().stem(t) for t in tokens]
lemmatizer = WordNetLemmatizer()
tagged = pos_tag(tokens)
 
 
def nltk_pos_to_wordnet(tag):
    if tag.startswith("V"):
        return "v"
    if tag.startswith("J"):
        return "a"
    if tag.startswith("R"):
        return "r"
    return "n"
 
 
lemmas = [lemmatizer.lemmatize(t, nltk_pos_to_wordnet(tag)) for t, tag in tagged]

word_tokenize handles contractions, Unicode, edge cases your regex misses. PorterStemmer runs all five phases. WordNetLemmatizer needs the POS tag translated from NLTK's Penn Treebank scheme to WordNet's abbreviation set. The translation wiring above is the bit most tutorials skip.

spaCy

spaCy trades that separable pipeline for a single call that does tokenization, tagging, and lemmatization together:

import spacy
 
nlp = spacy.load("en_core_web_sm")
doc = nlp("The cats were running.")
 
for token in doc:
    print(token.text, token.lemma_, token.pos_)

The      the     DET
cats     cat     NOUN
were     be      AUX
running  run     VERB
.        .       PUNCT

spaCy hides the whole pipeline behind nlp(text): tokenization, POS tagging, and lemmatization all run together, faster than NLTK at scale and more accurate out of the box. The tradeoff is that you can't easily swap individual components.

When to pick which

The two failure modes nobody warns you about

Most tutorials teach the algorithms and stop. Two things will bite a real preprocessing pipeline, and they are almost never covered.

Reproducibility drift. NLTK and spaCy change tokenization and lemmatizer behavior between versions: what produced ['do', "n't"] in spaCy 2.x may produce ["don't"] in 3.x. Your model trained on one distribution, and inference is now running on a different one, so accuracy quietly degrades and nobody knows why. Pin library versions in requirements.txt, and write a preprocessing regression test that freezes expected tokenization for 20 sample sentences. Run it on every upgrade.

Training / inference mismatch. Train with aggressive preprocessing (lowercase, stopword removal, stemming), deploy on raw user input, watch performance crater. This is the single most common production NLP failure. If you preprocess during training, you must run the identical function during inference. Ship preprocessing as a function inside the model package, not as a notebook cell the serving team rewrites.

A reusable prompt that helps engineers pick a preprocessing strategy without reading three textbooks.

Save as outputs/prompt-preprocessing-advisor.md:

---
name: preprocessing-advisor
description: Recommends a tokenization, stemming, and lemmatization setup for an NLP task.
phase: 5
lesson: 01
---
 
You advise on classical NLP preprocessing. Given a task description, you output:
 
1. Tokenization choice (regex, NLTK word_tokenize, spaCy, or transformer tokenizer). Explain why.
2. Whether to stem, lemmatize, both, or neither. Explain why.
3. Specific library calls. Name the functions. Quote the POS-tag translation if NLTK is involved.
4. One failure mode the user should test for.
 
Refuse to recommend stemming for user-visible text. Refuse to recommend lemmatization without POS tags. Flag non-English input as needing a different pipeline.

1. Easy. Extend tokenize to keep URLs as single tokens. Test: tokenize("Visit https://example.com today.") should produce one URL token.
2. Medium. Implement Porter step 1b. If a word contains a vowel and ends in ed or ing, remove it. Handle the double-consonant rule (hopping -> hop, not hopp).
3. Hard. Build a lemmatizer that uses WordNet as a lookup table but falls back to your Porter stemmer when WordNet has no entry. Measure accuracy on a tagged corpus against plain WordNet and plain Porter.

• Porter, M. F. (1980). An algorithm for suffix stripping — the original paper, five pages, still the clearest explanation.
• spaCy 101 — linguistic features — how a real pipeline is wired.
• NLTK book, chapter 3 — tokenization edge cases you haven't thought of yet.