Multilingual NLP

English has billions of labeled examples. Urdu has thousands. Maithili has almost none. Any system that has to serve a global audience runs straight into that long tail: most of the world's languages never had, and never will have, a labeled dataset built for your specific task.

Multilingual models get around this with zero-shot cross-lingual transfer: train one model across many languages at once, so a representation it learns from a high-resource language carries over to a low-resource one, then fine-tune on a task in whatever language you have labeled data for and run it on a language you don't. Fine-tune on English sentiment analysis, and the same model, with no Urdu training data at all, produces solid sentiment predictions on Urdu. That's the whole trick, and it's the reason NLP ships to the whole world instead of just the languages with big labeled datasets.

The tradeoffs and the canonical models are below. The one decision that trips up almost every team new to this: which language to fine-tune on. Most default to English without checking whether that's actually the right call.

Shared vocabulary. Multilingual models use a SentencePiece or WordPiece tokenizer trained on text from every target language at once, so the same subword unit represents the same morpheme across related languages: anti- in English and Italian gets the same token.

Shared representation. A transformer pretrained on masked language modeling across many languages learns that semantically similar sentences in different languages produce similar hidden states. mBERT, XLM-R, and NLLB all show this. Embeddings for "cat" in English cluster near "chat" in French and "gato" in Spanish, and full-sentence embeddings do the same thing.

Zero-shot transfer. Fine-tune the model on labeled data in one language, usually English, then run it at inference on any other language the model supports. No target-language labels needed. Results are strong for typologically related languages and weaker for distant ones.

Few-shot fine-tuning. Add 100-500 labeled examples in the target language and accuracy jumps to 95-98% of the English baseline on classification tasks. Of everything in this lesson, this is the single most cost-effective lever you have.

Pick by use case, not by whichever model is trending. XLM-R-base is the sane default for classification. For generation, reach for mT5 or NLLB depending on whether you're translating or generating open text. For LLM-style work, pair Aya-23 or Claude with explicit multilingual prompting.

Most teams default to English as the fine-tuning source, mostly because English is where the labeled data already sits. Research from 2026 says that default is wrong more often than people assume.

Language similarity predicts transfer quality better than raw corpus size does. For Slavic targets, German or Russian often beat English. For Indic targets, Hindi often beats English. The qWALS similarity metric (2026, based on World Atlas of Language Structures features) quantifies this directly. LANGRANK (Lin et al., ACL 2019) is a separate, earlier method that ranks candidate source languages using a combination of linguistic similarity, corpus size, and genetic relatedness.

Practical rule: if your target language has a typologically close high-resource relative, fine-tune on that one first and compare it against the English fine-tune before you commit to either.

Step 1: zero-shot cross-lingual classification

There's no labeled sentiment data in any of the three languages below, and the model was never fine-tuned on sentiment at all. It was fine-tuned on NLI (natural language inference: given a premise and a hypothesis, decide whether the premise entails, contradicts, or stays neutral toward the hypothesis). The zero-shot trick is to turn each candidate label into a hypothesis, "This text is about {label}.", and ask the model how strongly the input text entails it. Whichever label's hypothesis scores highest wins. logits[2] below reads out the entailment class in this checkpoint's label order (contradiction=0, neutral=1, entailment=2), so softmax-ing the logits and taking index 2 is exactly that entailment score.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
 
tok = AutoTokenizer.from_pretrained("joeddav/xlm-roberta-large-xnli")
model = AutoModelForSequenceClassification.from_pretrained("joeddav/xlm-roberta-large-xnli")
 
 
def classify(text, candidate_labels, hypothesis_template="This text is about {}."):
    scores = {}
    for label in candidate_labels:
        hypothesis = hypothesis_template.format(label)
        inputs = tok(text, hypothesis, return_tensors="pt", truncation=True)
        with torch.no_grad():
            logits = model(**inputs).logits[0]
        entail_score = torch.softmax(logits, dim=-1)[2].item()
        scores[label] = entail_score
    return dict(sorted(scores.items(), key=lambda x: -x[1]))
 
 
print(classify("I love this product!", ["positive", "negative", "neutral"]))
print(classify("मुझे यह उत्पाद पसंद है!", ["positive", "negative", "neutral"]))
print(classify("J'adore ce produit !", ["positive", "negative", "neutral"]))

One model, three languages, one API, and zero target-language labels: the entailment trick is what turns an NLI-tuned XLM-R into a zero-shot classifier for any language it covers.

Step 2: multilingual embedding space

If the model's shared representation actually holds, the same sentence translated into different languages should land close together in embedding space, and an unrelated sentence should land farther away regardless of language. The check below encodes one English sentence, its translations into French, Spanish, and German, and one unrelated English sentence, then compares each pair with cosine similarity.

from sentence_transformers import SentenceTransformer
import numpy as np
 
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
 
pairs = [
    ("The cat is sleeping.", "Le chat dort."),
    ("The cat is sleeping.", "El gato está durmiendo."),
    ("The cat is sleeping.", "Die Katze schläft."),
    ("The cat is sleeping.", "The dog is barking."),
]
 
for eng, other in pairs:
    emb_eng = model.encode([eng], normalize_embeddings=True)[0]
    emb_other = model.encode([other], normalize_embeddings=True)[0]
    sim = float(np.dot(emb_eng, emb_other))
    print(f"  {eng!r} <-> {other!r}: cos={sim:.3f}")

Translations land close together in embedding space. A different English sentence lands farther away. That gap is what makes cross-lingual retrieval, clustering, and similarity search work at all.

Step 3: few-shot fine-tuning strategy

Zero-shot gets a model in the door; a small pile of labeled examples in the target language is what actually closes the remaining gap to English-level accuracy. The function below runs that fine-tune given a base model, its tokenizer, and a short list of {text, label} examples.

from transformers import TrainingArguments, Trainer
from datasets import Dataset
 
 
def few_shot_finetune(base_model, base_tokenizer, examples):
    ds = Dataset.from_list(examples)
 
    def tokenize_fn(ex):
        out = base_tokenizer(ex["text"], truncation=True, max_length=128)
        out["labels"] = ex["label"]
        return out
 
    ds = ds.map(tokenize_fn)
    args = TrainingArguments(
        output_dir="out",
        per_device_train_batch_size=8,
        num_train_epochs=5,
        learning_rate=2e-5,
        save_strategy="no",
    )
    trainer = Trainer(model=base_model, args=args, train_dataset=ds)
    trainer.train()
    return base_model

For 100-500 target-language examples, num_train_epochs=5 and learning_rate=2e-5 are safe defaults. Push the learning rate any higher and the multilingual alignment collapses, and you're left with a model that's effectively English-only.

• Per-language accuracy on held-out sets. Not aggregated. The aggregate hides the long tail.
• Benchmark against monolingual baseline. For languages with enough data, a monolingual model trained from scratch sometimes beats the multilingual one. Test.
• Entity-level tests. Named entities in the target language. Multilingual models often have weak tokenization for scripts far from Latin.
• Cross-lingual consistency. Same meaning in two languages should produce the same prediction. Measure the gap.

The 2026 stack:

Always budget for fine-tuning in the target language if performance matters. Zero-shot is a starting point, not a final answer.

The tokenization tax (what goes wrong for low-resource languages)

Multilingual models share one tokenizer across all their languages. That vocabulary is trained on a corpus dominated by English, French, Spanish, Chinese, German. For any language outside the dominant set, three taxes compound silently:

• Fertility tax. Low-resource language text tokenizes into far more tokens per word than English. A Hindi sentence can need 3-5x the tokens of an equivalent English sentence. That 3-5x eats your context window, training efficiency, and latency.
• Variant recovery tax. Every typo, diacritic variant, Unicode normalization mismatch, or case variation becomes a cold-start unrelated sequence in embedding space. The model cannot learn orthographic correspondences that a native speaker takes as obvious.
• Capacity spillover tax. Taxes 1 and 2 consume context positions, layer depth, and embedding dimensions. What remains for actual reasoning is systematically smaller than what a high-resource language gets from the same model.

The practical symptom: your model trains normally on Hindi, the loss curve looks right, eval perplexity looks reasonable, and production outputs are subtly wrong. Morphology collapses mid-sentence. Rare inflections stay unrecoverable. You cannot data-scale your way out of a broken tokenizer.

Mitigations: pick a tokenizer with good coverage for your target language (XLM-V's 1M-token vocabulary is a direct fix); verify tokenization fertility on held-out target text before training; use byte-level fallback (SentencePiece byte_fallback=True, GPT-2-style byte-level BPE) for truly long-tail scripts so nothing is ever OOV.

Save as outputs/skill-multilingual-picker.md:

---
name: multilingual-picker
description: Pick source language, target model, and evaluation plan for a multilingual NLP task.
version: 1.0.0
phase: 5
lesson: 18
tags: [nlp, multilingual, cross-lingual]
---
 
Given requirements (target languages, task type, available labeled data per language), output:
 
1. Source language for fine-tuning. Default English; check LANGRANK or qWALS if target language has a typologically close high-resource language.
2. Base model. XLM-R (classification), mT5 (generation), NLLB (translation), Aya-23 (generative LLM).
3. Few-shot budget. Start with 100-500 target-language examples if available. Zero-shot only if labeling is infeasible.
4. Evaluation plan. Per-language accuracy (not aggregate), cross-lingual consistency, entity-level F1 on non-Latin scripts.
 
Refuse to ship a multilingual model without per-language evaluation — aggregate metrics hide long-tail failures. Flag scripts with low tokenization coverage (Amharic, Tigrinya, many African languages) as needing a model with byte-fallback (SentencePiece with byte_fallback=True, or byte-level tokenizer like GPT-2).

1. Easy. Run the zero-shot classification pipeline on 10 sentences per language across English, French, Hindi, and Arabic. Report accuracy on each. You should see strong French, decent Hindi, variable Arabic.
2. Medium. Use paraphrase-multilingual-MiniLM-L12-v2 to build a cross-lingual retriever over a small mixed-language corpus. Query in English, retrieve documents in any language. Measure recall@5.
3. Hard. Compare English-source and Hindi-source fine-tuning for a Hindi classification task. Use 500 target-language examples for few-shot fine-tuning under both regimes. Report which source produces better Hindi accuracy and by how much. This is the LANGRANK thesis in miniature.

• Conneau et al. (2019). Unsupervised Cross-lingual Representation Learning at Scale — the XLM-R paper.
• Pires, Schlinger, Garrette (2019). How Multilingual is Multilingual BERT? — the analysis paper that started the cross-lingual transfer research line.
• Costa-jussà et al. (2022). No Language Left Behind — NLLB-200 paper.
• Üstün et al. (2024). Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model — Aya, Cohere's multilingual LLM.
• Language Similarity Predicts Cross-Lingual Transfer Learning Performance (2026) — the qWALS / LANGRANK source-language paper.