T5, BART — Encoder-Decoder Models

An encoder-decoder model is a transformer built from two stacks. A bidirectional encoder reads the whole input and produces a dense representation of it. A causal decoder generates the output one token at a time, cross-attending to that representation as it goes. Training is shift-by-one on the output side: the same loss as GPT, conditioned on the encoder output.

GPT and BERT each strip the 2017 architecture down for a different goal. Plenty of tasks are naturally input-to-output, though:

• Translation: English → French.
• Summarization: 5,000-token article → 200-token summary.
• Speech recognition: audio tokens → text tokens.
• Structured extraction: prose → JSON.

For these, encoder-decoder is the cleanest fit. The source gets its own stack, and the output gets its own.

Two papers set the modern playbook:

1. T5 (Raffel et al. 2019), the "Text-to-Text Transfer Transformer." NLP tasks are reframed as text in, text out, with a single architecture, a single vocabulary, and a single loss. Pretraining is masked span prediction: corrupt spans in the input, decode them in the output.
2. BART (Lewis et al. 2019), the "Bidirectional and Auto-Regressive Transformer." A denoising autoencoder: corrupt the input in several ways (shuffle, mask, delete, rotate) and ask the decoder to reconstruct the original.

In 2026 the format survives where input structure matters:

• Whisper (speech → text).
• Google's translation stack.
• Some code-completion and repair models that have distinct context-and-edit structures.
• Flan-T5 and variants for structured reasoning tasks.

Decoder-only took the spotlight, but encoder-decoder stayed in use.

The forward loop

source tokens ─▶ encoder ─▶ (N_src, d_model)  ──┐
                                                 │
target tokens ─▶ decoder block                   │
                 ├─▶ masked self-attention       │
                 ├─▶ cross-attention ◀───────────┘
                 └─▶ FFN
                ↓
              next-token logits

The encoder runs once per input. The decoder runs autoregressively but cross-attends to the same encoder output at each step, so you can cache that output. For long inputs it's a free speedup.

T5 pretraining: span corruption

Span corruption is T5's pretraining objective. Pick random spans of the input and replace each with a unique sentinel token, then train the decoder to output the missing spans. The spans average 3 tokens and cover 15% of the input in total (the T5 paper's rate). The sentinels are <extra_id_0>, <extra_id_1>, and so on. The decoder outputs only the corrupted spans, each with its sentinel prefix:

source: The quick <extra_id_0> fox jumps <extra_id_1> dog
target: <extra_id_0> brown <extra_id_1> over the lazy

That's a cheaper signal than predicting the whole sequence, and it was competitive with MLM (BERT) and prefix-LM (UniLM) in the T5 paper's ablation.

BART pretraining: multi-noise denoising

Denoising is BART's objective. Apply a noise function to the input, then train the decoder to reconstruct the original text. BART tries five noising functions:

1. Token masking.
2. Token deletion.
3. Text infilling (mask a span, decoder inserts the right length).
4. Sentence permutation.
5. Document rotation.

Text infilling combined with sentence permutation gave the best downstream numbers. The decoder's target is the full original sequence, not just the corrupted spans, so pretraining compute is higher than T5's.

Inference

Generation works the same way as in GPT, and greedy, beam, and top-p sampling all apply. Beam search (width 4–5) is standard for translation and summarization because the output distribution is narrower than chat.

When to pick each variant in 2026

Since about 2022, decoder-only has taken over tasks that encoder-decoder used to own, for three reasons: (a) instruction-tuned decoder-only LLMs generalize to new tasks through prompting, (b) one architecture scales more easily than two, and (c) RLHF assumes a decoder. Encoder-decoder holds on where the input modality differs (speech, images) or where beam search quality matters.

See code/main.py. It implements T5-style span corruption on a toy corpus. This is the piece of the lesson worth building yourself, because span corruption is the core of T5's pretraining.

Step 1: span corruption

def corrupt_spans(tokens, mask_rate=0.15, mean_span=3.0, rng=None):
    """Pick spans summing to ~mask_rate of tokens. Return (corrupted_input, target)."""
    n = len(tokens)
    n_mask = max(1, int(round(n * mask_rate)))
    n_spans = max(1, int(round(n_mask / mean_span)))
    ...

The default mask_rate is the paper's 0.15, but the demo in main() passes 0.20. The target format is the T5 convention: each <extra_id_N> sentinel followed by its span, then one closing sentinel after the last span (<extra_id_0> span0 <extra_id_1> span1 <extra_id_2>). The Concept example leaves that closing sentinel out for brevity. The corrupted input interleaves unchanged tokens with the same sentinels at the span locations.

Step 2: verify round-trip

Given the corrupted input and target, reconstruct the original sentence. If the corruption is reversible, the forward pass is well-defined. This is a sanity check. Real training doesn't do it, but it's cheap and catches off-by-one bugs in your span bookkeeping.

Step 3: BART noising

Five functions: token_mask, token_delete, text_infill, sentence_permute, document_rotate. main() demonstrates each one on its own, so run it and compare each output against the original.

HuggingFace reference:

from transformers import T5ForConditionalGeneration, T5Tokenizer
tok = T5Tokenizer.from_pretrained("google/flan-t5-base")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base")
 
inputs = tok("translate English to French: Attention is all you need.", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))

The T5 trick is to put the task name in the input text. One model handles dozens of tasks because each task is text in, text out. Instruction-tuned decoder-only models have since generalized the pattern, but T5 codified it first.

See outputs/skill-seq2seq-picker.md. The skill picks between encoder-decoder and decoder-only for a new task, given its input-output structure, latency, and quality targets.

1. Easy. Run code/main.py, read the span-corruption output for its 25-token demo sentence, and verify that concatenating the non-sentinel source tokens with the decoded target spans reproduces the original.
2. Medium. main.py already implements BART's text_infill, which replaces each span with a single <mask> token and draws span lengths from a Gaussian. Extend it to draw lengths from a Poisson distribution with mean 3, as the BART paper does, and show one example.
3. Hard. Fine-tune flan-t5-small on a tiny English → pig-Latin corpus (200 pairs). Measure BLEU on a held-out 50-pair set. Compare against fine-tuning Llama-3.2-1B on the same data with the same compute.

• Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer: T5.
• Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension: BART.
• Chung et al. (2022). Scaling Instruction-Finetuned Language Models: Flan-T5.
• Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision: Whisper, the canonical 2026 encoder-decoder.
• HuggingFace modeling_t5.py: reference implementation.