Speech Recognition (ASR) — CTC, RNN-T, Attention

You have a 10-second, 16 kHz clip and you want a string out of it: "turn on the kitchen lights". The problem is structural, not perceptual. Audio frames don't line up one-to-one with characters: the word "okay" might take 200 ms or 1200 ms, silence punctuates the utterance, some phonemes just run longer than others, and you don't even know the output length in advance.

Three formulations solve this:

1. CTC (Connectionist Temporal Classification). Emit per-frame token probabilities, including a special blank, then collapse repeats and blanks at decode time. Non-autoregressive and fast. Used by wav2vec 2.0, MMS.
2. RNN-T (Recurrent Neural Network Transducer). A joint network predicts the next token from the encoder frame and the token history. Streamable. Used by Google's on-device ASR, NVIDIA Parakeet.
3. Attention encoder-decoder. The encoder compresses audio into hidden states, and the decoder cross-attends to generate tokens autoregressively. Used by Whisper, SeamlessM4T.

In 2026, SOTA WER on LibriSpeech test-clean sits at 1.4% (Parakeet-TDT-1.1B, NVIDIA) and 1.58% (Whisper-Large-v3-turbo). Barely any daylight between them on accuracy — the gap that actually matters is deployment.

CTC intuition. Let the encoder output T frame-level distributions over V+1 tokens (V chars + blank). For a target string y of length U < T, any frame alignment that collapses to y counts, and CTC loss sums over all such alignments. Inference is simple: per-frame argmax, collapse repeats, remove blanks.

Advantages: non-autoregressive, streamable, zero lookahead. The drawback is the conditional independence assumption: each frame prediction is independent of the others, so there's no internal language model. You fix that with an external LM via beam search or shallow fusion.

RNN-T intuition. RNN-T adds a predictor network that embeds the token history and a joiner that combines predictor state with the encoder frame into a joint distribution over V+1 (the +1 is a null / no-emit). That explicitly models the conditional dependence CTC ignores, and it's still streamable, because each step conditions only on past frames and past tokens.

Advantages: streamable, with an internal LM built in. Drawback: training is more complex and memory-hungry (a 3D loss lattice). RNN-T loss kernels are their own whole library category for that reason alone.

Attention encoder-decoder. An encoder (6-32 transformer layers) runs over log-mel frames, and a decoder (6-32 transformer layers) cross-attends to the encoder outputs to generate tokens autoregressively. There's no alignment constraint: attention can look anywhere in the audio. That's what makes it non-streamable unless you restrict attention (chunked Whisper-Streaming, 2024).

Advantages: the highest quality on offline ASR, and it's easy to train with standard seq2seq tooling. Drawback: autoregressive latency is proportional to output length, so it can't stream without extra engineering.

WER: the one number

Word Error Rate = (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=reference word count. It's the Levenshtein edit distance at the word level, and lower is better. A WER above 20% is generally unusable; below 5% counts as human-parity for read speech. 2026 numbers on standard benchmarks:

All of these are encoder-decoder or RNN-T based. Pure CTC systems (wav2vec 2.0) sit around 1.8–2.1% on test-clean.

Step 1: greedy CTC decode

def ctc_greedy(frame_logits, blank=0, vocab=None):
    # frame_logits: list of per-frame probability vectors
    preds = [max(range(len(p)), key=lambda i: p[i]) for p in frame_logits]
    out = []
    prev = -1
    for p in preds:
        if p != prev and p != blank:
            out.append(p)
        prev = p
    return "".join(vocab[i] for i in out) if vocab else out

Two rules: collapse consecutive repeats, then drop blanks. Example: a a _ _ a b b _ c → a a b c.

Step 2: beam-search CTC

def ctc_beam(frame_logits, beam=8, blank=0):
    import math
    beams = [([], 0.0)]  # (tokens, log_prob)
    for p in frame_logits:
        log_p = [math.log(max(pi, 1e-10)) for pi in p]
        candidates = []
        for seq, lp in beams:
            for t, lpt in enumerate(log_p):
                new = seq[:] if t == blank else (seq + [t] if not seq or seq[-1] != t else seq)
                candidates.append((new, lp + lpt))
        candidates.sort(key=lambda x: -x[1])
        beams = candidates[:beam]
    return beams[0][0]

Production systems use prefix-tree beam search with LM fusion. What's above is the conceptual skeleton, not the real implementation.

Step 3: WER

def wer(ref, hyp):
    r, h = ref.split(), hyp.split()
    dp = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1):
        dp[i][0] = i
    for j in range(len(h) + 1):
        dp[0][j] = j
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            cost = 0 if r[i - 1] == h[j - 1] else 1
            dp[i][j] = min(
                dp[i - 1][j] + 1,
                dp[i][j - 1] + 1,
                dp[i - 1][j - 1] + cost,
            )
    return dp[len(r)][len(h)] / max(1, len(r))

Step 4: inference against Whisper

import whisper
model = whisper.load_model("large-v3-turbo")
result = model.transcribe("clip.wav")
print(result["text"])

That's the strongest general ASR available in 2026, in one line — it runs on a 24 GB GPU at ~20× realtime.

Step 5: streaming with Parakeet or wav2vec 2.0

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="nvidia/parakeet-tdt-1.1b")
for chunk in streaming_audio():
    print(asr(chunk, return_timestamps=True))

Streaming ASR needs chunked encoder attention and carryover state, so use a library that already handles it: NeMo for Parakeet, or the transformers pipeline with chunk_length_s.

The 2026 stack:

• No VAD. Running Whisper on silence produces hallucinations ("Thanks for watching!"). Gate it with VAD, always.
• Character vs word vs subword WER. Report word-level WER after normalization: lowercase, punctuation stripped.
• Language ID drift. Whisper's auto LID mis-routes noisy clips to Japanese or Welsh. Force language="en" whenever you already know the language.
• Long clips without chunking. Whisper has a 30-second window. Use chunk_length_s=30, stride=5 for anything longer.

Save it as outputs/skill-asr-picker.md: pick model, decoding strategy, chunking, and LM fusion for whatever deployment target you're building for.

1. Easy. Run code/main.py. It greedily decodes a hand-crafted CTC output and computes WER against a reference.
2. Medium. Implement the prefix-tree beam search in Step 2 properly (account for the blank merge rule). Compare with greedy on a 10-example synthetic dataset.
3. Hard. Use whisper-large-v3-turbo on LibriSpeech test-clean. Compute WER on the first 100 utterances. Compare with published numbers.

• Graves et al. (2006). Connectionist Temporal Classification: the CTC paper.
• Graves (2012). Sequence Transduction with RNNs: the RNN-T paper.
• Radford et al. / OpenAI (2022). Whisper: Robust Speech Recognition via Large-Scale Weak Supervision: the 2022 canonical paper, extended by v3-turbo in 2024.
• NVIDIA NeMo: Parakeet-TDT card: 2026 Open ASR Leaderboard leader.
• Hugging Face: Open ASR Leaderboard: live benchmark across 25+ models.