Whisper — Architecture & Fine-Tuning

Whisper, released by OpenAI in September 2022, was the first ASR model to ship as a commodity: paste in audio, get text back, 99 languages, robust to noise, runs on a laptop. OpenAI shipped Large-v3 and Turbo variants by 2024, and by 2026 Whisper is the default baseline for everything from podcast transcription to voice assistants to YouTube subtitles.

Treat it as a black box forever, though, and it will fail you. Domain shift kills it: technical jargon, speaker accents, proper nouns, short clips, silence. You need to know:

1. What it actually is inside.
2. How to feed it chunked, streaming, or long-form audio correctly.
3. When to fine-tune it, and how.

Architecture. Standard transformer encoder-decoder.

• Input: 30-second log-mel spectrogram, 80 mels, 10 ms hop → 3000 frames. Clips shorter are zero-padded, clips longer are chunked.
• Encoder: conv-downsample (stride 2) + N transformer blocks. For Large-v3: 32 layers, 1280-dim, 20 heads.
• Decoder: N transformer blocks with causal self-attn + cross-attn to encoder output, the same size as the encoder.
• Output: BPE tokens over a 51,865-token vocab.

Large-v3 has 1.55B params. Turbo drops the decoder from 32 layers to 4, cutting latency 8× for a WER hit under 1%.

The prompt format. Whisper is a multitask model steered by special tokens in the decoder prompt:

<|startoftranscript|><|en|><|transcribe|><|notimestamps|> Hello world.<|endoftext|>

• <|en|>: the language tag, and it decides translate-vs-transcribe behavior.
• <|transcribe|> or <|translate|>: verbatim output in the source language, or an English translation from any source language.
• <|notimestamps|>: skip word-level timestamps for faster decoding.

The prompt is what lets one model do many tasks. Change <|en|> to <|fr|> and it transcribes French instead.

30-second window. Everything is pinned to 30 seconds: longer clips need chunking, shorter clips get padded. Windows aren't streamed natively, which is why WhisperX, Whisper-Streaming, and faster-whisper exist.

Log-mel normalization. (log_mel - mean) / std, where the stats come from Whisper's own training corpus. You must use Whisper's preprocessing (whisper.audio.log_mel_spectrogram), not librosa.feature.melspectrogram.

Variants in 2026

Fine-tuning

Canonical workflow in 2026:

1. Collect 10–100 hours of target-domain audio with aligned transcripts.
2. Run transformers.Seq2SeqTrainer with a generate_with_loss callback.
3. Go parameter-efficient: LoRA (low-rank adapter matrices trained in place of the full attention weights) on q_proj and v_proj cuts GPU memory 4× for a WER cost under 0.3.
4. Freeze the encoder if you have fewer than 10 hours of data. Only tune the decoder.
5. Use Whisper's own tokenizer and prompt format. Never swap tokenizers.

Community results back this up: fine-tuning Medium on 20 hours of medical dictation drops WER from 12% to 4.5% on medical vocabulary. Fine-tuning Turbo on 4 hours of Icelandic drops WER from 18% to 6%.

Step 1: run Whisper out of the box

Three of Whisper's defaults are worth overriding on every call. Pin temperature=0.0, because left alone, sampling falls back through 0.0 → 0.2 → 0.4 … the moment confidence dips. Set condition_on_previous_text=False, because left on, a single hallucinated window seeds every window after it. Set no_speech_threshold=0.6 explicitly for cleaner silence detection:

import whisper
model = whisper.load_model("large-v3-turbo")
result = model.transcribe(
    "clip.wav",
    language="en",
    task="transcribe",
    temperature=0.0,
    condition_on_previous_text=False,  # prevents runaway repetition
    no_speech_threshold=0.6,  # silence detection
)
print(result["text"])
for seg in result["segments"]:
    print(f"[{seg['start']:.2f}–{seg['end']:.2f}] {seg['text']}")

Step 2: chunked long-form

# whisperx is the 2026 reference for long-form with word-level timestamps
import whisperx
model = whisperx.load_model("large-v3-turbo", device="cuda", compute_type="float16")
segments = model.transcribe("1hour.mp3", batch_size=16, chunk_size=30)

WhisperX adds three things on top of vanilla Whisper: Silero VAD gating, word-level alignment via wav2vec 2.0, and diarization via pyannote.audio. It's the 2026 workhorse for production transcription.

Step 3: fine-tune with LoRA

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import LoraConfig, get_peft_model
 
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3-turbo")
lora = LoraConfig(
    r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1, bias="none", task_type="SEQ_2_SEQ_LM",
)
model = get_peft_model(model, lora)
# model.print_trainable_parameters()  -> ~3M trainable / 809M total

From there it's a standard Trainer loop: checkpoint every 1000 steps and evaluate WER on a held-out set.

Step 4: inspect what each layer learns

# Grab cross-attention weights during decode to see what the decoder attends to.
with torch.inference_mode():
    out = model.generate(
        input_features=features,
        return_dict_in_generate=True,
        output_attentions=True,
    )
# out.cross_attentions: layer × head × step × src_len

Visualize it as a heatmap and you'll see diagonal alignment: decoder steps scanning through encoder frames in order. That diagonal is Whisper's notion of word timestamps.

The 2026 stack:

faster-whisper (CTranslate2 backend) is the fastest CPU+GPU inference runtime in 2026, running 4× faster than vanilla with identical output.

• Hallucinated text on silence. Whisper trained on captions that include "Thanks for watching!", "Subscribe!", and song lyrics, so it reproduces them out of noise. Always VAD-gate before you call it.
• condition_on_previous_text cascade. One hallucination pollutes every window after it. Leave this False unless you specifically need fluency across chunks.
• Short-clip padding. A 2-second clip padded to 30 seconds can hallucinate into the trailing silence. Use pad=False or VAD-gate instead.
• Wrong mel stats. Using librosa's mels instead of Whisper's own produces near-random output. Use whisper.audio.log_mel_spectrogram.

outputs/skill-whisper-tuner.md is a reusable skill for designing a Whisper fine-tune or inference pipeline for a given domain.

1. Easy. Run code/main.py. It tokenizes a Whisper-style prompt, computes decoded shape budgets, and prints the chunk schedule for a 10-minute clip.
2. Medium. Install faster-whisper, transcribe a 10-minute podcast, compare WER against a human transcript. Try language="auto" vs forced language="en".
3. Hard. Using HF datasets, pick a language Whisper struggles with (e.g., Urdu), fine-tune Medium with LoRA for 2 epochs on 2 hours, and report WER delta.

• Radford et al. (2022). Whisper paper: the original architecture and training recipe.
• OpenAI (2024). Whisper Large-v3-turbo release: the 4-layer decoder, 8× speedup.
• Bain et al. (2023). WhisperX: long-form, word-aligned, diarized transcription.
• SYSTRAN, faster-whisper repo: CTranslate2-backed, 4× faster.
• HuggingFace, Whisper fine-tune tutorial: the canonical LoRA / full-FT walkthrough.