Audio Transformers — Whisper Architecture

Automatic speech recognition (ASR) is the task of turning audio into text. Before Whisper (OpenAI, Radford et al. 2022), state-of-the-art ASR meant wav2vec 2.0 and HuBERT: self-supervised feature extractors plus a fine-tuned head. Quality was high, but the data pipelines were expensive and the models were domain-brittle. Multilingual recognition needed separate models per language family.

Whisper made three bets:

1. Train on everything. 680,000 hours of weakly-labeled audio scraped from the internet across 97 languages. No clean academic corpus, no phoneme labels.
2. Multi-task single model. One decoder trained jointly on transcription, translation, voice activity detection, language ID, and timestamping via task tokens.
3. Standard encoder-decoder transformer. The encoder consumes log-mel spectrograms and the decoder produces text tokens autoregressively. No vocoder, no CTC, no HMM.

The result is that Whisper large-v3 holds up across accents, noise, and languages with no clean labeled data. In 2026 it is the default speech front-end for open-source voice assistants and many commercial ones.

Whisper is an encoder-decoder transformer that reads a 30-second log-mel spectrogram and writes out text tokens. The pipeline runs in six steps.

Step 1: resample + window

Audio is resampled to 16 kHz and clipped or padded to 30 seconds. The log-mel spectrogram has 80 mel bins at a 10 ms stride, which gives ~3,000 frames × 80 features. That is the "input image" Whisper sees.

Step 2: convolutional stem

Two Conv1D layers with kernel 3 and stride 2 reduce the 3,000 frames to 1,500. That halves the sequence length without adding many parameters.

Step 3: encoder

A 32-layer (for large) transformer encoder runs over the 1,500 timesteps, with sinusoidal positional encoding, self-attention, and a GELU FFN. It produces 1,500 × 1,280 hidden states.

Step 4: decoder

A 32-layer (for large) transformer decoder produces tokens autoregressively from a BPE vocabulary that is a superset of GPT-2's plus a few audio-specific special tokens.

Step 5: task tokens

The decoder prompt starts with control tokens that tell the model what to do:

<|startoftranscript|>  <|en|>  <|transcribe|>

or

<|startoftranscript|>  <|fr|>  <|translate|>

The model was trained on this convention, so you pick the task with the prefix. With timestamps on, the model itself then emits <|0.00|> as its first output token; to turn timestamps off you append <|notimestamps|> to the prefix. It is the 2026 equivalent of instruction-tuning, applied to speech.

Step 6: output

Decoding uses beam search (width 5) with a log-prob threshold. When the <|notimestamps|> token is absent, timestamps are predicted every 0.02 seconds of audio.

Whisper sizes

Large-v3-turbo (2024) cut the decoder from 32 layers to 4, which gives 8× faster decoding with <1 WER point regression. That decode speed is why Whisper-turbo is the default for real-time voice agents in 2026.

What Whisper does not do

• No diarization (who is speaking). Pair it with pyannote for that.
• No native real-time streaming, because the 30-second window is fixed. Wrappers such as faster-whisper and WhisperX add streaming through VAD and overlap.
• No long-form context beyond 30 s without external chunking. That works well in practice because human speech rarely needs long-range context for transcription.

2026 landscape

See code/main.py. It doesn't train or run Whisper. It builds a simplified version of the preprocessing (framing, a per-frame log-energy stand-in for the mel features, padding to 3,000 frames) plus the task-token prompt formatter, the parts you touch in production. It has no mel filterbank or FFT, does not build the 30-second waveform clip (it only pads or clips the list of frames), and implements no convolutional stem, encoder, decoder, beam search, or timestamp prediction.

Step 1: synthesize audio

Generate a 1-second sine wave at 440 Hz sampled at 16 kHz, which is 16,000 samples.

Step 2: framing and log-energy (simplified mel stand-in)

A full log-mel spectrogram needs an FFT and a mel filterbank. This version does framing plus per-frame log sum-of-squares energy, which shows the pipeline's shape without requiring librosa:

def frame_signal(x, frame_size=400, hop=160):
    frames = []
    for start in range(0, len(x) - frame_size + 1, hop):
        frames.append(x[start:start + frame_size])
    return frames

A frame is 25 ms and the hop is 10 ms, which matches Whisper's windowing. Per-frame log-energy stands in for mel bins: one number per frame instead of Whisper's 80. For the 1-second signal this gives 98 frames, not 100, because the last frame must fit entirely inside the signal: (16000 - 400) // 160 + 1 = 98.

Step 3: pad to 30 s

Whisper processes 30-second chunks, so pad_or_clip pads (or clips) the list of frames to 3,000 with all-zero frames. It works on frames, not on the waveform.

Step 4: build the prompt tokens

def whisper_prompt(lang="en", task="transcribe", timestamps=True):
    tokens = ["<|startoftranscript|>", f"<|{lang}|>", f"<|{task}|>"]
    if not timestamps:
        tokens.append("<|notimestamps|>")
    return tokens

That function is the whole task-control surface. By default it returns a 3-token prefix (start token, language, task); with timestamps=False it appends <|notimestamps|> for a 4-token prefix.

Step 5: the size table

main.py also prints a size table in large-v3 geometry. It lists large-v3 and turbo but not the plain Large row from the table above. The turbo row shows 32 layers (the encoder depth); the 4-layer decoder is only mentioned in the line printed after the table.

import whisper
model = whisper.load_model("large-v3-turbo")
result = model.transcribe("meeting.wav", language="en", task="transcribe")
print(result["text"])
print(result["segments"][0]["start"], result["segments"][0]["end"])

Faster, OpenAI-compatible:

from faster_whisper import WhisperModel
model = WhisperModel("large-v3-turbo", compute_type="int8_float16")
segments, info = model.transcribe("meeting.wav", vad_filter=True)
for s in segments:
    print(f"{s.start:.2f} - {s.end:.2f}: {s.text}")

When to pick Whisper in 2026:

• Multilingual ASR with one model.
• Robust transcription of noisy, diverse audio.
• Research or prototype ASR, where it is the fastest starting point.

When to pick something else:

• Ultra-low latency streaming on edge: Moonshine beats Whisper at matched quality.
• Real-time conversational AI needing <200 ms: a dedicated streaming ASR.
• Speaker diarization: Whisper does not do this, so bolt on pyannote.

The skill in outputs/skill-asr-configurator.md picks an ASR model, decoding parameters, and preprocessing pipeline for a new speech application.

1. Easy. Run code/main.py. Confirm the frame count for a 1-second signal at 16 kHz with a 400-sample window and 160-sample (10 ms) hop. The formula (n - 400) // 160 + 1 gives 98 frames, a little under the ~100 you'd get from 1 s / 10 ms. For 30 seconds the same formula gives 2,998, which is why padding to 3,000 matters.
2. Medium. Build the full log-mel spectrogram using numpy.fft. Verify 80 mel bins match librosa.feature.melspectrogram(n_mels=80) within numerical error.
3. Hard. Implement streaming inference: chunk audio into 10 s windows with 2 s overlap, run Whisper on each chunk, merge transcripts. Measure word-error rate vs single-pass on a 5-minute podcast sample.

• Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision: Whisper paper.
• OpenAI Whisper repo: reference code + model weights. Read whisper/model.py to see the Conv1D stem + encoder + decoder top-to-bottom in ~400 lines.
• OpenAI Whisper: whisper/decoding.py: the beam-search + task-token logic described in Steps 5–6 is here; 500 lines, fully readable.
• Baevski et al. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations: precursor; still SOTA features in some settings.
• SYSTRAN/faster-whisper: production wrapper, 4× faster than reference.
• Jia et al. (2024). Moonshine: Speech Recognition for Live Transcription and Voice Commands: 2024 edge-friendly ASR, Whisper-shaped but smaller.
• HuggingFace blog: "Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers": canonical fine-tuning recipe including mel spectrogram preprocessor and token-timestamp handling.
• HuggingFace modeling_whisper.py: full implementation (encoder, decoder, cross-attention, generation) that mirrors the architecture diagram in The Concept.