BuildPython

Audio-Language Models: the Whisper to Audio Flamingo 3 Arc

Whisper (Radford et al., December 2022) settled speech recognition with 680k hours of weakly-supervised multilingual speech, a simple encoder-decoder transformer, and a benchmark that later ASR releases kept citing. But recognition is not reasoning. Asking "what instruments are in this recording" or "what emotion is the speaker expressing" or "what happened at minute 3" requires audio understanding, not transcription. Qwen-Audio, SALMONN, LTU, and NVIDIA's Audio Flamingo 3 (AF3, July 2025) built that stack step by step: keep Whisper-class encoders, bolt on Q-formers, train on audio-text instruction data, add chain-of-thought reasoning. This lesson walks the arc.

TIME: ~180 minutesPREREQUISITES: Phase 6 (Speech and Audio), Phase 12 · 03 (Q-Former)

Learning Objectives

Compute a log-Mel spectrogram from a waveform: windowing, FFT, filter banks, log transform.
Compare encoder options: Whisper encoder, BEATs, AF-Whisper hybrid. When each wins.
Build an audio Q-former: N learnable queries cross-attending to spectrogram patches.
Explain cascaded (Whisper-then-LLM) vs end-to-end audio-LLM training: why end-to-end scales better for reasoning.

The Problem

Whisper handled speech recognition, and transcription is now a commodity. That's where the commodity ends. If the model can't reason over what it heard (timing, speakers, emotion, music structure, environmental sounds), a transcript alone can't drive product features.

Three routes:

Cascade: Whisper transcribes, an LLM reasons over the transcript. Good for pure-speech scenarios. Struggles with music, environmental audio, overlapping speakers, and emotion.
End-to-end audio-LLM: an audio encoder feeds audio tokens straight into an LLM and skips transcription. This keeps the acoustic information (emotion, speaker, environment) but needs new training data.
Hybrid: an audio encoder plus a text decoder that can both transcribe and reason. Qwen-Audio and Audio Flamingo take this route.

Warm-up0/3

1. In a BLIP-2-style Q-former (Phase 12 · 03), what do the learnable queries do?Warm-up

A BLIP-2-style pipeline: an image goes to an image encoder that produces a grid of encoder features, then to a Q-former box with four small learnable query chips q1 to q4, then to an LLM. A dashed box asks what the learnable queries do.

2. In speech processing (Phase 6), why is the frequency axis warped to the Mel scale instead of keeping raw FFT bins?Warm-up

Typed schematic; 33 ticks are illustrative.

3. A short slice of audio is passed through a Fourier transform. What does the result describe?Warm-up

Waveform plotted from the real first frame of main.py's synthetic clip.

The Concept

Log-Mel spectrogram: the input feature

Most audio encoders start from the same feature, a log-Mel spectrogram.

Resample to 16 kHz.
Short-time Fourier transform with 25ms windows, 10ms hop.
Take magnitude of the FFT result.
Apply Mel filter banks (typically 80 filters log-spaced 0-8000 Hz) to warp to perceptual frequency.
Log compress (log(1 + x)) for dynamic range.

The result is a 2D array of shape (T, 80), where T is the number of time frames. A 30-second clip at a 100 Hz frame rate gives (3000, 80).

Whisper's encoder

Whisper's encoder is a 12-layer ViT-style transformer that reads the log-Mel spectrogram as a sequence of time frames and outputs one hidden-state vector per frame.

For ASR, the decoder is a cross-attention transformer that generates text tokens conditioned on the encoder output. Standard encoder-decoder.

For ALMs (audio-LLMs) you want the encoder output to feed a different LLM. The pattern is a frozen Whisper encoder, a trainable Q-former, and an LLM that is either frozen or tuned.

BEATs and audio-specific encoders

Whisper was trained on speech-dominant data, so it is weaker on music and environmental audio.

BEATs (Chen et al., 2022) is a self-supervised transformer trained on AudioSet. At the same parameter count it captures music and environmental sounds better than Whisper.

AF-Whisper, the hybrid in Audio Flamingo 3, concatenates Whisper and BEATs features as the audio input. Whisper carries the linguistic signal and BEATs carries the acoustic one.

Audio Q-former

Same pattern as BLIP-2's visual Q-former. A fixed number of learnable queries (often 32 or 64) cross-attend over the audio encoder's output frames, and the queries become the audio tokens the LLM consumes.

Training happens in two stages. In the alignment stage only the Q-former trains, with contrastive and captioning losses on audio-text pairs (AudioCaps, Clotho). In the instruction stage you train end-to-end, unfreeze the LLM, and use instruction data.

The arc: SALMONN, Qwen-Audio, AF3

SALMONN (Tang et al., 2023) is Whisper + BEATs + Q-former + LLaMA, the first open audio-LLM with serious reasoning ability. Benchmarks on MMAU show ~0.55 composite.

Qwen-Audio (Chu et al., 2023) has a similar architecture, a richer training set, and tuning for multi-turn dialogue. MMAU ~0.60.

LTU (Listen, Think, Understand; Gong et al., 2023) uses explicit reasoning data and focuses on chain-of-thought over audio clips. Smaller, more focused.

Audio Flamingo 3 (Goel et al., July 2025) is the current open SOTA. It has an 8B LLM backbone (Qwen2 7B), a Whisper-large encoder concatenated with BEATs, a 64-query Q-former, and training on 1M+ audio-text instruction pairs. MMAU 0.72, and it matches the proprietary frontier on some sub-tasks.

AF3 also adds on-demand chain-of-thought for audio. The model can optionally emit thinking tokens ("let me identify the instruments first: ...") before the final answer. Accuracy on complex reasoning tasks goes up 3-5 points when thinking is enabled.

Cascaded vs end-to-end

The cascaded pipeline has two steps:

Whisper transcribes audio to text.
An LLM reasons over the text.

That works well for "summarize this podcast." It falls short on:

"What's the mood of this song?" The mood is in the sound, not the words.
"Who is speaking, Alice or Bob?" This needs speaker identification.
"At what second does the explosion happen?" Temporal grounding is lost in the text.
"Is this real or generated audio?" Deepfake detection needs acoustic features.

End-to-end keeps the acoustic signal. Qwen-Audio and AF3 handle music, environment, and emotion natively.

2026 production recipe

For a new audio-understanding product:

Go cascaded if transcription is the goal, with no music and no emotion inference.
Go with AF3 or the Qwen-Audio family if you need music, emotion, multi-speaker handling, or complex audio reasoning.

Cascaded is cheaper and simpler. End-to-end is more capable.

MMAU: the audio reasoning benchmark

MMAU (Massive Multimodal Audio Understanding) is the 2024-2025 audio reasoning benchmark:

10,000 audio-text QA pairs across speech, music, environmental sounds.
Covers classification, temporal reasoning, causal reasoning, open-ended QA.
Tests what cascaded pipelines tend to miss.

Open SOTA (AF3) is at 0.72, and the proprietary frontier is ~0.78 (Gemini 2.5 Pro, Claude Opus 4.7). The gap is smaller than VideoMME's open-vs-closed delta, which suggests audio-LLMs are maturing.

Check: The Concept0/25

1. What is a log-Mel spectrogram?Define

A heatmap computed from main.py's synthetic clip: brightness varies from cell to cell. Two horizontal bright lines run across the whole width. — Real computation: main.py's own functions on every frame; brighter is larger.

2. How does Whisper's encoder process its input, per Whisper's encoder in The Concept?Define

A pipeline: audio, an input block, a dashed encoder box, a decoder box, then text. A dashed box below asks how the encoder processes its input. — Typed schematic. The decoder is the cross-attention half.

3. What is BEATs?Define

A card naming BEATs, Chen et al., 2022, beside a list of places the lesson mentions it: Whisper, SALMONN and AF-Whisper from Audio Flamingo 3. A dashed box asks what BEATs is.

4. What is an audio Q-former?Define

A pipeline of four boxes: an audio clip, an audio encoder, an audio Q-former, and an LLM. A dashed box asks what the audio Q-former does. — Typed schematic.

5. What does cascaded mean for an audio-understanding pipeline?Define

An audio clip box, a dashed box labelled cascaded pipeline with a question mark, and an answer box joined by arrows. — Typed schematic. The dashed box stands for the inside of the pipeline.

6. What is an end-to-end audio-LLM?Define

An audio clip box, a dashed box labelled end-to-end audio-LLM with a question mark, and an answer box joined by arrows, below a line naming Qwen-Audio and Audio Flamingo 3 as example systems. — Typed schematic. The dashed box stands for the inside of the model.

7. What does AF3's on-demand thinking do, and with what effect?Define

Audio Flamingo 3 as a box that takes an audio clip and a question and produces an answer, with an off and on switch labelled on-demand thinking, and a dashed box asking what thinking does and its effect on accuracy. — Typed schematic.

8. What is MMAU?Define

A card naming MMAU, Massive Multimodal Audio Understanding, 2024-2025, beside a list of models the lesson mentions with it: SALMONN, Qwen-Audio, AF3 and Gemini 2.5 Pro. A dashed box asks what MMAU is.

9. Which sequence matches the log-Mel spectrogram steps in The Concept?Define

A raw waveform on the left, a dashed box labelled the steps in order with a question mark in the middle, and a log-Mel heatmap on the right, joined by arrows. — Waveform and heatmap are real output of main.py's functions.

10. Which ordering of MMAU composite scores matches The arc: SALMONN, Qwen-Audio, AF3?Define

Score boxes are placeholders, not drawn to scale.

11. Why does the log-Mel pipeline finish with a log(1 + x) compression?Explain

Real values from main.py; every band not labelled is near zero.

12. Why does AF-Whisper pair BEATs with Whisper instead of using Whisper alone?Explain

A diagram of Audio Flamingo 3's audio input: an audio clip fans out to a Whisper box and a BEATs box, their features are concatenated, then pass to a Q-former and LLM. A dashed box asks why BEATs is added to Whisper. — Typed schematic.

13. In the alignment stage of Audio Q-former training, what is trained and on what?Explain

A pipeline for stage 1, alignment: audio, audio encoder, Q-former, LLM, text, with a dashed box asking what is trained and on what data. — Typed schematic.

14. Why does an audio Q-former hand the LLM the same number of audio tokens for a short clip and a long one?Explain

Two rows, a short clip and a long clip. Each clip is read as a row of encoder frames, the long clip having more frames, and passes through the same Q-former to a dashed box labelled tokens with a question mark. — Illustrative frame counts, typed.

15. Why can an end-to-end audio-LLM answer 'What is the mood of this song?' when a cascaded pipeline struggles?Explain

A song clip with the typed question what is the mood of this song, then two rows: a cascaded pipeline that struggles and an end-to-end audio-LLM that can answer, with a dashed box asking why. — Typed schematic.

16. The Cascaded vs end-to-end subsection lists tasks where a Whisper-then-LLM pipeline falls short. Which task is NOT one of them?Explain

A cascaded pipeline: audio clip, Whisper, transcript, text LLM, answer, with a dashed box asking for the tasks where it falls short. — Typed schematic.

17. Why is a cascaded pipeline still the right pick for 'summarize this podcast'?Explain

A podcast episode goes through Whisper and a text LLM to a summary, with a dashed box asking why this pipeline is still the right pick. — Typed schematic.

18. A 30-second clip at a 100 Hz frame rate with 80 Mel bins gives what spectrogram shape?Apply

A 30 second clip drawn as a bar with a tick every half second, two chips reading frame rate 100 Hz and 80 Mel bins, and a dashed box asking for the spectrogram shape.

19. Using the same settings (16 kHz, 25ms window, 10ms hop, 80 Mel bins), what shape does a 10-second clip give?Apply

A staircase of four overlapping window bars, each starting a little after the last, chips for a 10 second clip, 16 kHz, 25 ms window, 10 ms hop and 80 Mel bins, and a dashed box asking for the spectrogram shape. — Window staircase is a schematic; not drawn to scale.

20. A support-call tool built from Whisper plus an LLM summarizes calls well, but it never notices that a caller sounds angry when the words are polite. What causes this?Analyze

A support-call tool: call audio, labelled as a caller who sounds angry, passes through Whisper and an LLM to a call summary with no anger flagged. A typed example of the caller's polite words sits below, and a dashed box asks why the anger is never noticed. — Typed schematic.

21. A team replaces the BEATs half of an AF-Whisper-style encoder with a second Whisper copy. Transcription stays fine but instrument recognition collapses. Why?Analyze

An AF-Whisper-style encoder where the BEATs half is replaced by a second Whisper copy: audio fans out to two Whisper boxes, their features are concatenated and passed to a Q-former and LLM. Text says transcription stays fine and instrument recognition collapses, and a dashed box asks why. — Typed schematic.

22. A team moves a podcast-summary feature from Whisper plus an LLM to an end-to-end audio-LLM. Quality is unchanged and cost goes up. Which assumption was wrong?Analyze

Typed schematic of the situation in the question.

23. A security product must label non-speech sounds in recordings (glass breaking, sirens, engines) before an LLM reasons over them. Which encoder input fits best?Design

Typed schematic.

24. You are building a tool that must tell which of several speakers sounds frustrated in call recordings. Which design fits?Design

Three speaker lanes (A, B, C) of call-recording waveform, a note that the goal is to tell which speaker sounds frustrated, and a dashed box asking for the design that fits. — Waveforms are drawn at random for illustration.

25. A team is building a tool that turns English sales calls into searchable transcripts and written summaries, with no emotion or music analysis. Which pipeline fits, per the 2026 production recipe?Design

English sales calls on the left, two outputs (searchable transcripts and written summaries), two constraints (no emotion analysis, no music analysis), and a dashed box asking for the pipeline that fits the 2026 recipe. — Typed schematic.

Use It

main.py:

Computes a log-Mel spectrogram in stdlib: windowing, naive DFT, and a simplified Mel filter bank.
Runs an audio Q-former skeleton: given encoder output frames, compute Q, K, V, attention, and emit N tokens.
Prints a cascaded-vs-end-to-end comparison table for a set of audio tasks.

Check: Use It0/6

1. In main.py, window_frames runs on a 1-second clip at 16 kHz with a 25ms window and a 10ms hop. How many frames does it return?Apply

A 1 second clip drawn as a bar, with two overlapping 25 ms window bars below it, the second starting 10 ms after the first, and a dashed box asking how many frames window_frames returns. — Window sizes are a schematic; not drawn to scale.

2. In the Q-former demo, 98 frames of dimension 20 go in with n_queries=8. If you change it to n_queries=64, what changes in the output?Apply

A terminal-style panel showing the demo output for n_queries equal to 8: 8 output tokens of dim 20 with the frame count line left out, and below it a panel for n_queries equal to 64 whose output is a question mark. — First box: real output line, label added. The second is not shown here.

3. In the toy QFormer.forward in main.py, how is one output token computed?Apply

A query chip, a column of frames f1, f2, f3 up to fN, and a dashed box asking how one token is computed, leading to a single output token chip. — Typed schematic; N and d are symbols.

4. In the task table printed by main.py, which tasks does the cascaded pipeline handle only partially?Apply

A table of ten audio tasks from main.py, with the cascaded column hidden behind question marks. — Task names are from main.py's table; the cascaded column is masked.

5. A reader runs main.py, sees the Q-former attention weights, and says token 3 learned to focus on the loud frames. What is wrong with that conclusion?Analyze

A bar chart of attention weights for token 3 across the input frames, with bars of uneven height, and a quoted claim from a reader that token 3 learned to focus on the loud frames. — Weights replayed with the demo's seed; main.py does not print them.

6. A long recording is overflowing the LLM's context budget because of the audio tokens. Which change to the Q-former fits, and what does it cost?Design

A long recording passes through an encoder and a Q-former to an LLM, below a bar for the LLM context budget where a prompt segment and a long audio-tokens segment run past a dashed budget limit, and a dashed box asks for the change to the Q-former and its cost. — Bar widths are a schematic; no real token counts.

Ship It

This lesson produces outputs/skill-audio-llm-pipeline-picker.md. Given an audio task (transcription, music tagging, emotion inference, multi-speaker diarization, environment classification), it picks cascaded, end-to-end AF3, or a hybrid.

Author's Note

I'd run main.py first and read its output knowing what it simplifies. The spectrogram demo uses 20 mel bins instead of 80, and its filter bank splits 64 DFT bins into equal linear bands rather than warping to the Mel scale. The naive DFT runs on the first frame only. What carries over is the shape arithmetic: a 1-second clip with a 25ms window and 10ms hop gives 98 frames, and the same hop gives roughly 100 frames per second of audio.

The Q-former is the part I'd poke at. Its queries are random draws and nothing trains them, and the input frames are random Gaussian noise rather than the spectrogram above, so the attention weights mean nothing. What you can see is the mechanics: 98 frames in, 8 tokens out, each token a softmax-weighted average over every frame. Try n_queries=64 and watch only the token count change. That count is the bottleneck the LLM sees.

The cascaded-vs-end-to-end table is hardcoded, so it records a claim and measures nothing. I'd treat it as a rule of thumb for picking a pipeline, then test on my own audio before committing to one.

Exercises

Compute the log-Mel spectrogram dimension for a 30-second clip at 16kHz, 25ms window, 10ms hop, 80 Mel bins. How does this change at 48kHz?
Why does Whisper underperform on music? What audio features does BEATs capture that Whisper does not?
Audio Q-former with 64 queries vs 32: at what task complexity does 64 pay off? 32 save compute for what?
Read AF3 Section 4 on on-demand thinking. Propose three audio tasks where chain-of-thought helps the most.
Implement a minimal diarization pipeline using AF3's output. How do you signal speaker changes?

Key Terms

Term	What people say	What it actually means
Log-Mel spectrogram	"Mel features"	2D (time, frequency) array of log-magnitude values after Mel filter banks
Audio Q-former	"Audio Perceiver"	Cross-attention bottleneck from audio encoder output to fixed-length queries feeding the LLM
Cascaded	"ASR-then-LLM"	Pipeline where Whisper transcribes and a text LLM reasons; loses acoustic information
End-to-end	"Audio-LLM"	Audio features enter the LLM directly via Q-former; preserves acoustic signal
BEATs	"Audio AudioSet encoder"	SSL transformer trained on AudioSet; strong on music + environmental sounds
MMAU	"Audio reasoning bench"	10k QA pairs across speech, music, environment; 2024 eval standard
On-demand thinking	"Audio CoT"	Model can optionally emit reasoning tokens before final answer, lifts accuracy 3-5 pts