BuildPython

Omni Models: Qwen2.5-Omni and the Thinker-Talker Split

GPT-4o's May 2024 demo was disruptive because of the product shape, not the underlying model: you talk, the model sees what the camera sees, and it talks back in under 250ms. The open ecosystem spent the rest of 2024 and 2025 racing to reach that surface. Qwen2.5-Omni (March 2025) is the reference open design: a Thinker (a large text-generating transformer) plus a Talker (a parallel speech-generating transformer), linked by streaming speech tokens. Mini-Omni simplified it, Moshi matched its latency, and GLM-4-Voice extended it to Chinese. This lesson reads the Thinker-Talker architecture and the latency budget that makes real-time streaming dialogue work.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 19 (audio-LLMs), Phase 12 · 16 (any-to-any)

Learning Objectives

Split the inference pipeline into Thinker (text reasoning) and Talker (speech synthesis) and explain why parallel streaming works.
Compute the time-to-first-audio-byte (TTFAB) budget for a conversational interaction, component by component.
Describe TMRoPE's time-aligned position encoding across vision, audio, and text within the Thinker.
Name the three real-time conversational patterns: half-duplex, turn-taking, full-duplex.

The Problem

A real-time voice assistant has to do four things, fast:

Hear the user. That means real-time speech tokenization, plus voice activity detection (VAD) to know when they're done speaking.
Optionally see. Camera input at 2-4 FPS gets streamed into the Thinker alongside the audio.
Think. Compose a response conditioned on the conversation history.
Speak. Synthesize audio tokens, decode them to a waveform, and stream that to the user's speakers.

Each step adds latency. A conversational feel needs a total round trip under 500ms. Below that, the user stops noticing the lag. GPT-4o claims ~250ms, Moshi ~160ms, Qwen2.5-Omni ~350-500ms.

The components have to stream. A pipeline that batches a full response and then decodes it blows the budget before the user hears anything.

Warm-up0/3

1. Why does a pipeline that generates the full text reply before starting speech synthesis feel slow in a voice call?Warm-up

Timeline of a voice call with two lanes. After the user finishes asking, a lane for the text reply shows one long bar for the whole reply being generated. Only after that bar ends does a second lane begin a bar for turning the reply into audio. — Schematic, not to scale.

2. A video player starts playing a file while it is still downloading instead of waiting for the whole file. What is the main benefit?Warm-up

Two rows of ten numbered chunks. The top row is the video file on a server. The bottom row shows what has reached the device: chunks 1 to 4 filled, chunk 5 arriving, chunks 6 to 10 still empty dashed slots. — Schematic. The chunk count is illustrative.

3. A speaker needs 50 audio chunks every second to play without gaps, but the system behind it generates only 30 per second. What does the listener hear?Warm-up

Two rows of small squares for one second of audio. The top row has 50 outlined squares for the chunks the speaker plays each second. The bottom row has 30 filled squares for the chunks the system generates each second. — Schematic. One square is one chunk.

Check: The Problem0/2

1. In a streaming voice assistant, what job does voice activity detection (VAD) do on the input side?Define

Typed illustration of a waveform, not a recording.

2. What total round-trip latency does a voice interaction need to stay under before it feels conversational?Define

A person on the left says Question and an AI on the right says Answer. A bracket between the end of the question and the start of the answer is labelled round trip: how long? — Schematic.

The Concept

Thinker and Talker

Qwen2.5-Omni splits the work into three parts:

Thinker: a 7B-80B text-generating transformer. It consumes interleaved text, image, and audio tokens and outputs text tokens representing what to say.
Talker: a smaller speech-generating transformer (200M-1B). It consumes the Thinker's text output tokens plus recent speech-context tokens, and outputs discrete speech tokens (residual-VQ indices).
Speech decoder: a streaming waveform decoder (SNAC, MoVQGAN family) that turns speech tokens into audio samples in real time.

The separation matters. The Thinker has to be big for good reasoning. The Talker can be small because its job is local: convert text to speech tokens. A bigger Talker is not more expressive, only slower.

Both run in parallel:

Thinker emits text token t_i.
Talker consumes t_i (via streaming) and emits speech tokens s_i, s_{i+1}, ..., s_{i+k}.
Speech decoder consumes speech tokens as they arrive and emits audio samples.
By the time the Thinker is at text token t_{i+3}, the Talker has already streamed audio for t_0..t_{i+2}.

TMRoPE: time-aligned multimodal positions

The Thinker has to integrate image frames (arriving at, say, 4 FPS), audio frames (arriving at 50 frames/second), and text from conversation history. A naive sequence order, with all images first, then audio, then text, loses the temporal alignment.

TMRoPE assigns an absolute timestamp to each token. A vision token lands at t=2.3s, an audio token at t=2.32s, and the user's text token "stop" at t=2.35s. RoPE rotates attention by timestamp, so the model sees them as temporally concurrent.

That is the infrastructure behind "he waved while saying hello": the model sees the video frame and the audio at the same conceptual moment.

Streaming speech synthesis

Speech tokens have to stream. Mini-Omni (Xie & Wu, 2024) introduced "language models can hear, talk while thinking in streaming": Thinker output tokens and Talker output tokens interleave in the same sequence. The Talker fires as soon as the Thinker commits the next text token, with no batch boundaries.

Moshi (Défossez et al., October 2024) is the fastest open implementation, at 160ms TTFAB on a single A100. It is a single 7B transformer that emits text and speech tokens on alternating positions, with an "inner monologue" that separates the thinking stream from the speaking stream. In effect that is Thinker + Talker fused into one model through careful training.

VAD and turn-taking

Voice activity detection runs on the input side. Two patterns:

Half-duplex: the user speaks while the model listens, then the model speaks while the user listens. VAD silence detection (~200ms) makes the handoff.
Full-duplex: both can speak at once. The model can backchannel ("uh-huh") or interrupt. This is much harder, and Moshi supports it.

Qwen2.5-Omni supports half-duplex by default, with turn-taking via a silence threshold. Full-duplex needs application-layer handling.

Qwen3-Omni (November 2025)

The successor has a Qwen3-80B Thinker, a larger Talker, and an improved TMRoPE-v2. Latency comes close to GPT-4o's 250ms. Weights are open, and benchmarks on OmniBench are competitive with Gemini 2.0 Live.

Production latency budget

For a typical streaming interaction:

Mic -> audio tokens: 40-80ms.
Prefill (prompt + history): 100-200ms at 7B, much more at 70B.
First Thinker text token: 40ms.
Talker processes first text token: 20ms.
First speech tokens commit: 40ms.
Residual-VQ decode: 30ms.
Speech waveform decode: 50-80ms.

Total TTFAB: 320-510ms at 7B, 600-900ms at 70B. Frontier quality usually means 70B+, which is where the frontier latency gap comes from.

Token-rate math

At 16kHz speech with 50 Hz base speech tokens, you need 50 speech tokens per second of output, so the Talker must emit ≥50 tok/s to keep up. Typical LLM throughput on an H100 is 30-80 tok/s. A small (200-300M) Talker is fast enough, and a 7B Talker would fall behind.

That is why small dedicated Talker models exist instead of reusing the main model.

Check: The Concept0/23

1. What is the Thinker in a Thinker-Talker voice model?Define

A voice question enters a row of three named boxes in order, Thinker, Talker and Speech decoder, and a spoken reply leaves at the end. The Thinker box is highlighted. — Names only. The highlighted part is the one the question asks about.

2. What is the Talker in a Thinker-Talker voice model?Define

A voice question enters a row of three named boxes in order, Thinker, Talker and Speech decoder, and a spoken reply leaves at the end. The Talker box is highlighted. — Names only. The highlighted part is the one the question asks about.

3. What does the speech decoder do in the Thinker-Talker pipeline?Define

A voice question enters a row of three named boxes in order, Thinker, Talker and Speech decoder, and a spoken reply leaves at the end. The Speech decoder box is highlighted. — Names only. The highlighted part is the one the question asks about.

4. What does TTFAB measure?Define

A time axis with four labelled event dots in order: user stops speaking, Thinker emits first text token, first audio sample plays, reply finishes. — Typed event order. Spacing is not measured.

5. What is TMRoPE?Define

A table of three tokens that reach the Thinker: a vision token at 2.3 s, an audio token at 2.32 s and the text token 'stop' at 2.35 s. — Example values from the lesson text.

6. What is Moshi's inner monologue?Define

A waveform labelled what is heard above a line of text labelled what is being said, reading Sure, I can help with that. Both belong to one spoken reply from Moshi. — Typed example sentence and drawn waveform, not Moshi output.

7. What is a half-duplex voice interaction?Define

Two lanes over time. In the user lane a block labelled user speaks sits on the left. In the assistant lane a block labelled assistant speaks sits to the right, after a short stretch of silence between them. — Typed illustration of one exchange.

8. What is a full-duplex voice interaction?Define

Two lanes over time. A long assistant block reads the flight leaves at nine, and the gate is. A short user block saying uh-huh sits under the middle of it, in the same stretch of time. — Typed example, not a recording.

9. What changes in Qwen3-Omni (November 2025) relative to Qwen2.5-Omni?Define

A timeline with four models: Mini-Omni in 2024, Moshi in October 2024, Qwen2.5-Omni in March 2025 and Qwen3-Omni in November 2025, the last one highlighted with the question what changed versus Qwen2.5-Omni. — Dates from the lesson and main.py. Spacing follows months; Mini-Omni is placed in 2024 only roughly.

10. Why can the Talker be much smaller than the Thinker (200M-1B versus 7B-80B)?Explain

A log-scale axis of parameter counts from 100M to 100B. A short bar for the Talker spans 200M to 1B. A longer bar for the Thinker spans 7B to 80B. — Ranges from the lesson text.

11. When the Thinker is at text token t_{i+3}, the Talker has already streamed audio for t_0 to t_{i+2}. Why does this work?Explain

Two rows of six slots. The top row is Thinker text tokens t[0], an ellipsis, t[i], t[i+1], t[i+2] and t[i+3], with the Thinker currently at t[i+3]. The bottom row shows audio already streamed out for the first five slots and a dashed empty slot under t[i+3]. — Typed schematic. t[i+3] stands for t_{i+3}.

12. At 50 Hz base-layer speech tokens, why would a 7B Talker fall behind?Explain

Schematic. Each tick is one token.

13. Why does placing all image tokens first, then audio tokens, then text tokens hurt a model that must link a wave to a spoken 'hello'?Explain

Typed schematic. wave marks a video frame and hello marks the spoken word.

14. How does Mini-Omni's interleaving differ from a batch pipeline?Explain

Top: a batch pipeline timeline where a Thinker bar covering the whole reply is followed by a Talker bar. Below it, a dashed empty box labelled Mini-Omni by Xie and Wu 2024 asks how Thinker and Talker line up in that design. — Schematic of the batch baseline, not to scale.

15. How does Moshi's architecture differ from the explicit Thinker-Talker split?Explain

Top: three boxes in a row, Thinker 7B-80B, Talker 200M-1B and Speech decoder. Below, a dashed empty box next to the name Moshi asks what Moshi's architecture is. — Sizes from the lesson text.

16. What separates half-duplex from full-duplex conversation?Explain

Two panels, Call A and Call B, each with a user lane and an assistant lane over time. In Call A the user block ends before the assistant block begins. In Call B the user block and the assistant block overlap in time. — Typed illustration of two calls.

17. Why is the TTFAB budget computed by adding each component's latency instead of taking the slowest one?Explain

A table of seven pipeline components with their latency in ms at 7B: mic to audio tokens 40-80, prefill 100-200, first Thinker text token 40, Talker processes first text token 20, first speech tokens commit 40, residual-VQ decode 30 and speech waveform decode 50-80. The last row, Total TTFAB, shows a question mark. — Rows from the lesson's Production latency budget.

18. Why does frontier-quality reasoning usually come with worse TTFAB in this pipeline?Explain

A list of four stages that run before the first audio plays: mic to audio tokens, prefill, first Thinker text token, and Talker and decoders. Beside it, a card for a frontier-quality Thinker shows TTFAB with a question mark. — Stage names from the lesson's latency budget.

19. A reply has 4 seconds of speech at 50 base speech tokens per second. A Talker emits 40 tokens per second. How long does it take to produce all the tokens?Apply

A bar covering four seconds of playback with ticks at 0 to 4 seconds. Below it, two facts: speech tokens needed are 50 per second of speech, and the Talker emits 40 tokens per second. — Figures from the stem.

20. A voice assistant starts speaking within 300ms of the user finishing, but the audio keeps breaking up with gaps mid-sentence. Which ingredient is the most likely culprit?Analyze

A time axis starting when the user finishes a question. The first sound begins 300 ms later. The speech audio after that is drawn as five blue blocks separated by gaps. — Typed illustration. Block and gap sizes are not measured.

21. Users of a Qwen2.5-Omni app say the assistant keeps talking when they interrupt and never gives 'uh-huh' backchannels. What is the most likely reason?Analyze

Two lanes over time. A long assistant block reads assistant keeps talking. A shorter user block reading wait, stop sits under the middle of it. A note says the assistant gives no uh-huh while the user talks. — Typed illustration of the reported behavior.

22. A team ships a voice assistant on a 70B Thinker and measures about 800ms to first audio, well over their 400ms target. Which change most directly closes the gap?Design

A millisecond axis from 0 to 1000. An orange bar reaches about 800 ms for the measured time to first audio on a 70B Thinker, and a green dashed line marks the 400 ms target. — Figures from the stem.

23. You need an open-weights voice model that can be interrupted mid-sentence and give backchannels, with the lowest TTFAB you can get. Which design fits?Design

A checklist of four requirements, each with an empty box: open weights, can be interrupted mid-sentence, can give backchannels such as uh-huh, and lowest TTFAB available. — Requirements from the stem.

Use It

code/main.py is a calculator, with no audio in it. It:

Simulates a Thinker-Talker pipeline with mock token-emission rates.
Computes TTFAB for configurable model sizes and mic sample rates.
Demonstrates half-duplex turn-taking with a VAD silence threshold.

Check: Use It0/7

1. `ttfab` runs with Thinker=7B, Talker=200M, mic=16000, no vision. What TTFAB does the script print?Apply

A code view of the ttfab calculation from main.py. A config line sets Thinker 7B and Talker 200M with the default 16000 mic rate and no vision. Below are the lines computing mic_ms, prefill, first_text, talker_first, rvq_decode and wave_decode, and a loop that adds each component's milliseconds to total. — Lines quoted from main.py; unrelated lines left out.

2. `ttfab` runs with Thinker=7B, Talker=300M, vision on. What does the script print?Apply

A code view of the ttfab calculation from main.py with a config of Thinker 7B, Talker 300M and vision on. It lists the component formulas, the loop that sums them, and the four label thresholds: under 250 GPT-4o class, under 400 conversational, under 700 noticeable but usable, otherwise sluggish. — Lines quoted from main.py. The last four lines condense its print_ttfab if/elif chain.

3. In `ttfab`, what does `talker_first` evaluate to when `talker_m` is 200?Apply

A code view showing the line talker_first = max(15, 20 * (cfg.talker_m / 300.0)), then talker_m set to 200 and talker_first with a question mark. — Line quoted from main.py.

4. With `simulate_turn_taking(200)`, how long after the user stops speaking does 'Talker first audio out' fire?Apply

A code view of simulate_turn_taking from main.py. Events are listed at 0 and 450 ms, user stops speaking at 3800, and three later events at 3800 plus the silence threshold, plus 200 and plus 400. The call simulate_turn_taking(200) is shown below. — Code quoted from main.py. Three lines are shortened with ... where each begins with 3800.

5. For `StreamConfig(thinker_b=72, talker_m=300, include_vision=True)`, what Thinker prefill does `ttfab` compute?Apply

A code view of the prefill lines from ttfab: prefill is 100 times thinker_b over 7.0, and 80 is added if include_vision is true. Below it, a config with thinker_b 72, talker_m 300 and vision on, and prefill with a question mark. — Lines quoted from main.py.

6. The script prints about 1.7 seconds for its 70B and 72B configs, while the lesson's budget quotes 600-900ms at 70B. What is the best explanation?Analyze

A millisecond axis from 0 to 2000 with three bars. A blue range bar spans 600 to 900 for the lesson budget at 70B. Two orange bars end near 1690 and 1697 for the script's 72B and 70B vision configs. — Totals printed by main.py, range quoted from the lesson.

7. You must add camera input to a voice assistant and still hit a 400ms TTFAB target. Which configuration does the budget support?Design

Schematic of the stem's requirement.

Ship It

This lesson produces outputs/skill-omni-streaming-budget.md. Given a real-time voice product's target TTFAB and feature set (vision-in, bilingual, full-duplex), it picks Qwen2.5-Omni, Qwen3-Omni, Moshi, or Mini-Omni and sizes the Thinker and Talker.

Author's Note

I'd run main.py and read the four configs side by side before trusting any latency figure in the body. The 7B Thinker with a 200M Talker comes out at 305ms, which lands in the conversational band. Add vision and a 300M Talker and it moves to 390ms, almost all of it from the extra prefill.

My bet is that the Talker rows are the ones worth staring at. In the code the Talker's first-token cost is max(15, 20 * (talker_m / 300.0)), so going from 300M to 1B adds about 47ms while the Thinker terms scale with thinker_b / 7.0. That matches the argument for a small Talker, though the toy only prices size in latency and does not model the token-rate constraint at all.

I'd also treat the 70B rows with suspicion. Prefill and first-token time scale linearly with parameter count here, so the 70B and 72B configs print about 1.7 seconds, well outside the 600-900ms the budget section quotes. Real serving stacks use batching, tensor parallelism, and quantization, none of which are in this model. Use the script to see where the milliseconds go, and use measured numbers from your own hardware for a real product decision.

The turn-taking demo is a fixed timeline, not a VAD. The silence threshold only shifts when the end-of-turn event fires, and the "~400ms" it prints is a hard-coded string covering the two 200ms steps after the trigger. Add the 200ms threshold and the timeline actually shows 600ms from the user stopping to first audio.

Exercises

Your target TTFAB is 300ms. On a 7B Thinker and 300M Talker, write out every component's latency.
Qwen2.5-Omni uses TMRoPE. Describe what the model sees for a prompt where the user starts speaking at t=1s and the camera catches a gesture at t=1.2s.
Full-duplex support requires the model to emit audio while listening. Propose a training data format that teaches this.
Read Moshi's paper Section 4. Describe the "inner monologue" separation and why it avoids the Thinker-Talker split.
Compute the throughput budget: how fast must a Talker emit tokens to keep up with 16kHz speech at 50 base-layer tokens/sec?

Key Terms

Term	What people say	What it actually means
Thinker	"Reasoning brain"	Large text-generating transformer producing what to say
Talker	"Speech-generating mouth"	Small transformer producing discrete speech tokens from Thinker's text
TTFAB	"Latency budget"	Time-to-first-audio-byte: from user speech end to first audio sample out
TMRoPE	"Time-aligned RoPE"	Position encoding using absolute timestamps across vision, audio, text
Half-duplex	"Turn-taking"	User and model alternate; VAD silence detects user-done
Full-duplex	"Simultaneous"	Model can speak and listen at the same time; backchannel capable
Inner monologue	"Moshi separation"	Single-model design where thinking-stream and speaking-stream interleave