LearnPython

MIO and Any-to-Any Streaming Multimodal Models

GPT-4o ships something most open models could not match: an agent that hears voice, sees video, and speaks back in real time. The open answer by late 2024 was MIO (Wang et al., September 2024). It tokenizes text, image, speech, and music, trains one causal transformer over the interleaved sequences, and generates any modality from any other. AnyGPT (Zhan et al., February 2024) was the proof of concept, MIO is the scale-up, and Unified-IO 2 (Allen AI, December 2023) is the cousin with vision and action grounding. This lesson reads the any-to-any pattern: four tokenizers, one transformer, streaming-friendly decode.

TIME: ~120 minutesPREREQUISITES: Phase 12 · 11 (Chameleon), Phase 6 (Speech and Audio)

Learning Objectives

Design a shared vocabulary that hosts text, image, speech, and music tokens without collisions.
Compare SEED-Tokenizer (images) and SpeechTokenizer residual-VQ (speech) on compression + reconstruction trade-offs.
Explain the four-stage curriculum that builds up any-to-any generation.
Name the three open any-to-any recipes and their main trade-offs: MIO, AnyGPT, Unified-IO 2.

The Problem

A unified multimodal model is easy to claim and hard to build at scale. Most "any-to-any" systems until 2024 were pipelined: vision model → text representation → speech model → audio. Each hop loses information, adds latency, and makes training harder. GPT-4o's demo video showed a single-model alternative with subsecond response; open systems trailed by months.

The engineering challenges:

You need a tokenizer per modality. It has to compress well enough to save compute, reconstruct well enough to sound or look right, and emit tokens at a rate the transformer can consume.
A single vocabulary must allocate space for text (32k+), image (16k+), speech (4k+), music (8k+). Forty-thousand-plus entries minimum.
Training data has to cover the input-output pairs you care about (text→image, image→speech, speech→image, and so on), or the model has to compose them.
Inference must stream output tokens fast enough for conversational latency (<500ms time-to-first-audio-byte).

Warm-up0/3

1. Chameleon turns images into discrete tokens that share one sequence with text. What does this early-fusion design give you that a captioner feeding an LLM does not?Warm-up

Two diagrams. Setup A: a photo goes to a captioner, then caption text, then an LLM. Setup B: a photo goes through an image tokenizer and the discrete image tokens sit in one sequence between text tokens. — Typed schematic of the two setups in the stem; the token counts are illustrative.

2. In a residual vector quantizer used by neural audio codecs, what does each additional codebook layer encode?Warm-up

A frame vector feeds codebook 1, then codebook 2, then codebook 3 in a chain. Each codebook emits one index chip below it. A note asks what codebooks 2 and 3 encode. — Typed schematic of a 3-layer residual-VQ chain; real codecs use more layers.

3. Neural audio and image tokenizers often use vector quantization, which replaces each continuous vector with the index of its nearest codebook entry. Why does that help a transformer?Warm-up

Typed illustrative numbers, not from any real tokenizer.

Check: The Problem0/2

1. What does an any-to-any model do?Define

Three example requests drawn as input chips and output chips: photo plus question to spoken answer, spoken line to picture, melody clip to text. — Typed examples; chip colors mark modality: text blue, image orange, speech green, music purple.

2. Why do pipelined any-to-any systems (vision model, then text, then speech model) fall short of a single model?Explain

A left-to-right chain of five boxes: image, vision model, text, speech model, audio, joined by arrows. — Typed from the stem; the chain is the one in the lesson's Problem section.

The Concept

Four tokenizers for four modalities

MIO's tokenizer stack:

Text: standard BPE, vocab ~32000.
Image: SEED-Tokenizer (2023), a quantized VAE with a discrete codebook, 4096 entries, 32x32 tokens per image.
Speech: SpeechTokenizer residual-VQ (2023). It encodes a 16kHz waveform into 8 hierarchical codebooks. The first level is coarse content, and later levels add prosody and speaker identity.
Music: similar residual-VQ (Meta's MusicGen / Encodec family), 4-8 codebooks.

Each modality produces integer tokens, and each gets a disjoint ID range in the shared vocabulary:

text:   0..31999
image:  32000..36095  (4096 image tokens)
speech: 36096..40191  (4096 speech base tokens, plus residual layers)
music:  40192..48383  (8192 music tokens)
sep:    48384..48389  (<image>, <speech>, <music>, </...>, etc.)

Total: ~48k vocabulary. The table counts only the base speech tokens, so main.py, which also reserves a slot for speech layers 1..7, prints 52486. The input embedding and output projection span all of it.

Streaming decode

Speech generation uses residual-VQ. The transformer predicts the base (layer 0) speech tokens; a parallel-decoded residual quantizer predicts the subsequent layers. Each layer 0 token is roughly 50ms of audio at 16kHz.

The streaming pattern:

User speaks into mic; real-time audio tokenizer emits speech tokens every 50ms.
MIO consumes tokens as they arrive (prompt prefill + incremental forward).
Output tokens stream out as generated; a parallel speech decoder converts them to audio samples with ~50-150ms latency.
Time-to-first-audio-byte: ~300-500ms in MIO paper, approaching GPT-4o's ~250ms.

Mini-Omni (arXiv:2408.16725), GLM-4-Voice (arXiv:2412.02612), and Moshi (arXiv:2410.00037) are complementary streaming speech-LLM designs. Moshi in particular achieves 160ms round-trip on a single GPU.

Four-stage curriculum

MIO's training curriculum:

Stage 1: alignment. Large-scale modality-pair corpora: text-image, text-speech, text-music. Each pair uses its own token vocabulary segment. Trains the shared vocabulary.
Stage 2: interleaved. Multi-modality interleaved documents (blogs with images + video, podcasts with transcripts, etc.). Trains cross-modality context.
Stage 3: speech-enhanced. Extra audio data to lift speech quality without losing text capability.
Stage 4: SFT. Instruction tuning across modalities: VQA, captioning, narration, speech-to-speech dialogue.

Skipping a stage hurts specific capabilities. Without stage 2 the model loses cross-modality context, and without stage 3 speech is poor.

Chain-of-visual-thought

MIO introduces chain-of-visual-thought: the model emits intermediate image tokens as a reasoning step. For "is the cat climbing a tree?" the model:

Emits <image> tokens rendering the scene (from the input image or a sketch).
Emits text analyzing the sketch.
Emits the final answer.

The rendered intermediate image serves as a scratchpad. Benchmarks improve on spatial-reasoning tasks. The idea mirrors chain-of-thought for text reasoning.

Competitors in any-to-any

AnyGPT (arXiv:2402.12226): 4 modalities (text, image, speech, music), similar design.
Unified-IO 2 (arXiv:2312.17172): adds vision action outputs, depth, normals. More task diversity, smaller scale.
NExT-GPT (arXiv:2309.05519): LLM + modality-specific diffusion decoders. Not a single-model approach.
CoDi (arXiv:2305.11846): composable diffusion; any-to-any via shared latent.

MIO is the closest to pure-token any-to-any. AnyGPT is its conceptual ancestor.

Latency budget

For a conversational product, each component adds to the total:

Mic to audio tokens: ~50ms.
Prefill (audio tokens + history): ~100ms on an 8B model.
First output token: ~50ms.
Parallel residual-VQ + speech decoder: ~100-150ms.

Total time-to-first-audio-byte: ~300ms minimum. GPT-4o claims ~250ms. Moshi claims 160ms. MIO/AnyGPT are in the 400-600ms range per public benchmarks.

Why any-to-any stays hard

Even in 2026, open any-to-any models trail closed ones on two fronts:

Speech quality. The residual-VQ tokenizer is lossy; conversational speech sounds robotic compared to ElevenLabs-class voices.
Cross-modality reasoning. Asking the model "sing about what you see" still fails more often than pure-vision tasks.

Both are open research problems. Qwen3-Omni (Lesson 12.20) is the most advanced open attempt in 2025.

Check: The Concept0/22

1. In MIO's speech tokenizer, how do the 8 residual-VQ codebooks divide the work?Define

A 16 kHz waveform goes to an encoder that outputs a stack of eight codebook layers labelled layer 0 to layer 7. A note asks what each layer holds. — Typed schematic of the lesson's description of the speech tokenizer.

2. What is SEED-Tokenizer in MIO's stack?Define

Typed schematic; no output is shown.

3. What does time-to-first-audio-byte measure?Define

Two waveform lanes on one time axis. The user microphone lane has speech on the left; the speaker output lane has audio starting later on the right. — Typed schematic; no durations are shown.

4. Asked "is the cat climbing a tree?", how does chain-of-visual-thought answer?Define

Typed schematic; the photo is a generic icon, not a real image.

5. What is MIO's four-stage curriculum?Define

A system map: raw data in four modalities, four tokenizers, a transformer and a speech decoder. A dashed tag floating above reads 'four-stage curriculum: what is it?'. — Typed schematic; the tag is not attached to any part.

6. Why must each modality get its own disjoint ID range in the shared vocabulary?Explain

A bar split into five blocks labelled text, image, speech, music and sep, with a sequence below of text, image and separator tokens. — Typed schematic; block widths are not to scale.

7. How do SEED-Tokenizer (images) and SpeechTokenizer (speech) differ in how they represent content?Explain

Two rows. A photo goes into SEED-Tokenizer and a 16 kHz waveform goes into SpeechTokenizer; each leads to a dashed box asking how it represents content. — Typed schematic; no output structure is shown.

8. What is stage 3 (speech-enhanced) of MIO's curriculum for?Explain

A dashed box asking what data enters stage 3 leads to the MIO transformer, which is followed by four modality chips for text, image, speech and music. — Typed schematic; the data box is deliberately empty.

9. A team trains a MIO-style model but skips the interleaved multi-modality documents stage. Which capability is most likely to suffer?Explain

A sketch of a document mixing text lines, an image and an audio waveform, beside a red dashed box saying the interleaved stage was not run. — Typed schematic of the document type described in the lesson (blogs with images, podcasts with transcripts).

10. In streaming speech output, which tokens does the transformer predict, and what handles the rest?Explain

A transformer box leads to a grid of eight rows labelled layer 0 to layer 7 and six columns of empty cells, which leads to a speech decoder box. — Typed schematic; no row is marked as coming from a particular part.

11. Which description best separates NExT-GPT and CoDi from MIO?Explain

Three cards named NExT-GPT, CoDi and MIO, each with a dashed box asking how it works. — Typed schematic; no architecture is shown.

12. Which two gaps does the Concept section name between open and closed any-to-any models?Explain

Typed schematic.

13. A MIO-style model adds a 4096-entry video codebook to its shared vocabulary. Which change is required?Apply

Text listing the current vocabulary parts and a new 4096-entry video codebook to add, above two boxes for the input embedding matrix and the output projection matrix. — Typed schematic.

14. A user speaks for 3 seconds. At roughly one layer 0 speech token per 50 ms, about how many base speech tokens does the model consume?Apply

A waveform spanning 3 seconds with ticks at 0, 1, 2 and 3 seconds, and a zoom showing three token cells each covering 50 ms. — Typed schematic; only three of the tokens are drawn.

15. A prompt contains two images and ten text tokens. With SEED-Tokenizer's 32x32 tokens per image, how many image tokens does the model read?Apply

Two square grids of 32 by 32 cells for two images, and ten small text-token chips in two rows of five. — Typed schematic.

16. A developer gives music tokens the IDs 36096..44287, which overlap the speech range. After training, a request for a song returns speech-like noise. Which design rule was broken?Analyze

Three bars on an ID axis: image 32000 to 36095, speech 36096 to 40191 and music 36096 to 44287, each starting where it is drawn, followed by a request for a song that returns speech-like noise. — Typed from the stem and the lesson table.

17. A team runs alignment, interleaved and SFT stages, skipping the speech-enhanced stage. VQA works well, but spoken replies sound poor. Which diagnosis fits?Analyze

Two scorecards: visual question answering works well, spoken replies sound poor. — Typed from the stem; no training stages are shown.

18. A MIO-style assistant answers text and image questions well, but its spoken replies sound robotic even after more training. Which component is the most likely limit?Analyze

Three rows: text answers are fluent, image questions answered well, spoken replies sound robotic even after more training. — Typed from the stem.

19. A voice assistant must answer quick factual questions within a 300 ms budget. A teammate proposes chain-of-visual-thought on every query. What is the main problem?Analyze

A bar showing a 300 ms latency budget, an example factual query, and a note that a teammate proposes chain-of-visual-thought on every query. — Typed schematic; the example query is made up.

20. A product spec asks for a 250 ms time-to-first-audio-byte using a MIO-style stack on an 8B model. What is the best assessment?Design

A spec card reading 250 ms time-to-first-audio-byte on a MIO-style 8B stack, and a 0 to 600 ms axis with a marker at 250 ms and the 500 ms conversational limit. — Typed from the stem and the lesson; the 500 ms mark is the lesson's conversational threshold.

21. You must build a speech-in, speech-out assistant on one transformer with a 300 ms target. Which design follows the lesson's ideas?Design

A microphone box connects to a dashed box reading one transformer: design with a question mark, which connects to a speaker box; the requirement is a 300 ms target. — Typed schematic.

22. Your team must pick a starting point for an open any-to-any project over text, image, speech and music. How should you weigh AnyGPT, MIO and Unified-IO 2?Design

A requirement row of text, image, speech and music chips, and three candidate cards named AnyGPT, MIO and Unified-IO 2, each with a dashed box asking about its trade-offs. — Typed schematic; no trade-offs are shown.

Use It

code/main.py:

Defines the four-modality vocabulary allocation and prints it.
Routes a list of multimodal inputs (text, image, audio-clip, music) through the tokenizer router.
Simulates streaming decode for a text-to-speech response with latency counting.
Computes the expected time-to-first-audio-byte given encoder, prefill, and decoder latencies.

Check: Use It0/6

1. According to the Concept section's vocabulary table, how large is the shared vocabulary, and why does the code's printout differ?Apply

A table of five ID ranges from the Concept section on the left, and on the right a terminal box titled masked with the printout header, hidden slot rows and a total shown as question marks. — Left: table typed from the lesson. Right: the real printout with its rows and total masked.

2. In `main.py`, the slots run text (32000), image SEED (4096), speech L0 (4096), speech L1..L7 (4096), then music. At which ID does the music slot start?Apply

A table of slots with sizes: text BPE 32000, image SEED 4096, speech L0 4096, speech L1..L7 4096 and music 8192. The text slot starts at 0 and the music start is a question mark. — Sizes from the stem and main.py; start values other than the first are left out.

3. What total does `streaming_decode_latency(prompt_audio_seconds=4.0, model_size_b=8)` sum to, given the default of 270 ms at 2 seconds?Apply

A table of the five latency steps from main.py with columns for 2.0 seconds and 4.0 seconds of prompt audio. All step values are hidden, the total for 2.0 seconds is 270 ms and the total for 4.0 seconds is a question mark. — Step names from main.py; step values masked on purpose.

4. A team moves their voice agent from an 8B to a 70B backbone with the same audio input. Using the latency budget and the toy trace in `main.py`, which components grow?Apply

The five steps of main.py's latency trace listed vertically beside a box saying the backbone changes from 8B to 70B with the same audio input. — Step names from main.py; no values shown.

5. With `model_size_b=16` and 2 seconds of prompt audio, what does the `main.py` trace show?Apply

A table of the five latency steps from main.py with columns for an 8B and a 16B model. All step values are question marks, the 8B total is 270 ms and the 16B total is a question mark. — Step names from main.py; step values masked on purpose; the 8B total is the default given in the lesson.

6. What does `route_inputs` assign to a `voice` input and to a `video` input?Apply

Two input records, a voice clip user.wav and a video with no payload shown, go into route_inputs, and each leads to a dashed box reading path with a question mark. — Typed setup; the function body and output are not shown.

Ship It

This lesson produces outputs/skill-any-to-any-pipeline-auditor.md. Given a conversational product spec (modalities in, modalities out, latency target), it audits the MIO-family design choices and computes the latency budget.

Author's Note

I'd run main.py and read the vocabulary table it prints before anything else. It lays out the shared ID space slot by slot (text BPE, SEED image tokens, speech, music, then six separator tokens), so you can see that each modality gets a disjoint range and that the embedding matrix has to span all of it. The total it prints is 52486, not the ~48k in the table above. The code gives speech L1..L7 its own 4096-entry slot, and the table counts only the base speech tokens, so treat the table as a sketch and the printout as the arithmetic.

The latency trace is worth tweaking by hand. With the defaults (2 seconds of prompt audio, an 8B model) it sums to 270 ms: 40 for mic to tokens, 80 for prefill, 40 for the first output token, 30 for residual-VQ layers 1..7, and 80 for the speech decoder. Those are constants and linear scalings I'd call plausible, not measurements, and 270 ms sits below the ~300 ms floor quoted in the Concept section. Change model_size_b and watch which terms move. Only prefill and first-token scale with model size, which tells you where a bigger model hurts.

Keep the toy in proportion. The router is an if/else on a kind string that returns a tokenizer name, and the chain-of-visual-thought demo prints four fixed steps. Neither runs a tokenizer or a transformer. If you want to learn how any-to-any behaves, read the MIO paper next, and treat this file as a way to get the budget and the ID layout straight.

Exercises

Your product accepts speech input and returns speech output. What's the end-to-end latency budget target? List the components that spend time.
SpeechTokenizer residual-VQ uses 8 codebooks. Propose why parallel-decoding the residual levels is necessary (vs sequential) and what latency savings it brings.
Your vocabulary has 32k text + 4k image + 4k speech. Add 8k music and ~10 separators. What is the embedding-matrix parameter cost at hidden dim 4096?
Chain-of-visual-thought emits an intermediate image. What kinds of questions benefit? What kinds are hurt by the extra tokens?
Read Moshi (arXiv:2410.00037). Describe its "inner monologue" technique and compare to MIO's chain-of-visual-thought.

Key Terms

Term	What people say	What it actually means
Any-to-any	"Multimodal in/out"	A single model that accepts and emits text, image, speech, and music in any direction
Residual-VQ	"Speech tokenizer stack"	Multi-codebook tokenization where each layer adds information; base layer is content, later layers are prosody
SEED-Tokenizer	"Image codes"	Discrete image tokenizer with 4096-entry codebook used by MIO
Chain-of-visual-thought	"Visual scratchpad"	The model generates an intermediate image as a reasoning step before its final answer
Time-to-first-audio-byte	"TTFAB"	Latency from user voice to first audio output; <500ms for conversational feel
Four-stage curriculum	"Training recipe"	Alignment -> interleaved -> speech-enhanced -> SFT, in that order