Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue

Every voice agent built from Lessons 11 and 12 hits the same latency floor: 300-500 ms, because VAD, STT, the LLM, and TTS each carry their own minimum processing time. Tune each stage, parallelize what you can, and you still can't escape the shape of the pipeline itself.

Moshi (Kyutai, 2024-2026) asks a different question: what if there's no pipeline at all? One model takes audio in and sends audio out, continuously, with text running alongside as an inner monologue instead of a required stage.

That's full-duplex speech-to-speech: 160 ms theoretical latency (an 80 ms Mimi frame plus an 80 ms acoustic delay), 200 ms in practice on a single L4 GPU. Half the latency of a best-in-class pipelined voice agent.

The Moshi architecture

One 7B-parameter Temporal Transformer reads two Mimi audio streams (the user's and its own) plus a running text stream, and at every 80 ms step it writes back the next chunk of both text and audio.

Inputs. Two Mimi codec streams, both at 12.5 Hz × 8 codebooks:

• Stream 1: user audio (Mimi-encoded, constantly arriving)
• Stream 2: Moshi's own audio (generated by Moshi)

The transformer. At each 80 ms step, it:

1. Consumes the latest user Mimi tokens (8 codebooks).
2. Reads the most recent Moshi Mimi tokens (8 codebooks, as produced).
3. Generates the next Moshi text token (inner monologue).
4. Generates the next Moshi Mimi tokens (8 codebooks via a small Depth Transformer).

All three streams (user audio, Moshi audio, Moshi text) run in parallel, so Moshi can hear the user while it's still speaking, interrupt itself when the user interrupts, and back-channel with an "mhm" without breaking its main utterance.

The depth transformer. Within a frame, the 8 codebooks can't be predicted in parallel: they have inter-codebook dependencies. A small 2-layer "depth transformer" predicts them sequentially inside that same 80 ms window. It's the standard factorization for autoregressive codec LMs, the same one VALL-E and VibeVoice use.

Why inner-monologue text helps

Without explicit text, the model would have to model language implicitly, buried inside its acoustic stream. Moshi's fix: force it to emit text tokens alongside the audio, so the text stream doubles as a live transcript of what Moshi is saying. That one design choice improves semantic coherence, makes it easier to swap out the language-model head, and hands you a transcript for free.

Hibiki: streaming speech-to-speech translation

Same architecture, trained on translation pairs instead of dialogue: source audio in, target-language audio out, continuously. Hibiki-Zero (Feb 2026) drops the need for word-level aligned training data entirely. It trains on sentence-level data plus GRPO reinforcement learning for latency optimization instead.

Five language pairs ship out of the box; adapting the model to a new language takes roughly 1000 hours of data.

The broader Kyutai stack (2026)

• Moshi: full-duplex dialogue (French first, English well-supported)
• Hibiki / Hibiki-Zero: simultaneous speech translation
• Kyutai STT: streaming ASR (500 ms or 2.5 s look-ahead)
• Kyutai Pocket TTS: 100M-param TTS runs on CPU (Jan 2026)
• Unmute: full pipeline combining these on public servers

Throughput on an L40S GPU: 64 concurrent sessions at 3× real-time.

Sesame CSM: the cousin

Sesame CSM (2025) runs a similar idea: a Llama-3 backbone with a Mimi codec head. But CSM is single-directional, taking context plus text and producing speech, rather than full-duplex. It's the best "voice presence" TTS on the market, just not the same thing as Moshi's ability to listen and speak at once.

2026 performance numbers

Step 1: the interface

Moshi exposes a WebSocket server that takes in 80 ms chunks of Mimi-encoded audio and sends 80 ms chunks back out. Both directions, running the whole time.

import asyncio
import websockets
from moshi.client_utils import encode_audio_mimi, decode_audio_mimi
 
async def moshi_chat():
    async with websockets.connect("ws://localhost:8998/api/chat") as ws:
        mic_task = asyncio.create_task(stream_mic_to(ws))
        spk_task = asyncio.create_task(stream_from_to_speaker(ws))
        await asyncio.gather(mic_task, spk_task)

Step 2: the full-duplex loop

Neither direction can wait on the other: the mic keeps capturing while playback keeps draining, so each has to run as its own concurrent task instead of a request/response pair.

async def stream_mic_to(ws):
    async for chunk_80ms in mic_stream_at_12_5_hz():
        mimi_tokens = encode_audio_mimi(chunk_80ms)
        await ws.send(serialize(mimi_tokens))
 
async def stream_from_to_speaker(ws):
    async for msg in ws:
        mimi_tokens, text_token = deserialize(msg)
        audio = decode_audio_mimi(mimi_tokens)
        await play(audio)

Both directions run at the same time. Python asyncio or Rust futures are the standard transport for this.

Step 3: the training objective (conceptual)

For every 80 ms frame t:

• Input: user_mimi[0..t], moshi_mimi[0..t-1], moshi_text[0..t-1]
• Predict: moshi_text[t], then moshi_mimi[t, codebook_0..7]

Text comes first (the inner monologue), then audio, predicted codebook by codebook inside the depth transformer.

Step 4: where Moshi wins and where it doesn't

Moshi wins:

• Sub-250 ms end-to-end on cheap hardware.
• Natural back-channels and interruptions.
• No pipeline glue code.

Moshi does not win:

• Tool calling. It isn't trained for that, so you need a separate LLM path.
• Long reasoning. Moshi is a 7B dialogue model, not Claude or GPT-4.
• Factual accuracy on niche topics.
• Most production enterprise use cases, which still run pipelines in 2026.

• Limited tool calling. Moshi is a dialogue model, not an agent framework. Combine it with a pipeline when you need tools.
• Specific-voice conditioning. Moshi uses a single trained persona; cloning it is a separate training run.
• Language coverage. French and English are excellent, others limited. Hibiki-Zero helps, but you still need training data.
• Resource cost. A full Moshi session holds a GPU slot for the whole conversation, which rules out cheap shared-tenant deployment.

Save as outputs/skill-duplex-pipeline.md. Pick pipeline vs. full-duplex architecture for a voice-agent workload, and say why.

1. Easy. Run code/main.py. It simulates the two-stream plus inner-monologue architecture symbolically.
2. Medium. Pull Moshi from HuggingFace, run the server, and test one conversation. Measure wall-clock latency from end-of-user-speech to start-of-Moshi-response.
3. Hard. Take your Lesson 12 pipeline agent and compare P50 latency against Moshi on 20 matched test utterances. Write up the cases where a pipeline still wins architecturally.

• Défossez et al. (2024), Moshi: a speech-text foundation model: the paper.
• Kyutai Labs (2026), Hibiki-Zero: streaming translation without aligned data.
• Sesame (2025), Crossing the uncanny valley of voice: the CSM spec.
• Kyutai Labs, Moshi repo: install it and run the server.
• OpenAI, Realtime API: the closed commercial peer.
• Kyutai Labs, Delayed Streams Modeling: the STT/TTS framework underneath it all.