Build a Voice Assistant Pipeline — The Phase 6 Capstone

Build an end-to-end assistant:

1. Captures mic input (16 kHz mono).
2. Detects start/end of user speech.
3. Transcribes streaming.
4. Passes transcript to an LLM that can call tools (timer, weather, calendar).
5. Streams LLM text to a TTS.
6. Plays audio back to the user.
7. Stops if the user interrupts mid-response.

Latency target: first TTS audio byte within 800 ms of the user finishing their utterance, on a laptop CPU. Quality target: no missed words, no hallucinated subtitles on silence, no voice-cloning leakage, no successful prompt injection.

The seven components

1. Audio capture. The mic feeds 16 kHz mono audio in 20 ms chunks: sounddevice in Python, or native AudioUnit, ALSA, or WASAPI in production.
2. VAD (Lesson 11). Silero VAD at a 0.5 threshold, 250 ms minimum speech, 500 ms silence hangover. It signals "start" and "end" for each turn.
3. Streaming STT (Lessons 4-5). Whisper-streaming, Parakeet-TDT, or Deepgram Nova-3 over API, producing partial and final transcripts as the audio arrives.
4. LLM with tool calling. GPT-4o, Claude 3.5, or Gemini 2.5 Flash, given a JSON schema for tools and streaming tokens as it reasons.
5. Streaming TTS (Lesson 7). Kokoro-82M for the fastest open option, Cartesia Sonic if you want commercial quality. Start TTS after the first 20 LLM tokens, not the full response.
6. Playback. Audio goes out to the speaker, Opus-encoded when the network is thin.
7. Interruption handler. If VAD fires while TTS is playing, stop playback, cancel the LLM call, and restart STT.

Three failure modes you'll hit

1. First-word clip. VAD starts a beat too late, so the user's "hey" never makes it into the transcript. Start the threshold at 0.3, not 0.5.
2. Mid-response interrupt confusion. The LLM keeps generating after the user interrupts, so the assistant talks over them. Wire VAD to cancel the LLM directly.
3. Silence hallucination. Whisper outputs "Thanks for watching" on silent warm-up frames. Always VAD-gate before transcription runs.

2026 production reference stacks

Step 1: mic capture with chunking (pseudocode)

import sounddevice as sd
 
def mic_stream(chunk_ms=20, sr=16000):
    q = queue.Queue()
    def cb(indata, frames, time, status):
        q.put(indata.copy().flatten())
    with sd.InputStream(channels=1, samplerate=sr, blocksize=int(sr * chunk_ms/1000), callback=cb):
        while True:
            yield q.get()

Step 2: VAD-gated turn capture

Lowering the VAD threshold only closes part of the first-word-clip gap. You also need a pre-roll buffer: keep the last 200-400 ms of audio around at all times, so when VAD finally fires, the captured turn still includes the syllable it almost missed.

def capture_turn(stream, vad, pre_roll_ms=300, silence_ms=500):
    buf, pre, triggered = [], collections.deque(maxlen=pre_roll_ms // 20), False
    silent = 0
    for chunk in stream:
        pre.append(chunk)
        if vad(chunk):
            if not triggered:
                buf = list(pre)
                triggered = True
            buf.append(chunk)
            silent = 0
        elif triggered:
            silent += 20
            buf.append(chunk)
            if silent >= silence_ms:
                return b"".join(buf)

Step 3: streaming STT → LLM → TTS

async def turn(audio_bytes):
    transcript = await stt.transcribe(audio_bytes)
    async for token in llm.stream(transcript):
        async for audio in tts.stream(token):
            await speaker.play(audio)

Step 4: tool calling inside the LLM loop

tools = [
    {"name": "get_weather", "parameters": {"location": "string"}},
    {"name": "set_timer", "parameters": {"seconds": "int"}},
]
 
async for chunk in llm.stream(user_text, tools=tools):
    if chunk.type == "tool_call":
        result = dispatch(chunk.name, chunk.args)
        continue_streaming(result)
    if chunk.type == "text":
        await tts.stream(chunk.text)

Step 5: interruption handling

tts_task = asyncio.create_task(tts_loop())
while True:
    chunk = await mic.get()
    if vad(chunk):
        tts_task.cancel()
        await speaker.stop()
        await new_turn()
        break

code/main.py runs a full simulation that wires all seven components together with stub models, so you can see the pipeline's shape before you touch real hardware. To make it real, swap the stubs for:

• silero-vad (pip install silero-vad)
• deepgram-sdk or openai-whisper
• openai (gpt-4o) or anthropic
• kokoro or cartesia
• sounddevice for I/O

• Logging PII forever. Full-turn audio counts as PII in most jurisdictions. Retain it for 30 days, encrypted at rest, and no longer.
• No barge-in. Users will interrupt you. Your assistant has to stop talking when they do.
• TTS that blocks. Synchronous TTS blocks the event loop. Use async, or move it to a separate thread.
• No tool-call error handling. Tools fail. Feed the error back to the LLM, let it retry once, then degrade gracefully.
• Overzealous hallucination filters. Filter too hard and the assistant just repeats "I can't help with that." Filter too little and it says anything. Calibrate against a held-out set.
• No wake-word option. Always-listening is a privacy liability. Add a wake-word gate; Porcupine or openWakeWord both work.
• Untrusted transcripts and tool results. A prompt-injected instruction can ride in on a transcribed webpage or a tool's API response just as easily as on typed text. Treat every transcript and tool result as untrusted input: only the user's own speech authorizes an action.
• Cloned-voice leakage. If your TTS voice is a clone of a real person, don't let arbitrary user text drive it for someone else's audio. Anti-spoofing and watermarking (Lesson 16) are the real fix, not a policy promise.

Save this as outputs/skill-voice-assistant-architect.md. Feed it a budget, a scale target, a language, and a compliance constraint, and it produces a full stack spec.

1. Easy. Run code/main.py. It simulates one full turn end-to-end with stub modules and prints per-stage latency.
2. Medium. Replace the STT stub with a real Whisper model on a pre-recorded .wav. Measure WER and end-to-end latency.
3. Hard. Add tool calling: implement get_weather (any API) and set_timer. Route the LLM through the tools and verify that when the user says "set a 5 minute timer" the right function fires and the spoken reply confirms it.

• LiveKit's voice agent quickstart: the production-grade reference.
• Pipecat's voice agent examples: a DIY-friendly framework.
• OpenAI Realtime API: the managed voice-native path.
• Kyutai Moshi: the full-duplex reference (Lesson 15).
• Porcupine wake-word: wake-word gating.
• Anthropic's tool use guide: LLM function calling.