Capstone 03 — Real-Time Voice Assistant (ASR to LLM to TTS)

A realtime voice assistant is a pipeline that listens to speech, decides when the user has finished, and answers aloud while audio is still streaming in both directions. Voice has been the fastest-moving AI UX category of 2025-2026, and the technical ceiling dropped every quarter. OpenAI Realtime API, Gemini 2.5 Live, Cartesia Sonic-2, ElevenLabs Flash v3, LiveKit Agents 1.0, and Pipecat 0.0.70 all put sub-800ms first-audio-out within reach.

Latency alone doesn't make it feel right. The feel comes from not cutting the user off, not getting cut off, recovering from a mid-sentence interruption, calling a tool without stalling the audio, and surviving a jittery mobile network.

Three chained REST calls won't get you there. The architecture has to be streaming end to end, and building it makes the failure modes visible: a VAD tuned for phone audio firing on a background TV, a turn-detector waiting for punctuation that never comes, a TTS that buffers 400ms before emitting. The capstone is to fix these one at a time under load and publish a latency-and-quality report.

A realtime voice agent is a pipeline of five streaming stages, each passing partial results downstream before its own input is finished:

1. Audio in. WebRTC from a browser, or PSTN from a phone line.
2. ASR. Streaming partial transcripts from Deepgram Nova-3 or faster-whisper.
3. Turn detection. VAD plus a small turn-detector model that reads partial transcripts for completion cues.
4. LLM. Streams tokens as soon as the turn is judged complete.
5. TTS. Streams audio out within ~200ms of the first LLM token.

Three concerns cut across all of them. Barge-in means that when the user starts speaking while the agent is speaking, the TTS cancels and the ASR picks up immediately. Tool use covers mid-conversation function calls (weather, calendar), which run on a side channel so the audio never stalls. If a call takes longer than 300ms, the agent pre-fills an acknowledgement ("one second..."). Backpressure is what happens under packet loss: partial transcripts are held, VAD raises the speech-gate threshold, and the agent avoids speaking over a message the user hasn't acknowledged.

The measurement bar is quantitative, and the numbers are the deliverable:

• WER under 8% on the Hamming VAD benchmark at 15 dB SNR
• First-audio-out p50 under 800ms across 100 measured calls
• False-cutoff rate under 3%
• MOS above 4.2 on TTS
• 50 concurrent calls on a single g5.xlarge

• Transport: LiveKit Agents 1.0 (WebRTC) plus Twilio PSTN gateway; Pipecat 0.0.70 as the alternate framework
• ASR: Deepgram Nova-3 (streaming, sub-300ms first partial) or faster-whisper Whisper-v3-turbo self-hosted
• VAD: Silero VAD v5 plus the LiveKit turn-detector (small transformer that reads partial transcripts)
• LLM: OpenAI GPT-4o-realtime for tight integration, Gemini 2.5 Flash Live, or cascaded Claude Haiku 4.5 (streaming completions, separate audio path)
• TTS: Cartesia Sonic-2 (lowest first-byte), ElevenLabs Flash v3, or open-source Orpheus for self-host
• Tools: FastMCP side-channel for weather/calendar/booking; agent pre-emits filler if tool takes >300ms
• Observability: OpenTelemetry voice spans, Langfuse voice traces with audio replay
• Deployment: single g5.xlarge (24GB VRAM) for self-hosted Whisper + Orpheus; hosted APIs for lowest latency

1. WebRTC session. Stand up a LiveKit room and a web client that streams microphone audio. On the server, attach an agent worker that joins the room.

2. ASR streaming. Feed 20ms PCM frames to Deepgram Nova-3 (or faster-whisper on GPU). Subscribe to partial and final transcripts. Log per-partial latency.

3. VAD and turn detector. Run Silero VAD v5 on the frame stream. On speech-end event, fire the LiveKit turn-detector against the latest partial transcript. Commit to "turn complete" only when VAD reports 500ms of silence and the turn-detector scores completion above 0.6.

4. LLM stream. On turn complete, start the LLM call with the running conversation plus the final transcript. Stream tokens out. At the first token, hand off to TTS.

5. TTS stream. Cartesia Sonic-2 streams audio chunks back. The first chunk must leave the server within 200ms of the first LLM token. Emit chunks to the LiveKit room, and the client plays them through the WebRTC jitter buffer.

6. Barge-in. When VAD detects new user speech while TTS is playing, cancel the TTS stream immediately, drop the remaining LLM output, and re-arm the ASR. Publish a tts_canceled span.

7. Tool side channel. Register weather and calendar as function-calling tools. When one is invoked, fire the call concurrently. If it hasn't resolved within 300ms, have the LLM emit "one second, let me check" as a filler, then resume once the tool returns.

8. Eval harness. Record 100 calls. Compute WER (against a held-out transcript), false-cutoff rate (TTS cancelled while user was mid-sentence), first-audio-out p50, TTS MOS (human or NISQA), and a jitter-loss test (drop 3% of packets).

9. Load test. Drive 50 concurrent calls on a single g5.xlarge with a synthetic caller. Measure sustained first-audio-out p95.

Running python main.py replays a simulated caller through the scheduler. Session 1 (the weather call) prints this:

$ python main.py
=== session 1: clean call with tool (weather) ===
  120ms LISTENING
  2840ms TURN COMPLETE (score=0.75) partial='what is the weather in tokyo tomorrow'
  2840ms LLM call fired
  3140ms filler 'one second, let me check'
  3260ms tool result: 68/52 partly cloudy
  3400ms LLM first token
  3580ms TTS first audio-out
  turn_complete  @ 2840ms
  first_llm_tok  @ 3400ms
  first_audio_out @ 3580ms
  turn latency   = 740ms

The timestamps are simulated frame times, not wall-clock. The turn commits at 2840ms because seven words at 320ms each plus 120ms of leading silence end at 2360ms, and the 500ms silence gate then has to elapse. The 740ms latency is the three simulated delays stacked after the commit: the tool call (420ms), the LLM time-to-first-token (140ms), and the TTS start (180ms). The filler fires 300ms into the tool call, before the result arrives. These constants are stand-ins, so treat the 740ms as a demonstration of the accounting, not a benchmark.

outputs/skill-voice-agent.md is the deliverable. Given a domain (customer support, scheduling, or kiosk), it stands up a LiveKit agent whose ASR/VAD/LLM/TTS pipeline is tuned to the measurement bar. The rubric:

1. Swap Deepgram Nova-3 for faster-whisper v3 turbo on a g5.xlarge. Measure the latency and WER gap. Identify where CPU-vs-GPU decisions matter.

2. Add an interruption-arbitration policy: what does the agent do when the user barges in during a tool call? Compare three policies (hard cancel, finish-tool-then-stop, queue next turn).

3. Run an adversarial turn-detector test: give the user long pauses mid-sentence. Tune the VAD silence threshold and the turn-detector score threshold for lowest false-cutoff without blowing past 900ms.

4. Deploy the same agent on PSTN via Twilio. Compare PSTN first-audio-out to WebRTC. Explain the jitter-buffer and codec differences.

5. Add voice activity detection for non-English languages (Japanese, Spanish). Measure the Silero VAD v5 false-trigger rate versus language-specific fine-tunes.

• LiveKit Agents 1.0: reference WebRTC agent framework
• Pipecat: alternate Python-first streaming agent framework
• OpenAI Realtime API: reference for integrated speech models
• Deepgram Nova-3 documentation: streaming ASR reference
• Silero VAD v5: VAD reference model
• Cartesia Sonic-2: low-latency TTS reference
• Retell AI architecture: production voice agent architecture
• Vapi.ai production stack: alternate production reference