BuildPython

Voice Agents: Pipecat and LiveKit

Voice agents are a first-class production category in 2026. Pipecat gives you a Python frame-based pipeline (VAD → STT → LLM → TTS → transport), and LiveKit Agents bridges AI models to users over WebRTC. Production latency targets land at 450–600ms end-to-end for premium stacks.

TIME: ~60 minutesPREREQUISITES: Phase 14 · 01 (Agent Loop), Phase 14 · 12 (Workflow Patterns)

Learning Objectives

Describe Pipecat's frame-based pipeline: DOWNSTREAM (source→sink) and UPSTREAM (control).
Name the canonical voice pipeline stages and which transports Pipecat supports.
Explain LiveKit Agents' two voice agent classes (MultimodalAgent, VoicePipelineAgent) and when each fits.
Summarize 2026 production latency expectations and how they drive architecture choices.

The Problem

A voice agent is not a text loop with TTS bolted on. The latency budget is brutal (~600ms is the edge of a premium stack; 800–1200ms is what most deployments land at), partial audio is the default state, turn detection is itself a model, and the transports range from telephony SIP to WebRTC. You have two real options: build a frame-based pipeline yourself with Pipecat, or lean on a platform like LiveKit.

Warm-up0/3

1. A person is talking to a voice assistant and starts speaking while it is mid-answer. What should a good assistant do?Warm-up

2. In a basic agent loop, what happens right after the model asks for a tool call?Warm-up

3. Which delay does a caller actually notice when talking to a voice assistant?Warm-up

The Concept

Pipecat (pipecat-ai/pipecat)

Pipecat is a Python framework for building voice pipelines out of frames. A Frame is a typed unit of data (audio, a transcript, text), and a FrameProcessor is a stage that consumes frames and emits new ones. Chain processors together and you have a pipeline.

Frames travel in two directions:

DOWNSTREAM: source to sink. Audio comes in, TTS audio goes out.
UPSTREAM: feedback and control, such as cancellation, metrics, and barge-in.

PipelineTask manages the lifecycle through events (on_pipeline_started, on_pipeline_finished, on_idle_timeout) and observers for metrics, tracing, and RTVI.

The typical pipeline looks like this:

VAD (Silero) → STT → LLM (context alternates user/assistant) → TTS → transport

Transports: Daily, LiveKit, SmallWebRTCTransport, FastAPI WebSocket, and WhatsApp.

Two add-ons sit on top. Pipecat Flows adds structured conversations (state machines), and Pipecat Cloud is the managed runtime.

LiveKit Agents (livekit/agents)

LiveKit Agents bridges AI models to users over WebRTC. The key concepts are Agent, AgentSession, entrypoint, and AgentServer.

It ships two voice agent classes, and the choice between them is the main design decision:

MultimodalAgent: direct audio through OpenAI Realtime or an equivalent. No text sits in the middle.
VoicePipelineAgent: an STT → LLM → TTS cascade. You get text-level control at every step.

The rest of the feature list: semantic turn detection via a transformer model, native MCP integration, telephony via SIP, and 50+ models with no API keys through LiveKit Inference, with 200+ more via plugins.

Commercial platforms

Vapi (~450–600ms on an optimized premium stack) and Retell (~600ms end-to-end across 180 test calls) build on top of these. Pick a platform when you want a managed voice stack and don't have a WebRTC team.

Where this pattern goes wrong

No barge-in handling. The user interrupts and the agent keeps talking. Pipecat needs UPSTREAM cancel frames for this, and LiveKit has an equivalent.
STT confidence ignored. Low-confidence transcripts reach the LLM as if they were gospel. Gate on confidence, or ask the user to confirm.
TTS mid-sentence cutoff. When the pipeline cancels mid-utterance, TTS has to be told, or it keeps producing audio past the cancel.
Latency budget ignored. Every component adds 50–200ms. Sum your chain before shipping.

Typical 2026 latencies

VAD: 20–60ms
STT partial: 100–250ms
LLM first token: 150–400ms
TTS first audio: 100–200ms
Transport RTT: 30–80ms

End-to-end 450–600ms is premium. 800–1200ms is common. Anything over 1500ms feels broken.

Check: The Concept0/32

1. Which two directions can frames travel in a Pipecat pipeline, and what does each carry?Define

2. In a Pipecat-style pipeline, what is a Frame?Define

3. What is a Processor (FrameProcessor) in a voice pipeline?Define

4. What is the canonical Pipecat voice pipeline, in order?Define

5. What travels UPSTREAM in a Pipecat pipeline?Define

6. What is VAD in a voice pipeline?Define

7. What does Pipecat's PipelineTask manage?Define

8. Which of these is a transport that Pipecat supports, according to the lesson?Define

9. What do Pipecat Flows and Pipecat Cloud add on top of the base framework?Define

10. Which four concepts does LiveKit Agents name as its key building blocks?Define

11. What is a MultimodalAgent in LiveKit Agents?Define

12. What is a VoicePipelineAgent in LiveKit Agents?Define

13. Which two voice agent classes does LiveKit Agents ship?Define

14. How does LiveKit Agents decide, with semantic turn detection, that the user has finished speaking?Define

15. Which integration does LiveKit Agents support natively, according to its feature list?Define

16. How does LiveKit Agents connect to ordinary phone calls?Define

17. What does LiveKit Inference let you do, according to the lesson?Define

18. What end-to-end latency does the lesson describe as premium for a voice agent?Define

19. What latency figures does the lesson cite for the hosted platforms Vapi and Retell?Define

20. In the lesson's 2026 latency breakdown, which component has the widest and highest range, and so is the biggest lever on the budget?Explain

21. Why does a barge-in cancel travel UPSTREAM instead of DOWNSTREAM?Explain

22. What is the "TTS mid-sentence cutoff" pitfall the lesson describes?Explain

23. What goes wrong if STT confidence is ignored?Explain

24. Why does the lesson recommend summing component latencies before shipping?Explain

25. Why can a stack of individually fast components still feel broken to a caller?Explain

26. What does choosing MultimodalAgent over VoicePipelineAgent give up?Explain

27. A stack hits the top of every listed range: VAD 60, STT partial 250, LLM first token 400, TTS first audio 200, transport RTT 80 (all ms). Which description fits the total?Apply

28. The user talks over your voice agent and the agent keeps speaking. Which failure is this, and what fixes it?Analyze

29. A prototype measures 1,600 ms from the user finishing a sentence to the first reply audio. What follows?Analyze

30. A team on MultimodalAgent adds a check that blocks low-confidence transcripts before the model answers, but it never fires. Which wrong assumption explains it?Analyze

31. You must inspect and gate what the model hears before it answers, for example to filter low-confidence input. Which LiveKit class fits?Design

32. Your chain sums to about 560 ms, and a stakeholder wants one more text-level check stage costing 50-200 ms. What is the sound design move?Design

Build It

code/main.py is a toy frame-based pipeline. It has:

Frame types: audio, transcript, text, tts_audio, and control.
A Processor interface with process(frame).
Five scripted stages (VAD → STT → LLM → TTS → transport).
An UPSTREAM cancel frame that demonstrates barge-in.

Run it:

python3 code/main.py

The trace shows normal flow first, then a barge-in cancel that stops TTS mid-utterance.

Check: Build It0/5

1. In the toy pipeline, which five Frame types does the code define?Define

2. In the toy pipeline, Frame("audio_chunk", "hello") goes into the VAD with the five stages linked. How many words does the transport report sending?Apply

3. The toy pipeline receives Frame("audio_chunk", "") with an empty payload. What happens?Apply

4. The toy LLM has replies only for "hello" and "refund please". Frame("audio_chunk", "what time is it") goes in. What does TTS emit?Apply

5. In scenario 2 the cancel frame enters the transport with direction "upstream". In what order do the other stages see it?Apply

Use It

Pipecat when you want full control: custom processors, Python-first, pluggable providers.
LiveKit Agents for WebRTC-first deployments and telephony.
Vapi / Retell for hosted voice agents when you have no WebRTC team.
OpenAI Realtime / Gemini Live for direct audio-in/audio-out (the MultimodalAgent shape).

Check: Use It0/2

1. When should you reach for Vapi or Retell instead of Pipecat or LiveKit Agents?Design

2. You need callers reaching the agent by phone (SIP) with tools exposed over MCP, and your team has WebRTC experience. Which option fits best?Design

Ship It

outputs/skill-voice-pipeline.md scaffolds a Pipecat-shaped voice pipeline with VAD, STT, LLM, TTS, and transport, plus barge-in handling.

Author's Note

Get barge-in working before you polish anything else. The failure modes in this lesson (an agent that talks over the user, TTS that keeps going after a cancel, a bad transcript treated as truth) all make a voice agent feel broken. Write down each component's latency cost before you pick a stack, too, because the budget decides your architecture more than any feature list does.

On build versus buy, settle the cascade versus direct-audio question early. If you need to inspect or gate what the model hears, you need the text in the middle, and that means the cascade. If you don't, direct audio removes stages from the chain.

Exercises

Add a metrics observer to your toy pipeline: count frames per stage per second. Where does latency accumulate?
Implement confidence-gated STT: below threshold, request "could you repeat that?"
Add semantic turn detection: simple rule: if transcript ends with "?", end of turn.
Read Pipecat's transport docs. Swap the stdlib transport for the SmallWebRTCTransport config (stub).
Measure an OpenAI Realtime vs STT+LLM+TTS cascade on the same query. What latency cost does text-level control carry?

Key Terms

Term	What people say	What it actually means
Frame	"Event"	Typed unit of data in the pipeline (audio, transcript, text, control)
Processor	"Pipeline stage"	Handler with process(frame)
DOWNSTREAM	"Forward flow"	Source to sink: audio in, speech out
UPSTREAM	"Feedback flow"	Control: cancel, metrics, barge-in
VAD	"Voice activity detection"	Detects when user is speaking
Semantic turn detection	"Smart end-of-turn"	Model-based decision that the user is done
MultimodalAgent	"Direct audio agent"	Audio in, audio out; no text in the middle
VoicePipelineAgent	"Cascade agent"	STT + LLM + TTS; text-level control