Real-Time Audio Processing

You want a voice assistant that feels alive, not like a phone tree. Human turn-taking latency, the silence-to-response gap in an ordinary conversation, runs about 230 ms. Cross 500 ms and the assistant starts sounding robotic; cross 1500 ms and it sounds broken. That's the bar. Here's the budget for a full hear → understand → respond → speak loop in 2026:

Moshi (Kyutai, 2024) hit 200 ms full duplex. GPT-4o-realtime (2024) runs around 320 ms. Cascaded pipelines in 2022 shipped at 2500 ms, roughly 10× slower. Three techniques closed that gap: streaming everywhere, asynchronous pipelining with partial results, and interruptible generation.

The pipeline is seven pieces. In normal operation, a frame lands in the ring buffer, VAD checks it, and streaming ASR transcribes it; interruption can cut in at any point, and transport plus the jitter buffer handle getting the audio there in the first place.

Frame / chunk / window. Real-time audio flows as fixed-size blocks, not a continuous stream. The common choice is 20 ms, 320 samples at 16 kHz, and everything downstream has to keep pace with that cadence.

Ring buffer. A fixed-size circular buffer: a producer thread writes new frames, a consumer thread reads them, and nothing gets allocated in the hot path. Size it as maximum latency times sample rate. A 2-second ring at 16 kHz holds 32,000 samples.

VAD (Voice Activity Detection). A classifier, model or simple energy heuristic, that labels each frame speech or non-speech. It gates downstream work whenever nobody's speaking. Silero VAD 4.0 (2024) runs under 1 ms per 30 ms frame on CPU; webrtcvad is the older alternative.

Streaming ASR. A model that emits partial transcripts as audio arrives, rather than waiting for the whole utterance. Parakeet-CTC-0.6B in streaming mode (NeMo, 2024) hits 2–5% WER at 320 ms latency. Whisper-Streaming (Macháček et al., 2023) chunks Whisper into something close to streaming, at roughly 2 s of latency.

Interruption. When the user speaks while the assistant is talking, you must detect the barge-in, stop the TTS, and discard the remaining LLM output, all within 100 ms, or the user perceives a deaf assistant.

WebRTC Opus transport. 20 ms frames, 48 kHz, adaptive bitrate from 8 to 128 kbps. This is the standard for browser and mobile; LiveKit, Daily.co, and Pion are the 2026 stacks for building on top of it.

Jitter buffer. A small queue that reorders and smooths network packets that arrive out of order or late: too small and you get audible gaps, too large and you add latency. 60–80 ms is typical.

Common gotchas

• Thread contention. Python's GIL plus a heavy model can starve the audio thread. Use a C-callback audio library (sounddevice, PortAudio) and keep Python off the hot path.
• Sample-rate conversion latency. Resampling inside the pipeline adds 5–20 ms. Resample upfront, or use a zero-latency resampler (PolyPhase, soxr_hq).
• TTS priming. Even fast TTS like Kokoro has a 100–200 ms warm-up on the first request. Cache the model and warm it with a dummy run before the first real turn.
• Echo cancellation. Without AEC, TTS output re-enters the mic and triggers ASR on the bot's own voice. WebRTC AEC3 is the open-source default.

Step 1: ring buffer

import collections
 
class RingBuffer:
    def __init__(self, capacity):
        self.buf = collections.deque(maxlen=capacity)
    def write(self, frame):
        self.buf.extend(frame)
    def read(self, n):
        return [self.buf.popleft() for _ in range(min(n, len(self.buf)))]
    def level(self):
        return len(self.buf)

Capacity sets your maximum buffering latency: 32,000 samples at 16 kHz buys you 2 seconds.

Step 2: VAD gate

def simple_energy_vad(frame, threshold=0.01):
    return sum(x * x for x in frame) / len(frame) > threshold ** 2

In production, swap this for Silero VAD:

import torch
vad, _ = torch.hub.load("snakers4/silero-vad", "silero_vad")
is_speech = vad(torch.tensor(frame), 16000).item() > 0.5

Step 3: streaming ASR

Same idea as the VAD swap: start from an interface, then point it at a real streaming model. Parakeet emits partial text on every chunk instead of waiting for silence, which is what lets the pipeline show words on screen while the user is still talking.

# Parakeet-CTC-0.6B streaming via NeMo
from nemo.collections.asr.models import EncDecCTCModelBPE
asr = EncDecCTCModelBPE.from_pretrained("nvidia/parakeet-ctc-0.6b")
# chunk_ms=320 ms, look_ahead_ms=80 ms
for chunk in audio_stream():
    partial_text = asr.transcribe_streaming(chunk)
    print(partial_text, end="\r")

Step 4: interruption handler

class Dialog:
    def __init__(self):
        self.tts_task = None
 
    def on_user_speech(self, frame):
        if self.tts_task and not self.tts_task.done():
            self.tts_task.cancel()   # barge-in
        # then feed to streaming ASR
 
    def on_final_user_utterance(self, text):
        self.tts_task = asyncio.create_task(self.reply(text))
 
    async def reply(self, text):
        async for tts_chunk in llm_then_tts(text):
            speaker.write(tts_chunk)

This hinges on async I/O and cancellable TTS streaming; calling peerconnection.stop() on the WebRTC audio track is the canonical way to do it.

Here's the 2026 stack, layer by layer:

• Buffering 500 ms to be safe. The buffer is your latency floor. Shrink it.
• Not pinning threads. Give the audio callback a lower priority than the UI thread and you get glitches under load.
• TTS chunks too small. Sub-200 ms chunks make vocoder artifacts audible. 320 ms chunks are the sweet spot.
• No jitter buffer. Real networks are jittery. Skip the smoothing and you get pops.
• Single-shot error handling. Audio pipelines must be crash-proof. One exception kills the session.

Save this as outputs/skill-realtime-pipeline.md: a real-time audio pipeline design with a concrete latency budget for every stage.

1. Easy. Run code/main.py. Simulates a ring buffer + energy VAD; prints stage latencies for a fake 10-second stream.
2. Medium. Using sounddevice, build a passthrough loop that processes your mic in 20 ms frames and prints VAD state at each frame.
3. Hard. Build a full duplex echo test with aiortc: browser → WebRTC → Python → WebRTC → browser. Measure glass-to-glass latency with a 1 kHz pulse.

• Macháček et al. (2023). Whisper-Streaming: chunked near-streaming Whisper.
• Kyutai (2024). Moshi: full-duplex 200 ms latency.
• LiveKit Agents framework (2024): production audio agent orchestration.
• Silero VAD repo: sub-1 ms VAD, Apache 2.0.
• WebRTC AEC3 paper: echo cancellation under open source.