Optimizing Voice Agent Latency

The responsiveness of a voice agent is determined by TTFA (Time-to-first-audio), the total latency from the moment the user finishes speaking to when the agent starts answering. Latency isn't caused by a single model but accumulates across multiple stages, so the first step in optimization is to understand how long each stage takes and set a Latency Budget.

A voice agent pipeline typically consists of the following 5 stages.

• Capture (microphone): Voice capture and VAD/Turn-detection
• STT: Converting speech to text and finalizing it
• LLM: Text generation (Time-to-first-token)
• TTS: Converting text to speech (Time-to-first-audio)
• Playback: Client-side playback and buffering

According to actual measurements, the biggest contributors to latency are the LLM's time-to-first-token (TTFT) and Endpointing delay, which detects the end of a conversation turn. For optimization, rather than running each stage sequentially, the key is to overlap work between stages to resolve bottlenecks across the entire pipeline.

For effective management, performance should be measured by region using P50 (median) and P95 (95th percentile latency) metrics, and millisecond (ms)-level performance improvements can be achieved through adopting streaming, choosing the right codec, and tuning player buffers.