AI BriefingHow to Optimize Latency in Conversational AI
Key point
To reduce the latency that determines the immersiveness of conversational AI, the latency of each stage—ASR, TTI, LLM, and more—must be managed holistically.
Details
In conversational AI, Latency goes beyond being a simple performance metric—it is the key factor that determines whether users feel the immersive sense of talking with a human.
Rather than being determined by a specific bottleneck section, the latency of conversational AI has the characteristic of appearing as the sum of the latency at each stage, such as ASR, TTI, LLM, and TTS.
-
ASR (Automatic Speech Recognition): This is the process of converting speech to text. Latency here does not refer to the time that progresses in the background while the user is speaking, but rather the time from when the utterance ends until text generation is complete. The open-source model Whisper incurs latency of 300ms or more, but an optimized custom implementation can reduce this to under 100ms.
-
Turn-Taking / Interruption (TTI): This is the process of determining whether the user has finished speaking, a role performed by the VAD (Voice Activity Detector). Since it's not simply about detecting silence but about setting a threshold to determine whether the user has truly finished their utterance, a certain level of latency is unavoidable.
-
Text Processing (LLM): This is the stage where a response is generated based on the produced text. Models such as Gemini Flash 1.5 show fast response speeds of under 350ms, and the choice of model has a significant impact on overall latency.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.