Webinar Recap: Giving Text Chatbots a Human-Sounding Voice

Text-based chatbots have a limitation in that they cannot pick up on nonverbal context embedded in a user's tone, such as frustration, urgency, or confusion. Introducing a voice interface to address this is emerging as a key challenge for companies.

The biggest technical challenges in building a voice system are maintaining Turn-taking and Context. Simply detecting silence disrupts the natural flow of conversation, and relying on text alone makes it difficult to distinguish differences in meaning that arise from tone even when the same words are used.

The optimal approach for efficient integration is to use a Dual WebSocket architecture.

• Create separate connections between the client and the ElevenLabs API, and between the server and the ElevenLabs API.
• Transcribe the user's speech in real time and pass it to the server.
• Immediately feed the streaming response being generated by the LLM into audio synthesis to minimize First-byte latency.
• Pass conversation history via the onTranscript method, and carry over the context of the text conversation into the voice session using contextualUpdate.

For a successful implementation, it is recommended to choose an LLM with fast real-time response speed, leverage WebRTC with built-in echo and noise cancellation, and apply an automatic language detection pattern so users don't have to manually select a language.