Interaction Model

Existing voice AI is designed to be turn-based. It's a series of sequential steps: listen to speech, convert it to text, process it, then respond. But real conversation is far more complex. Silence and filler words are mixed in, and context flows across turns.

Existing systems have three fundamental problems:

• They fail to detect user interruptions, either cutting off mid-speech or waiting in silence after a turn completes
• Once they start responding, they stop listening, ignoring the user even if they want to change direction
• They process each turn in isolation, so context from a few turns ago disappears

The interaction model is different. It runs the entire conversation at once, recognizing not just what is being said but when it's being said, and what emotional response that moment requires.

Features of the interaction model:

• Real-time response: Responds at conversational speed with an end-to-end cycle of under 1 second
• Natural interruption/silence handling: Distinguishes between thinking pauses and turn completion to maintain seamless flow
• Context continuity: Carries forward the entire conversation history, so the response at the 10th turn reflects all context from the beginning
• Adaptive delivery: Automatically adjusts vocal tone (calming, confident, clear, etc.) depending on the task
• Parallel task execution: Continues speaking while retrieving information or making tool calls

ElevenLabs implements this with a cascade architecture. Instead of a single model handling everything, each stage is an independent, specialized component. This allows each stage to be optimized independently, and because all components are built in-house, they pass rich context to each other so the whole system behaves like one coherent conversation.

Tech stack:

• Scribe v2 Realtime: An STT model that transcribes 90+ languages with 150ms latency. It's robust to background noise, accents, interruptions, and non-verbal events like laughter, and also handles domain-specific vocabulary such as medical terminology or financial jargon
• Speculative turn-taking: Reads the flow of conversation for natural turn management, instead of relying on a fixed silence threshold. Improved voice activity detection models also filter out background speech and short responses
• Eleven v3 Conversational: A TTS model for real-time bidirectional conversation. It maintains emotional temperature across the whole turn, so responses reflect all preceding context
• Expressive Mode: Calms the customer when they're frustrated, reassures them when they're confused, and switches to being direct when clarity is needed. It also reads expression tags like [laughs], [whispers], [sighs]
• Flash v2.5: Low-latency TTS that generates speech in under 75ms. Supports 32 languages
• Speech Engine: The layer that connects the stack via WebSocket. STT and TTS run on ElevenLabs, while the LLM runs on the user's own server

In actual conversation, each stage doesn't execute in a strict sequence. While speech is being streamed, Scribe already begins transcribing, and turn-taking evaluates the next stage even while transcription isn't finished. While the LLM is assembling context, parallel tool calls query the database, and while Eleven v3 is synthesizing speech, playback has already begun.

Throughout this rapid process, the conversation feels natural. The user doesn't adapt to the machine. They simply speak as if talking to a human.

This technology is currently in production, and ElevenAgents is being used by companies around the world to handle real customer conversations at scale. With SOC 2 Type II, ISO 27001, HIPAA, PCI DSS Level 1 certifications and data residency options, it can be deployed even in highly regulated, high-stakes environments.