Voice AI infrastructure: The 2026 stack and architecture

ElevenLabs launched Eleven v4, described as its most emotionally expressive TTS model, and defined the 9-layer production Voice AI stack: ASR, turn-taking, LLM, orchestration, tools, knowledge, synthesis, telephony, and monitoring.
Eleven v4 Turbo delivers a median time-to-first-speech of approximately 150 ms and median inference latency of 100 ms, supporting over 90 languages and ranking first on the September 2026 Artificial Analysis Provider Voice Arena Leaderboard with 75% listener preference in blind tests.
The production stack utilizes Scribe v2 Realtime for ASR with a 150 ms transcription latency and a 7.0% word error rate on the Eka Medical ASR benchmark, while ElevenAgents supports both Custom and Realtime modes for flexible LLM and TTS configuration.
Production reliability is demonstrated by handling over 10 million conversations per week, with Meesho automating 90% of support queries and the Czech Republic welfare hotline resolving 85% of calls independently.
Architecture features include LLM cascading with a 4-second default timeout, RAG latency reduction from 326 ms to 155 ms via self-hosted models, and Guardrails 2.0 supporting streaming verification with under 500 ms audio playback before blocking.
Telephony integration supports SIP trunking via Twilio, Vonage, and Sinch, with batch calls limited to 50% of workspace concurrency or 70% of agent limits, and SDKs available for Python, JavaScript, React, Swift, Kotlin, and Unity.

Key point