agents: Build Real-Time Voice Agents with Flexible STT, LLM, and TTS Combinations

A framework for building real-time conversational voice agents that run on the server. It provides code-level control over the entire process, from listening to the user's speech to understanding it and generating a spoken response. Beyond simple chatbots, it features multimodal capabilities for both seeing and hearing.

You can freely combine models at each stage, including STT, LLM, TTS, and Realtime APIs. It supports major providers such as Deepgram, OpenAI, and Cartesia, and integrates MCP server tools with a single line of code. It also includes built-in support for phone call integration and a WebRTC client SDK.

Transformer-based semantic turn detection accurately identifies when a user finishes speaking to reduce interruptions. It includes built-in role switching and task scheduling across multiple agents. An AI-judgment-based testing framework is also included to verify the non-deterministic behavior of LLMs.

It offers various example codes, including voice calls, video avatars, and restaurant reservations. It supports both Python and JS/TS libraries and is open source, allowing you to run the entire stack on your own server. It is suitable for developers looking to standardize the backend logic of complex voice AI applications.