AI BriefingAdding voice to agents
Key point
Cloudflare adds a real-time voice pipeline to the Agents SDK, letting voice and text be handled by the same agent.
Details
Cloudflare has released @cloudflare/voice, an experimental voice pipeline for the Agents SDK. The core idea is not to separate voice into its own framework, but to build it directly on top of the existing Agent, Durable Object, SQLite conversation history, and WebSocket structure.
The package provides the following:
withVoice(Agent): for conversational voice agentswithVoiceInput(Agent): for voice-input-only use cases like dictation, voice searchuseVoiceAgent,useVoiceInput: React hooksVoiceClient: a framework-agnostic client- Default Workers AI providers: Deepgram Flux STT, Deepgram Nova 3 STT, Deepgram Aura TTS
The flow of operation is clear. When the browser sends microphone audio as 16 kHz mono PCM, the STT session receives it, detects end of speech, and produces a stable transcript. That transcript is passed to onTurn(), and the response is synthesized via TTS and sent back to the client. If onTurn() returns a stream, it is split by sentence to start audio playback immediately, reducing Time-to-First Audio.
The structural advantage is that voice becomes just one feature of the agent. Users can type, switch to voice, and switch back to text, all while using the same agent, the same state, and the same tools.
It also minimizes the paths audio and text travel across multiple services to reduce backend latency. By leveraging the Cloudflare network and Workers AI bindings, it reduces network hops, and voice responses can start streaming immediately.
In the example code, you configure the server with withVoice(Agent) and only need to implement transcriber, tts, and onTurn(). In more complex examples, streamText() and context.signal are used to stream LLM responses and cancel immediately if the user interrupts.
Non-voice-only input scenarios are also supported. withVoiceInput provides a lightweight pattern that processes voice as text only, and if text is sent directly via sendText() on the same connection, it skips STT and goes straight to onTurn(). As a result, voice, text, tools, scheduling, and persistence can all be handled together within a single agent design.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.