How OpenAI Delivers Large-Scale, Low-Latency Voice AI

OpenAI redesigned its WebRTC infrastructure for voice products like ChatGPT voice and the Realtime API, where conversational latency determines perceived quality. At the scale of over 900 million weekly active users, fast connection setup, low jitter, and low packet loss were essential.

The core decision was choosing a transceiver model instead of an SFU. A transceiver at the edge terminates the client's WebRTC connection, and internally converts it into a simple protocol for inference, transcription, voice generation, and tool calls. This transceiver owns all session state, including ICE, DTLS, and SRTP.

The initial implementation was a single service based on Go and Pion, but the traditional one-port-per-session approach didn't fit well on Kubernetes.

• Large UDP port ranges complicated load balancer, firewall, and health check operations.
• ICE and DTLS are stateful, so if packets went to a different process, sessions could break.
• Frequent pod scaling up, scaling down, and rescheduling meant that operating with fixed port ranges hurt elasticity.

The solution was separating relay + transceiver. The relay is a lightweight UDP forwarder that reads only packet metadata without encryption or codec negotiation, while the transceiver behind it fully handles WebRTC session state. Each transceiver handles multiple sessions through a single shared UDP socket, signaling is handled by the transceiver, and the SDP answer contains the shared relay VIP and UDP port. The destination is determined by reading the ICE ufrag from the client's first packet, the STUN binding request.

The relay maintains sessions with minimal in-memory state and timers only, and recovers via the next STUN packet after a restart. When needed, it stores a <client IP + Port, transceiver IP + Port> mapping in Redis to recover the path more quickly.

For global deployment, Global Relay and Cloudflare's geo/proximity steering are used to route users to the nearest ingress and cluster. This keeps the public UDP surface small while reducing first-hop latency, jitter, and loss bursts, making voice conversations feel more natural.