OpenAI unveils low-latency voice AI architecture

OpenAI has revealed how it redesigned its WebRTC stack for ChatGPT voice and the Realtime API.

The core idea is to expose only a small, fixed UDP surface to the public internet while keeping session state inside an internal relay + transceiver structure. The Transceiver handles ICE, DTLS, SRTP, and session lifetime and state, while the Relay reads only minimal metadata without decrypting packets and forwards them to the appropriate transceiver. The relay does not handle ICE state management or codec negotiation, and clients see the standard WebRTC flow as-is.

• The previous one-port-per-session model created port exhaustion and operational complexity on Kubernetes and cloud load balancers.
• SFU works well for multi-party scenarios, but is overkill for workloads like OpenAI's primary traffic, which is mostly 1:1 and sensitive to turn-taking latency.
• Each transceiver receives multiple sessions through a single shared UDP socket, and the first packet determines its destination using a routing hint placed in the ICE ufrag.
• Even if the relay restarts, a session can be restored via the next STUN packet, with Redis assisting through a <client IP+Port, transceiver IP+Port> mapping.

OpenAI also explained that it applied Global Relay and Cloudflare geo/proximity steering so that media is received from a location close to the user, reducing first-hop latency, jitter, and packet loss. This architecture is applied to ChatGPT voice, the Realtime API WebRTC endpoint, and real-time research projects.